Method for generating an output information with a confidence information in response to an input information and system

Abstract

A method for generating an output information with a confidence information is provided, the method comprising: - receiving the input information characteristic of a user query, - determining a first response information based on a stand-alone question lookup mechanism, - determining a second response information based on a conversation lookup mechanism, - generating a third response information based on an RAG mechanism, - generating the output information and a corresponding aggregated score dependent on the first response information, the second response information and / or the third response information as well as dependent on historical data stored in a database based on an LLM answer check mechanism, - generating the confidence information of the corresponding output information based on a lookup score generator and storing the confidence information in the database.

Description

[0001] P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0002] 1

[0003] Description

[0004] METHOD FOR GENERATING AN OUTPUT INFORMATION WITH A CONFIDENCE INFORMATION IN RESPONSE TO AN INPUT INFORMATION AND SYSTEM

[0005] A method for generating an output information with a confidence information in response to an input information is provided. Furthermore, a system is provided.

[0006] Generally, Retrieval-Augmented Generation, RAG, is an artificial intelligence, Al, technique that combines information retrieved from an authoritative knowledge source and incorporates it into generated text by a large language model, LLM. Typically, the RAG has a typical document retriever module, e . g. to retrieve a relevant document from a large corpus of documents in a database, and the LLM is configured, e . g. to generate an answer to the input information based on the document retrieved by the typical document retriever .

[0007] However, such RAG faces typical problems like reproducibility, un-reliability, e . g. due to hallucination, and repetitive computations expense, e . g. that repeated questions lead to repeated re-triggering of RAG systems .

[0008] A method for improving an overall performance within an RAG framework is to be provided. Furthermore, a system and a computer program which are able to perform such a method are provided. In addition, a computer-readable storage medium with such a computer program is provided.

[0009] Such a method is described by the subj ect-matter of the independent claims . Advantageous embodiments, implementations P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0010] 2

[0011] and further developments are the subj ect-matter of the respective dependent claims .

[0012] A method for generating an output information with a confidence information in response to an input information is described. Particularly, the method described herein is a computer-implemented method. Exemplarily, the method, in particular each method step, is performed by a processing unit, e . g. comprised by a computer device .

[0013] According to at least one embodiment of the method, the input information characteristic of a user query is received by the processing unit . The input information, particularly the user query, is exemplarily characteristic of a technical question. This means that the user query is in particular characteristic of a question. The technical question comprises, for example, questions related to device information, operating questions related to a device, and / or symptoms of a failure of a device . Exemplarily, the input information is characteristic of a new question or of a part of a historical conversation comprising at least one historical question, and / or a historical question. The historical question and / or the historical conversation, in particular a plurality thereof, is saved in a database . The database and the processing unit particularly communicate with one another .

[0014] According to at least one embodiment of the method, a first response information is determined, by the processing unit, based on a stand-alone question lookup mechanism dependent on the input information. Exemplarily, the stand-alone question lookup mechanism is characteristic of an identification and retrieval of the historical question and an evaluation of a P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0015] 3

[0016] quality of matched historical questions retrieved from the database . Subsequently, it is determined, for example, whether the matched historical question is retrieved from the database or not . A question output of the determination is characteristic of no matched historical questions or at least one, particularly a plurality of, matched historical questions, along with respective values, wherein the first response information is characteristic of the question output .

[0017] In particular, the second response information is characteristic of at least one pair of a question and an answer, being determined by the stand-alone question lookup mechanism.

[0018] According to at least one embodiment of the method, a second response information is determined, by the processing unit, based on a conversation lookup mechanism dependent on the input information. Exemplarily, the conversation lookup mechanism is characteristic of an identification and retrieval of the historical conversation and presenting the historical conversation to the user upon request or leveraging the historical conversation to extract at least one most relevant historical question from the database .

[0019] Subsequently, it is determined, for example, whether the matched historical conversation is retrieved from the database or not . A conversation output of the determination is characteristic of no matched historical conversations or at least one, particularly a plurality, matched existing conversations, along with respective values, wherein the second response information is characteristic of the conversation output . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0020] 4

[0021] In particular, the second response information is characteristic of at least one pair of a question and an answer, being determined by the conversation lookup mechanism.

[0022] According to at least one embodiment of the method, a third response information is generated, by the processing unit, dependent on the input information based on a retrieval-augmented generation, RAG, mechanism. The RAG mechanism comprises, for example, a retrieval mechanism, which is configured to retrieve relevant documents based on the input information .

[0023] Exemplarily, the RAG mechanism additionally comprises an LLM input context mechanism, wherein a context to be an input to the LLM is formulated. The context can include a range of prompts, the input information, and the retrieved relevant documents . If present, a summary of the question output and / or conversion output can also be added to the LLM input context mechanism.

[0024] Exemplarily, the RAG mechanism additionally comprises an LLM, which is configured to generate at least one RAG response to the input information, dependent on the context provided. In particular, the third response information is characteristic of the RAG response .

[0025] In particular, the third response information is characteristic of at least one pair of a question and an answer, being determined by the RAG mechanism.

[0026] According to at least one embodiment of the method, the output information and a corresponding aggregated score are P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0027] 5

[0028] generated, by the processing unit, dependent on the first response information, the second response information and / or the third response information as well as dependent on historical data stored in a database based on an LLM answer check mechanism.

[0029] The historical data comprises, for example, historical first response information, and / or historical second response information, and / or historical third response information, as well as historical input information and historical output information. Further, the historical data comprises, exemplarily, historical user feedback.

[0030] The LLM answer check mechanism is, for example, configured to validate a correctness of at least one answer included in the first information, the second information and / or the third information before sharing them with the user . In particular, all information derived from the LLM answer check mechanism is stored in the database .

[0031] The LLM answer check mechanism exemplarily serves two main purposes . The LLM answer check mechanism particularly acts as a filter, ensuring that accurate and helpful answers are highlighted for the user . Further, the LLM answer check mechanism is configured to produce scores considering the input information, the context, and the corresponding response information, e . g. generated by the LLM.

[0032] The LLM answer check mechanism is advantageously configured to generate preliminary scores, e . g. corresponding to the aggregated score, that assist in recognizing limitations and, when necessary, drawing upon domain expertise . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0033] 6

[0034] According to at least one embodiment of the method, the confidence information of the corresponding output information is generated, by the processing unit, based on a lookup score generator, and the confidence information is stored in the database .

[0035] The lookup score generator is configured to generate a unique final confidence score and / or a confidence class, e . g. comprised by the confidence information derived from user feedback, the corresponding generated response information, and the aggregated score . This advantageously contributes to an enhancement of a typical question matching process .

[0036] In particular, the confidence information is instrumental in enhancing the conventional question matching mechanism described herein above .

[0037] An idea of the method described herein is, inter alia, to use different mechanisms having an inter-mechanism connectivity, which advantageously enhances the overall efficiency and effectiveness of the method. In particular, the method advantageously enhances the RAG mechanism using a persistent storage mechanism for both stand-alone questions and entire conversations .

[0038] The proposed method further advantageously provides an efficient validation strategy at multiple stages, ensuring high-confidence answers from historical interactions . It also advantageously incorporates user feedback to estimate the confidence associated with each question-answer pair, resulting in a robust and reproducible method. This method can advantageously identify its limitations and avoid providing invalid answers, while instead directing users to P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0039] 7

[0040] domain experts when necessary. Further, the method is advantageously versatile and can be applied across various domains .

[0041] Therefore, the method advantageously enhances the validity and reliability of the answers provided by the method to the input information, e . g. user query, whether they occur as single answers or as part of the conversation.

[0042] Thus, a unique machine learning algorithm is provided, comprising a specialized system architecture to improve efficiency of the method. In particular, the interaction of the processing unit and the database provides the above-mentioned advantages . Furthermore, the generation of the confidence information particularly improves security and validity of the output information.

[0043] Exemplarily, the output information is characteristic of a technical answer . This means that the output information is in particular characteristic of an answer dependent on the technical question. The output information comprises, for example, operating information of the device and / or repair information of the device .

[0044] According to at least one embodiment of the method, the stand-alone question lookup mechanism comprises providing the input information to a retrieval data store module, a hypothetical document embeddings module and a hypothetical answer creation module .

[0045] According to at least one embodiment of the method, the retrieval data store module is configured to determine a retrieved matching question and a corresponding retrieved P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0046] 8

[0047] matching answer dependent on the input information.

[0048] Exemplarily, the retrieval data store module mainly serves as a foundational data repository. Exemplarily, a primary focus of the retrieval data store module is to archive previously posed user queries along with their associated embeddings and textual details . For example, a primary key composed of a conversation ID and a question ID is used, which facilitates the retrieval of supplementary information such as scores, answers, context, and more .

[0049] Exemplarily, the input information is provided to a keyword search, exemplarily a Bm25 keyword search. The keyword search is characteristic of fetching pertinent context relevant to the user query. Exemplarily, the input information is provided to a semantic similarity search, in particular parallel to the keyword search. The semantic similarity search is characteristic of identifying a semantic correlation between embeddings produced for texts by encoder models . For example, a semantic model, such as ADA-003, NV embed, Colbert2, among others, can be utilized for generating embeddings .

[0050] Subsequently, the user query is correlated with the historical data . In this step, additional details are obtained for the user query from alternate schemes using a foreign key, encompassing scores, answers, context, classification, and more .

[0051] The retrieved matching question and the corresponding retrieved matching answer are, for example, characteristic of the outcome of the keyword search and the semantic similarity search and in particular of the additional details . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0052] 9

[0053] According to at least one embodiment of the method, the hypothetical document embeddings module is configured to determine a relevant context and a relevant score dependent on the input information dependent on a second LLM.

[0054] Exemplarily, the second LLM is employed to generate a hypothetical document embedding. Furthermore, the relevant context and the relevant score are determined dependent on the input information, the hypothetical document embedding or relevant context, and the second LLM' s response .

[0055] According to at least one embodiment of the method, the hypothetical answer creation module is configured to generate a hypothetical answer by a third LLM and further configured to determine a relevance score between the hypothetical answer and a retrieved answer which is characteristic of the corresponding retrieved matching answer . Exemplarily, a plurality of hypothetical answers is generated. In this case, the relevance scores of all combinations are averaged to obtain a single relevance score .

[0056] According to at least one embodiment of the method, the hypothetical answer and the retrieved answer relevance is determined dependent on a retrieved user feedback score, and the retrieved user feedback score is dependent on the retrieved matching question and a corresponding retrieved matching answer . In particular, the retrieved user feedback score comprises the historical user feedback comprising historical confidence information such as a historical confidence class, a historical confidence score and / or other individual scores . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0057] - 10 -

[0058] According to at least one embodiment of the method, a hypothetical hallucination score is generated dependent on the hypothetical answer and the corresponding retrieved matching answer dependent on a question threshold. If an averaged relevance score, between the relevance score of the hypothetical answers and the retrieved user feedback score, is above the question threshold, the hypothetical hallucination score is generated. In particular, the hypothetical answer, particularly the plurality of hypothetical answers, is fed to a further LLM, e . g. a self-check-GPT, for generating the hypothetical hallucination score . Alternatively, a semantic entropy process is used for generating the hypothetical hallucination score .

[0059] According to at least one embodiment of the method, an individual score normalization and a scaling are performed on the score of the relevant context and / or the hypothetical hallucination score . Exemplarily, based on a targeted application the scaling of the scores is performed. As user feedback can vary across multiple scales depending on the targeted application and as the scores coming from the LLM answer check mechanism would also be not normalized, the score of the relevant context and / or the hypothetical hallucination score are adapted accordingly. Thus, normalizing and converting them to a uniform scale is advantageously provided.

[0060] According to at least one embodiment of the method, an additive weighted scoring is performed on the score of the relevant context and / or the hypothetical hallucination score on which the individual score normalization and the scaling is performed, dependent on the retrieved user feedback score and / or business defined weights per score . For the additive P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0061] weighted scoring, the score of the relevant context and / or the hypothetical hallucination score on which the individual score normalization and the scaling are performed are added based on the inputs from the business defined weights per score, and particularly the retrieved user feedback score, to get a single additive weighted score . This additive weighted score is, for example, stored in the database . A possible equation could be w1x1+ w2x2+W3X3 + w4x4+ +wNxN, where each wtis the weight to be defined by the business to be given for each scores xi fwhere 1 < i < N .

[0062] According to at least one embodiment of the method, ranked retrieved question and answer pairs are generated dependent on the score of the relevant context and / or the hypothetical hallucination score on which the additive weighted scoring is performed, wherein the ranked retrieved question and answer pairs are characteristic of the first response information.

[0063] Exemplarily, as there can be multiple matches and each match has to be ranked based on the additive weighted scoring, the retrieved question and answer pairs are sorted with a descending sort and wherein a top one match is selected for being the first response information.

[0064] According to at least one embodiment of the method, the conversation lookup mechanism comprises providing the input information to a further first retrieval data store module, a further second retrieval data store module, a further hypothetical document embeddings module and a further hypothetical answer creation module .

[0065] According to at least one embodiment of the method, the further first retrieval data store module is configured to P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0066] - 12 -

[0067] determine a retrieved matching conversation and a corresponding retrieved matching conversation answer dependent on the input information. For example, the further first retrieval data store module is functionally the same as the retrieval data store module, where conversations are matched instead of questions .

[0068] According to at least one embodiment of the method, the further second retrieval data store module is configured to determine a further retrieved matching question and a further corresponding retrieved matching answer dependent on the input information. For example, the further second retrieval data store module is functionally the same as the retrieval data store module .

[0069] In particular, all features in connection with the retrieval data store module are therefore also applicable to the further second retrieval data store module and analogously to the further first retrieval data store module .

[0070] According to at least one embodiment of the method, the further hypothetical document embeddings module is configured to determine a further relevant context and a further relevant score dependent on the input information and dependent on a fourth LLM. Exemplarily, the fourth LLM is employed to generate a further hypothetical document embedding. Furthermore, the further relevant context and the further relevant score are determined dependent on the input information, the further hypothetical document embedding or further relevant context, and the fourth LLM' s response .

[0071] According to at least one embodiment of the method, the further hypothetical answer creation module is configured to P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0072] generate a further hypothetical answer by a fifth LLM and is further configured to determine a further relevance between the further hypothetical answer and a further retrieved answer, which is characteristic of the corresponding retrieved matching conversation answer having an intersection match with the further corresponding retrieved matching answer . Exemplarily, a plurality of further hypothetical answers is generated. In this case, the further relevant scores of all combinations are averaged to obtain a further single relevant score .

[0073] Exemplarily, for the intersection match, intersection between matched conversations and questions is conducted, for example if 5 conversations are matched and 5 questions are matched independently, an intersection is checked between those, which independent matched questions are part of the independent matched conversations, wherein only the intersection is used for determining the further retrieved answer .

[0074] According to at least one embodiment of the method, the further hypothetical answer and the further retrieved answer relevance is determined dependent on a further retrieved user feedback score, and the further retrieved user feedback score is dependent on the intersection match. In particular, the further retrieved user feedback score comprises the historical user feedback comprising historical confidence information such as a historical confidence class, a historical confidence score and / or other individual stores .

[0075] According to at least one embodiment of the method, a further hypothetical hallucination score (A. 5. 18 ) is generated dependent on the further hypothetical answer and the further P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0076] - 14 -

[0077] corresponding retrieved matching answer (A. 5. 16. ) dependent on a conversation threshold (A. 5. 17. ) . If a further averaged relevance score, between the further relevance score of the further hypothetical answers and the further retrieved user feedback score, is above the conversation threshold, the further hypothetical hallucination score is generated. In particular, the further hypothetical answer, particularly the plurality of further hypothetical answers, is fed to a further LLM, e . g. a further self-check-GPT, for generating the further hypothetical hallucination score . Alternatively, a semantic entropy process is used for generating the hypothetical hallucination score .

[0078] According to at least one embodiment of the method, a further individual score normalization and a further scaling are performed on the further score of the further relevant context and / or the further hypothetical hallucination score . Exemplarily, based on the targeted application, the further scaling of the scores is performed. As user feedback can vary across multiple scales depending on the targeted application, and as the scores coming from the LLM answer check mechanism would also be not normalized, the score of the further relevant context and / or the further hypothetical hallucination score are adapted accordingly. Thus, normalizing and converting them to a uniform scale is advantageously provided.

[0079] According to at least one embodiment of the method, a further additive weighted scoring is performed on the further score of the further relevant context and / or the further hypothetical hallucination score on which the further individual score normalization and the further scaling is P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0080] 15

[0081] performed, dependent on the further retrieved user feedback score and / or further business defined weights per score .

[0082] Exemplarily, the further additive weighted scoring is slightly different from the additive weighted scoring in the sense that the equation which is created is for matched questions within the corresponding conversation.

[0083] According to at least one embodiment of the method, ranked retrieved conversation and answer pairs are generated dependent on the further score of the further relevant context and / or the further hypothetical hallucination score on which the further additive weighted scoring is performed, and the ranked retrieved conversation and answer pairs are characteristic of the second response information.

[0084] Based on the further additive weighted scoring of questions within conversations, the retrieved conversation and answer pairs are sorted with a descending sort and wherein a top one match is selected for being the second response information.

[0085] According to at least one embodiment of the method, the LLM answer check mechanism comprises : providing the input information, the first response, the second response and / or the third response to the LLM answer check mechanism, and generating a fact checking score .

[0086] According to at least one embodiment of the method, when the corresponding response has a correct fact, a hallucination score is generated. P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0087] According to at least one embodiment of the method, when the hallucination score is smaller than a first preset threshold, a retrieval relevance score is generated.

[0088] According to at least one embodiment of the method, when the retrieval relevance score is smaller than a second preset threshold, the aggregated score is generated, which comprises weightage scores defined for the fact checking score, the hallucination score and the retrieval relevance score .

[0089] According to at least one embodiment of the method, an LLM generated answer is generated characteristic of the output information based on the first response, the second response and / or the third response . Exemplarily, the LLM generated answer is additionally based on a historical first response, a historical second response and / or a historical third response stored in the database .

[0090] According to at least one embodiment of the method, a user feedback is provided dependent on the output information comprising at least one of a current answer feedback, a conversation feedback and / or a citation feedback.

[0091] Advantageously, due to the provision of the user feedback the final aggregated score and / or the confidence thereof can be determined comparatively good. Further, the respective user feedback is stored in the database, which can be used for further input information.

[0092] According to at least one embodiment of the method, the lookup score generator comprises : providing the output information to a first decision block, and providing the user feedback to a second decision block. P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0093] According to at least one embodiment of the method, the first decision block determines if the output information is LLM generated or a default message .

[0094] According to at least one embodiment of the method, the second decision block determines if the user feedback is present .

[0095] According to at least one embodiment of the method, a final individual score normalization and a final scaling are performed on the aggregated score .

[0096] According to at least one embodiment of the method, a final additive weighted scoring is performed on the aggregated score on which the final individual score normalization and the final scaling are performed, dependent on business defined weights per score .

[0097] According to at least one embodiment of the method, the confidence information is determined based on the aggregated score on which the final additive weighted scoring is performed, and the confidence information comprises a confidence score and / or a confidence class .

[0098] Furthermore, a system is specified. The system is configured to perform the method described herein. Therefore, all features and embodiments disclosed in connection with the method are also disclosed in connection with the system and vice versa .

[0099] In addition, a computer program is specified comprising instructions which, when the computer program is executed by P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0100] a computer, cause the computer program to execute the method described herein.

[0101] Further, a computer-readable storage medium is specified on which the computer program described herein is stored.

[0102] Exemplary embodiments of the method are explained in more detail below with reference to the Figures .

[0103] Figure 1 shows a flowchart of the method according to an exemplary embodiment .

[0104] Figure 2 shows a flowchart of a stand-alone question lookup mechanism of the method according to an exemplary embodiment .

[0105] Figure 3 shows a flowchart of a conversation lookup mechanism of the method according to an exemplary embodiment .

[0106] Figure 4 shows a flowchart of an LLM answer check mechanism of the method according to an exemplary embodiment .

[0107] Figure 5 shows a flowchart of a lookup score generator of the method according to an exemplary embodiment .

[0108] Elements of the same structure or function are marked with the same reference signs across all Figures .

[0109] The system 1 according to Figure 1 comprises a processing unit and a database 4, wherein the following method steps can be performed by the processing unit, being connected to the database 4 . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0110] - 19 -

[0111] Initially, the input information 2 characteristic of a user query is received by the processing unit . In order to determine if the user query, also referred to as user question, is a new question or part of a conversation, to then direct the user question through an appropriate pipeline : Pl, comprising the method stages A. 2 to A. 4 for new questions, or P2, comprising the method stages A. 5 to A. 12 for conversations . Advantageously, this ensures effective utilization of the historical questions and historical answers stored in the database 4 .

[0112] Stage A concerns a processing of the user question. In method stage A. l the decision block is configured to determine whether the incoming user question is a new question or part of an ongoing conversation. This determination can be made by monitoring the session details in which the user question arises .

[0113] For this, the input information 2 is provided to A. 2 and A. 3. In A. 2 and A. 3 a first response information is determined by the processing unit based on a stand-alone question lookup mechanism dependent on the input information 2. The standalone question lookup mechanism is characteristic of method step A. 2. The stand-alone question lookup mechanism is further described in Figure 2 .

[0114] In method stage A. 3, a decision-making component determines whether a retrieved matching question and a corresponding retrieved matching answer has been retrieved from the database 4 or not, i . e . if the retrieved matching question and the corresponding retrieved matching answer are part of first response information. In particular, a question output can yield no retrieved matching question or multiple P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0115] - 20 -

[0116] retrieved matching questions, along with their respective values .

[0117] In method stage A. 4 a retrieved top one question and answer pair is determined when a retrieved matching question is identified in A. 3. In particular, in A. 4 the retrieved matching question is determined that has the highest degree of matching, along with the corresponding retrieved matching answer and other relevant values .

[0118] Further, the input information 2 is provided to A. 5 and A. 6. In A. 5 and A. 6 a second response information is determined by the processing unit based on a conversation lookup mechanism dependent on the input information 2 . The conversation lookup mechanism is characteristic of method step A. 5. The conversation lookup mechanism is further described in Figure 3 .

[0119] Method stage A. 5, akin to A. 2, is distinctive in its targeted strategy for pinpointing a retrieved matching conversation and a corresponding retrieved matching answer within the database 4 .

[0120] In method stage A. 6, a decision-making component determines whether a retrieved matching conversation and a corresponding retrieved matching answer have been retrieved from the database 4 or not, i . e . if the retrieved matching conversation and the corresponding retrieved matching answer are part of second response information. In particular, a conversation output can yield no retrieved matching conversation or multiple retrieved matching conversations, along with their respective values . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0121] - 21 -

[0122] In method stage A. 7 a retrieved top one conversation and answer pair is determined when a retrieved matching conversation is identified in A. 6. In particular, in A. 7 the retrieved matching conversation is determined that has the highest degree of matching, along with the corresponding retrieved matching conversation and other relevant values .

[0123] Subsequently, in method stage A. 8 it is determined if a conversation score is higher than a specified threshold. This decision block assesses whether the retrieved top one conversation has the conversation score above the specified threshold. The specified threshold value can be adjusted based on the targeted application' s requirements to choose or rej ect the conversation.

[0124] If the conversation score is above the specified threshold, the retrieved top one conversation is forwarded to method stage A. 9, where it is prompted to a user to confirm whether the user wishes to utilize the conversations retrieved from the database 4 directly.

[0125] If the conversation score is not above the specified threshold, or if the user opts not to view conversations from the database 4, the retrieved top one conversation is forwarded to method stage A. 10, where a summary of the retrieved top one conversation is generated, which will be utilized subsequently.

[0126] If the user opts to view conversations from the database 4, in A. 11, the retrieved top one conversation is outputted. In particular, this method stage is characteristic of an output block that presents the user with the details of the retrieved top one conversation from the database 4 . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0127] In method stage A. 12 it is determined if the user is satisfied. A corresponding decision block determines whether the user is satisfied with the retrieved top one conversation from the database 4 based on a further input information 3, e . g. being a user input .

[0128] If the user is not satisfied, the input information 2, particularly the user question, and the summary of the conversation are forwarded to method stage A. 2. If the user is satisfied, the input information 2, particularly the user question, and the summary of the conversation are stored in the database 4 .

[0129] Further, the input information 2, particularly the user question, and the summary in method stage A. 10 are forwarded to method stage A. 2.

[0130] Stage B concerns an RAG mechanism for generating a third response information dependent on the input information 2, comprising a retrieval mechanism B . l, an LLM input context mechanism B .2 and an LLM B .3.

[0131] The retrieval mechanism B . l is dedicated to a retrieval aspect of the RAG mechanism, where relevant documents are pulled out based on the user question. In the LLM input context mechanism B .2, a context to be an input to the LLM B .3 is formulated. This includes a range of prompts, the user ' s questions, and the retrieved documents . Further, the summary of the conversation, if present, and / or the first response information when no retrieved matching question is identified in A. 3, are an input for the LLM input context mechanism B .2. P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0132] - 23 -

[0133] The LLM in B .3 is configured to generate responses to the user ' s question, drawing from the context provided, for forming the third response information. In particular, all outputs of the LLM in B .3 are saved in the database 4.

[0134] The outputs of stages A and B are provided to a scoring and feedback stage C . An output information and a corresponding aggregated score dependent on the first response information, the second response information and / or the third response information as well as dependent on historical data stored in a database 4 based on a large language model are generated by the LLM answer check mechanism C . l, and subsequently outputted to the user C .2.

[0135] Particularly, the LLM answer check mechanism C . l is configured to validate a correctness of the answers before sharing them with the user . The LLM answer check mechanism C . l is described in Figure 4. All scores derived from the LLM answer check mechanism are stored in the database 4 and are also utilized in C .4 for further processes .

[0136] Subsequently, user feedback on the system-generated or retrieved answers is provided in method stage C .3, and the respective user feedback is stored in the database 4. The user feedback can be gathered in various forms, ranging from binary to multiple levels, with values from 0 to M at different intervals based on the targeted application.

[0137] The output information and a corresponding aggregated score as well as the user feedback of C .3 are forwarded to the lookup score generator C .4, where a unique final confidence score and / or a confidence class is generated. This score is P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0138] - 24 -

[0139] advantageously instrumental in enhancing the conventional question matching mechanism. The lookup score generator C .4 is described in Figure 5.

[0140] According to Figure 2, the input information 2 of A. l and / or the summary of the conversation of A. 10 are provided to a retrieval data store module A. 2. 2, a hypothetical document embeddings module A. 2. 14 and a hypothetical answer creation module A. 2. 10. If there are no previous conversation questions and only a new user query, the module forwards the user query alone for further processing. If there is input from previous conversations, as indicated in A. 10 and A. 12, the summary of the conversation is used in the subsequent processing flow.

[0141] The retrieval data store (Q: Q search) in step A. 2. 2 serves mainly as the foundational data repository. In particular, the retrieval data store utilizes a primary key composed of conversation ID and question ID, which facilitates the retrieval of supplementary information such as scores, answers, context, and more .

[0142] Subsequently, the process parallelly moves on to two operations A. 2. 3 and A. 2. 4 being conducted on the retrieval data store . Method stage A. 2. 3 is characteristic of a semantic similarity search for identifying semantic correlations between embeddings produced for texts by encoder models, and method stage A. 2. 3 is characteristic of a keyword search for fetching pertinent context relevant to a given user query.

[0143] In A. 2. 5 the user query is correlated with the historical data . In this step, additional details are obtained for the P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0144] - 25 -

[0145] query-aligned question from alternate schemes using a foreign key, encompassing scores, answers, context, classification, and more, for generating a retrieved matching question and a corresponding retrieved matching answer .

[0146] In step A. 2. 6 it is determined if a retrieved matching question and a corresponding retrieved matching answer is found, i . e . if a match is found. If yes, the sequence continues to A. 2. 7, or else it moves on to A. 3 with no match found .

[0147] In step A. 2. 7, historical questions are selected with a confidence class from the lookup score generator as positive, e . g. in step A. 2. 7, those questions are filtered out which belong to the positive confidence class only.

[0148] In step A. 2. 8 it is determined if any question is present, e . g. a presence of the historical question is checked. If present, then next step A. 2. 9 is triggered, or else step A. 3 is triggered.

[0149] In step A. 2. 9 a retrieved user feedback score is obtained, e . g. comprising a historical confidence class, a historical confidence score and other individual stores of the database 4 .

[0150] Parallel to stage A. 2. 2, in method stage A. 2. 10 a hypothetical answer creation module is triggered, which receives the information of A. 2.1 and is configured to regenerate hypothetical answers again by a third LLM multiple times, passing them to the subsequent stage A. 2. 11 for a comparison with retrieved matching answers from 2.5 over 2.9. P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0151] - 26 -

[0152] The hypothetical answer creation module A. 2. 10 is configured to generate the hypothetical answer by the third LLM.

[0153] Subsequently, in step A. 2. 11, a relevance score between the hypothetical answer and the retrieved answer, which is characteristic of the corresponding retrieved matching answer, is determined, particularly when a positive confidence class questions has been matched and exists .

[0154] Subsequently, in stage A. 2. 12, if an average relevance score between hypothetical answers and retrieved user feedback scores is above a predetermined threshold, wherein the predetermined threshold is based on the target application, the method moves on to the next stage A. 2. 13, or the method proceeds with A. 3.

[0155] In stage A. 2. 13, a hypothetical hallucination score is generated dependent on the hypothetical answer and the corresponding retrieved matching answer of A. 2. 11. Stage A. 2. 13 is in particular similar to step C .1.3 described in connection with Figure 4 .

[0156] Parallel to stage A. 2. 2 and stage A. 2. 10, in method stage A. 2. 14 a hypothetical document embeddings module is triggered, which receives the information of A. 2.1 and is configured to determine a relevant context and a relevant score in stage A. 2. 15 dependent on the input information 2 dependent on a second LLM.

[0157] The hypothetical document embeddings module in step A. 2. 14 corresponds to the module described in stage C .1.5 first part regarding Figure 4. Further, the determination of the relevant context and the relevant score in stage A. 2. 15 P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0158] - 27 -

[0159] corresponds to the module described in stage C .1.5 second part regarding Figure 4 .

[0160] In stage A. 2. 16, an individual score normalization and a scaling are performed on the score of the relevant context of stage A. 2. 15 and the hypothetical hallucination score of stage A. 2. 13. Stage A. 2. 16 corresponds to stage C .1.5 described in connection with Figure 4 .

[0161] Subsequently, in stage A. 2. 18 an additive weighted scoring is performed on the score of the relevant context of stage A. 2. 15 and / or the hypothetical hallucination score of stage A. 2. 13 on which the individual score normalization and the scaling of stage A. 2. 16. is performed, dependent on the retrieved user feedback score of stage A. 2. 9, and / or business defined weights per score of7 stage A. 2. 17.

[0162] The additive weighted scoring of stage A. 2. 18 corresponds to stage C .4.7 described in connection with Figure 5.

[0163] Subsequently, in stage A. 2. 19 a final score normalization and a final scaling is performed on the output of stage A. 2. 18. The score normalization and the scaling stage A. 2. 19 corresponds to stage C .4.8 described in connection with Figure 5.

[0164] Subsequently, in stage A. 2. 20 ranked retrieved question and answer pairs are generated for the output of stage A. 2. 19, wherein the ranked retrieved question and answer pairs are characteristic of the first response information in stage A. 3. In particular, as there can be multiple matches and each match has to be ranked based on the final score normalization and the final scaling generated in stage A. 2. 19, in stage P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0165] - 28 -

[0166] A. 2. 20 a descending sort is performed and returns the top one match being passed to stage A. 3.

[0167] According to Figure 3, the input information 2 of A. l is provided to a further first retrieval data store module A. 5. 3, a further second retrieval data store module A. 5. 7, a further hypothetical document embeddings module A. 5. 19 and a further hypothetical answer creation module A. 5. 15.

[0168] Whenever the input information 2, e . g. the user question, is part of a conversation, stage A. 5.1 is triggered, characteristic of an existing user conversation, where the current conversations are made available to the subsequent stages .

[0169] A conversation context summarization is performed in stage A. 5. 2, where the current user conversation as a whole is summarized using an LLM.

[0170] Subsequently, in stage A. 5. 3, the further first retrieval data store (conversation : conversation) serves mainly as the foundational data repository. In particular, the further first retrieval data store utilizes a primary key composed of conversation ID and question ID, which facilitates the retrieval of supplementary information such as scores, answers, context, and more .

[0171] Subsequently, the process parallelly moves on to two operations A. 5. 4 and A. 5. 5 being conducted on the further first retrieval data store . Method stage A. 5. 4 is characteristic of a semantic similarity search, where the summarized conversations are matched using semantic P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0172] - 29 -

[0173] similarity, and method stage A. 5. 5, where the summarized conversations are matched with historic conversations .

[0174] To enhance conversation matching, additional features can be incorporated beyond merely using responses from the database 4. In particular, domain or application-specific features can be utilized if the application aims to comprehend previously encountered user and / or customer issues and their corresponding solutions . Rather than directly comparing with the conversation, a problem encountered can be provided, along with detailed domain-specific information such as model numbers for hardware equipment, accompanied by a summary of the solution to make a better conversation matching.

[0175] Similarly, multiple approaches can be considered for improving this matching mechanism.

[0176] Subsequently, in stage A. 5. 6 a retrieved matching conversation and a corresponding retrieved matching conversation answer are determined, similar as in stage A. 2. 5 described in Figure 2 .

[0177] Stages A. 5. 7 to A. 5. 10 are similar to stages A. 2. 2 to A. 2. 5. In particular, a further second retrieval data store module is configured in stage A. 5. 7 to determine a further retrieved matching question and a further corresponding retrieved matching answer as an output in stage A. 5. 10.

[0178] In stage A. 5. 11, an intersection match is determined, where intersection between retrieved matching questions and retrieved matching conversations is performed. For example, if five conversations are matched and five questions are matched independently, in method stage A. 5. 11 intersection between those is checked, which independent matched questions P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0179] are part of the independent matched conversations, only the intersection is passed down to next method stage A. 5. 12.

[0180] If an intersection match is determined, method stage A. 5. 12 is performed, where conversations are selected with respective historical questions with a confidence class from the lookup score generator marked as positive . Only selected conversations with a positive class are passed to the subsequent method stage A. 5. 13.

[0181] If no intersection match is determined, no conversation matched output is passed to method stage A. 2.

[0182] The remaining method stages in Figure 3 correspond to the method stages as already described in connection with Figure 2. Particularly, in stage A. 5. 13 it is determined if any conversation is present similar to stage A. 2. 8 in Figure 2. In stage A. 5. 14, further retrieved user feedback scores and outputs are determined, similar to stage A. 2. 9 in Figure 2. In stage A. 5. 15, the further hypothetical answer creation module is configured to generate a further hypothetical answer by a fifth LLM and is further configured to determine a further relevance between the further hypothetical answer and a further retrieved answer in stage A. 5. 16, similar to stages A. 2. 10 and A. 2. 11 in Figure 2, respectively.

[0183] A further hypothetical hallucination score is generated in stage A. 5. 18 dependent on the further hypothetical answer and the further corresponding retrieved matching answer dependent on a conversation threshold of method stage A. 5. 17, similar to stages A. 2. 13 and A. 2. 12 in Figure 2, respectively. P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0184] - 31 -

[0185] A further hypothetical document embeddings module in stage A. 5. 19 is configured to determine a further relevant context and a further relevant score in stage A. 5. 20 dependent on the input information 2 and dependent on a fourth LLM, similar to stages A. 2. 14 and A. 2. 15 in Figure 2, respectively.

[0186] A further individual score normalization and a further scaling is performed in stage A. 5. 21 on the further score of the further relevant context and / or the further hypothetical hallucination score, similar to stage A. 2. 16 in Figure 2.

[0187] A further additive weighted scoring in method stage A. 5. 23 is performed on the further score of the further relevant context and / or the further hypothetical hallucination score on which the further individual score normalization and the further scaling is performed, dependent on the further retrieved user feedback score A. 5. 14 and / or further business defined weights per score A. 5. 22, similar to stage A. 2. 18 -with the relating stages - in Figure 2 . The further additive weighted scoring is slightly different from the question matching mechanism in the sense that the equation which is created is for matched questions within the conversation.

[0188] A further final score normalization and further final scaling is performed question-wise in stage A. 5. 24, similar to stage A. 2. 19 in Figure 2. The only addition to this method stage A. 5. 24 with respect to stage A. 2. 19 is that the further final scores are matched based on the matched questions .

[0189] Subsequently, in stage A. 5. 25 ranked retrieved conversations and answer pairs are generated for the output of stage

[0190] A. 5. 24, wherein the ranked retrieved conversations and answer P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0191] - 32 -

[0192] pairs are characteristic of the second response information in stage A. 6, similar to stage A. 2. 20 in Figure 2.

[0193] According to Figure 4, multiple inputs are provided from the database 4, as well as the corresponding response information, to method stage C . l . The inputs are, inter alia, the input information stored in the database, the first, the second and the third response, e . g. including a respective answer, exemplarily generated by a LLM, and retrieved context including all input passed to the LLM as well as intermediate values .

[0194] In method stage C .1.1, a fact checking score is generated, in order to get an output which states if the generated answer has the correct fact or not .

[0195] The fact checking score generation makes use of any fact checking mechanism which validates if all relevant facts are present or not in the answer generated with respect to the user question.

[0196] The generation of the fact checking score can include four steps :

[0197] A query generation: The third response information is deconstructed into several verification queries, configured to probe for the veracity of each fact presented in the output .

[0198] A retrieval : Relevant information is then fetched for each individual query to ensure accurate context . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0199] - 33 -

[0200] An agreement : An advanced language model serves as an adjudicator, determining consensus between the original statement by the LLM and the context retrieved for factual accuracy .

[0201] A consolidation: In cases where a factual inconsistency is detected, the answer is withheld from the user and assigned a score of 0. Conversely, if no such discrepancies are found, a score of 1 is given and the answer is considered valid.

[0202] A decision block in stage C .1.2 checks if a correct fact is present, e . g. where the presence of the facts in the answer is validated. If the correct fact is present, a hallucination score is generated in stage C .1.3.

[0203] Based on various inputs and outputs available for a questionanswer pair, a score is generated to represent the amount of hallucination given by the LLM in the generated answer in stage C .1.3. The generation of the hallucination score can include three steps :

[0204] A hypothetical answer generation: this approach takes the user ' s question as input and uses an LLM to produce a hypothetical document embedding. From this embedding, the LLM generates a prospective answer .

[0205] A multi-iteration hypothetical answer production: In this stage, the previously mentioned method of hypothetical answer generation is repeated multiple times to create an array of potential answers .

[0206] A self-check-GPT : Serving as a black-box method for detecting hallucinations, this stage involves feeding the multiple P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0207] - 34 -

[0208] hypothetical answers into the system to generate a hallucination score .

[0209] A decision block in stage C .1.4 checks if the hallucination score is smaller than a first preset threshold. If the hallucination score is smaller than the first preset threshold, a retrieval relevance score is generated in stage C .1.5.

[0210] The generation of the retrieval relevance score can include two steps :

[0211] A hypothetical document embedding: An LLM is employed to create an assumed document embedding that will be directly used in the following step .

[0212] A retrieval relevance score : The score is generated, considering the user ' s question, the hypothetical document embedding or context, and the LLM' s response . Tools such as RAGAS, DeepEval, Langchain bench, among others, can generate this metric .

[0213] A decision block in stage C .1. 6 checks if the retrieval relevance score is smaller than a second preset threshold. If the retrieval relevance score is smaller than the second preset threshold, the aggregated score is generated in stage C .1.8, which comprises weightage scores defined for the fact checking score, the hallucination score and the retrieval relevance score, and an LLM-generated answer is generated in stage C . 1 . 9 . P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0214] - 35 -

[0215] If one of the decision blocks is negative, a default message in stage C .1.7, e . g. "reach out to tech support", is generated and provided to the user in stage C .2.

[0216] According to Figure 5, multiple inputs are provided from method stage C . l, from C .2 as well as from method stage C .3 to the lookup score generator C .4. The user feedback of stage C .3 includes, for example, a current answer feedback, a conversation feedback and / or a citation feedback.

[0217] The output to the user in stage C .2 is provided to a decision block in stage C .4.1, where each of the answers given to the user is passed to the decision block if the output is LLM generated or a default message . In particular, in stage C .4.1 the output is checked for the value i . e . , if the output is the default message or the answer from the LLM.

[0218] If the output comprises the default message, in method step C .4.2, no score generation is triggered, where no score is generated for this question and all relevant information is stored in the database 4 .

[0219] If the output comprises the LLM answer, in method step C .4.3 a decision block is triggered, checking if the user feedback is provided. If yes, a final individual score normalization and a final scaling is triggered in stage C .4.5. If no, a module user-wise historic average is performed in stage C .4.4, where an average of the earlier feedbacks given by the same user separately for an answer, conversation as well as citation would be considered, and if it is above a specific threshold, wherein the threshold is based on the targeted application, an average is estimated for each of the feedbacks . 92024, 0927 WO L / P24-1594WO01 January 3, 2025

[0220] - 36 -

[0221] The final individual score normalization and the final scaling in stage C .4.5 is performed based on the targeted application. The final scaling is beneficial because user feedback can vary across multiple scales depending on the targeted application as well as the scores coming from the LLM answer check mechanism, which are typically also not normalized. Therefore, normalizing and converting them to a uniform scale is advantageously performed.

[0222] Subsequently, in method stage C .4.7, a final additive weighted scoring is performed on the aggregated score on which the final individual score normalization and the final scaling is performed, dependent on business defined weights per score . In particular, the final scores from method stage C .4.5 are added based on the inputs from the stage C .4. 6, where business defined weights per score are provided, to get a single score .

[0223] In a next method stage C .4.8, a final score scaling is performed since the scores from the previous stages are cumulative and can exceed bounds, particularly for determining the confidence score . A universal scaling technique is applied, setting maximum and minimum values of the previously generated scores . Subsequently, min-max normalization is performed to adjust the values to a range of 0-1, which is then multiplied by 5 to bring the final scores within a range of 0-5.

[0224] Subsequently, in stage C .4.9 the confidence class is determined, e . g. comprising at least 3 classes : absolute correct, absolute wrong, answer correct but user feedback wrong. This means that the confidence class classifies the P2024, 0927 WO L / P24-1594WO01 January 3, 2025

[0225] - 37 -

[0226] confidence score into the 3 classes . The confidence score and confidence class are stored in the database 4. P2024 , 0927 WO L / P24- 1594WO01 January 3 , 2025

[0227] 38

[0228] List of references

[0229] 1 system

[0230] 2 input information

[0231] 3 further input information 4 database

[0232] A. 1 . . C . 5 method stages

Claims

P2024, 0927 WO L / P24-1594WO01 January 3, 2025Claims1 . Method for generating an output information with a confidence information in response to an input information (2 ) , the method comprising:receiving, by a processing unit, the input information (2 ) characteristic of a user query,determining, by the processing unit, a first response information (A. 3) based on a stand-alone question lookup mechanism (A. 2 ) dependent on the input information (2 ) , determining, by the processing unit, a second response information (A. 6) based on a conversation lookup mechanism (A. 5) dependent on the input information (2 ) , generating, by the processing unit, a third response information dependent on the input information (2 ) based on a retrieval-augmented generation, RAG, mechanism (B) , generating, by the processing unit, the output information and a corresponding aggregated score dependent on the first response information, the second response information and / or the third response information as well as dependent on historical data stored in a database (4 ) based on a large language model, LLM, answer check mechanism (C . l ) ,generating, by the processing unit, the confidence information of the corresponding output information based on a lookup score generator (C .4 ) and storing the confidence information in the database (4 ) .

2. Method according to claim 1, wherein the stand-alone question lookup mechanism (A. 2 ) comprises :providing the input information (2 ) to a retrieval data store module (A.

2. 2 ) , a hypothetical document embeddingsP2024, 0927 WO L / P24-1594WO01 January 3, 2025- 40 -module (A.

2. 14 ) and a hypothetical answer creation module (A.

2. 10) , whereinthe retrieval data store module (A.

2. 2 ) is configured to determine a retrieved matching question and a corresponding retrieved matching answer (A.

2. 5) dependent on the input information (2 ) ,the hypothetical document embeddings module (A.

2. 14 ) is configured to determine a relevant context and a relevant score (A.

2. 15) dependent on the input information (2 ) dependent on a second LLM, andthe hypothetical answer creation module (A.

2. 10) is configured to generate a hypothetical answer by a third LLM and further configured to determine a relevance score between the hypothetical answer and a retrieved answer (A.

2. 11 ) , which is characteristic of the corresponding retrieved matching answer .

3. Method according to claim 2, whereinthe hypothetical answer and the retrieved answer relevance (A.

2. 11 ) is determined dependent on a retrieved user feedback score (A.

2. 9) , and the retrieved user feedback score (A.

2. 9) is dependent on the retrieved matching question and a corresponding retrieved matching answer (A.

2. 5) .

4. Method according to claims 2 or 3, whereina hypothetical hallucination score (A.

2. 13) is generated dependent on the hypothetical answer and the corresponding retrieved matching answer (A.

2. 11 ) dependent on a question threshold (A.

2. 12 ) ,an individual score normalization and a scaling (A.

2. 16) are performed on the score of the relevant context (A. 2.1 ) and / or the hypothetical hallucination score (A.

2. 13) , andP2024, 0927 WO L / P24-1594WO01 January 3, 2025an additive weighted scoring (A.

2. 18 ) is performed on the score of the relevant context (A.

2. 15) and / or the hypothetical hallucination score (A.

2. 13) on which the individual score normalization and the scaling (A.

2. 16) is performed, dependent on the retrieved user feedback score (A.

2. 9) and / or business defined weights per score(A.2 . 17 ) .

5. Method according to claim 4, whereinranked retrieved question and answer pairs (A.

2. 20) are generated dependent on the score of the relevant context (A.

2. 15) and / or the hypothetical hallucination score (A.

2. 13) on which the additive weighted scoring (A.

2. 18 ) is performed, whereinthe ranked retrieved question and answer pairs (A.

2. 20) are characteristic of the first response information (A. 3) .

6. Method according to any one of claims 1 to 5, wherein the conversation lookup mechanism (A. 5) comprises :providing the input information (2 ) to a further first retrieval data store module (A.

5. 3) , a further second retrieval data store module (A.

5. 7 ) , a further hypothetical document embeddings module (A.

5. 19) and a further hypothetical answer creation module (A.

5. 15) , whereinthe further first retrieval data store module (A.

5. 3) is configured to determine a retrieved matching conversation and a corresponding retrieved matching conversation answer (A.

5. 6) dependent on the input information (2 ) ,the further second retrieval data store module (A.

5. 7 ) is configured to determine a further retrieved matchingP2024, 0927 WO L / P24-1594WO01 January 3, 2025- 42 -question and a further corresponding retrieved matching answer (A.

5. 10) dependent on the input information (2 ) , the further hypothetical document embeddings module(A.

5. 19) is configured to determine a further relevant context and a further relevant score (A.

5. 20) dependent on the input information (2 ) and dependent on a fourth LLM, andthe further hypothetical answer creation module (A.

5. 15) is configured to generate a further hypothetical answer by a fifth LLM and is further configured to determine a further relevance between the further hypothetical answer and a further retrieved answer (A.

5. 16) , which is characteristic of the corresponding retrieved matching conversation answer having an intersection match with the further corresponding retrieved matching answer (A.

5. 11 ) .

7. Method according to claim 6, whereinthe further hypothetical answer and the further retrieved answer relevance (A.

5. 16) is determined dependent on a further retrieved user feedback score (A.

5. 14 ) , and the further retrieved user feedback score (A.

5. 14 ) is dependent on the intersection match (A. 5.1 ) .

8. Method according to claims 6 or 7, whereina further hypothetical hallucination score (A.

5. 18 ) is generated dependent on the further hypothetical answer and the further corresponding retrieved matching answer(A.

5. 16) dependent on a conversation threshold (A.

5. 17 ) , a further individual score normalization and a further scaling (A.

5. 21 ) are performed on the further score of the further relevant context (A.

5. 20) and / or the further hypothetical hallucination score (A.

5. 18 ) , andP2024 , 0927 WO L / P24- 1594WO01 January 3 , 2025a further additive weighted scoring (A. 5 . 23 ) is performed on the further score of the further relevant context (A. 5 . 20 ) and / or the further hypothetical hallucination score (A. 5 . 18 ) on which the further individual score normali zation and the further scaling (A. 5 . 21 ) is performed, dependent on the further retrieved user feedback score (A. 5 . 14 ) and / or further business defined weights per score (A. 5 . 22 ) .9 . Method according to claim 8 , whereinranked retrieved conversation and answer pairs (A. 5 . 25 ) are generated dependent on the further score of the further relevant context (A. 5 . 20 ) and / or the further hypothetical hallucination score (A. 5 . 18 ) on which the further additive weighted scoring (A. 5 . 23 ) is performed, whereinthe ranked retrieved conversation and answer pairs(A. 5 . 25 ) are characteristic of the second response information (A. 6 ) .10 . Method according to any one of claims 1 to 9 , wherein the LLM answer check mechanism ( C . l ) comprises :providing the input information ( 2 ) , the first response , the second response and / or the third response to the LLM answer check mechanism ( C . l ) , andgenerating a fact checking score ( C . 1 . 1 ) and, when the corresponding response has a correct fact ( C . l . 2 ) , generating a hallucination score ( C . l . 3 ) and, when the hallucination score is smaller than a first preset threshold ( C . l . 4 ) ,generating a retrieval relevance score ( C . l . 5 ) and, when the retrieval relevance score is smaller than a second preset threshold ( C . l . 6 ) ,P2024, 0927 WO L / P24-1594WO01 January 3, 2025- 44 -generating the aggregated score (C .1.8 ) , which comprises weightage scores defined for the fact checking score, the hallucination score and the retrieval relevance score, and generating an LLM generated answer (C .1.9) characteristic of the output information based on the first response, the second response and / or the third response .

11. Method according to any one of claims 1 to 10, wherein a user feedback (C .3) is provided dependent on the output information comprising at least one of a current answer feedback, a conversation feedback and / or a citation feedback .

12. Method according to any one of claims 1 to 11, wherein the lookup score generator (C .4 ) comprises :providing the output information to a first decision block (C .4.1 ) , and providing the user feedback to a second decision block (C .

4. 3) , whereinthe first decision block determines if the output information is LLM generated or a default message, the second decision block determines if the user feedback is present, andfinal individual score normalization and final scaling are performed on the aggregated score (C .

4. 5) .

13. Method according to claim 12, whereina final additive weighted scoring (C .

4. 7 ) is performed on the aggregated score on which the final individual score normalization and the final scaling are performed, dependent on business defined weights per score .

14. Method according to claim 13, whereinP2024, 0927 WO L / P24-1594WO01 January 3, 2025- 45 -the confidence information is determined based on the aggregated score on which the final additive weighted scoring is performed, andthe confidence information comprises a confidence score and / or a confidence class .

15. System, which is configured to perform the method according to any one of claims 1 to 14.

Images