Method and system for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit

Abstract

The invention relates to enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit. To this end a splitting operation (1) splits an input document (ID) into multiple chunks (C), wherein each chunk fits into a token limit of a language model (LM). A computing operation (3) computes an essential information score (EIS) for each token of each chunk, wherein for each chunk, the tokens of that chunk are provided as input to the language model and output is received from the language model that is used for computing the essential information scores. A selecting operation (4) selects for each chunk informative tokens (IT) based on the highest essential information scores. An appending operation (5) appends the informative tokens to a prompt containing a question and / or a task description, and sends the prompt to a large language model (LLM). A receiving operation (6) receives a response (O) from the large language model. At least some of the embodiments are fully LLM-based and limit the loss of context by substantially increasing any off-the-shelve LLM context length, in spite of the LLM's token limit. This has the potential to significantly increase the number of industrial applications of LLMs, as well as to improve the efficiency and performance of LLM integration into industrial applications such as engineering assistance.

Description

[0001] 202404684 foreign version

[0002] 1

[0003] Description

[0004] Method and system for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit

[0005] Technical Field

[0006] This invention relates to enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit. The invention further relates to reducing computational and memory resources required for processing such an input document under a fixed token-budget constraint of a transformer-based language model.

[0007] Background Art

[0008] The token limit in the context of large language models (LLMs) refers to a maximum number of tokens that the model can process in a single input sequence. This limit is determined by the architecture and design of the specific model. For example, one of the largest publicly available LLMs - GPT-4 - has a token limit of 32.000 tokens.

[0009] Documents typically found in industry (e.g., product specifications, documentation) can contain hundreds of pages, i.e., millions of tokens. As a direct consequence, even the largest state-of-the-art LLMs are incapable of processing these documents, and therefore, assisting a user with a task that requires the context of the entire document. For example, a typical Bill-of-Materials (BoM) contains thousands of components, each possessing hundreds of technical attributes. The simple task of retrieving all components of a specified type cannot be easily solved with the help of LLMs simply because the textual representation of the BoM would not fit within the token limit. The same holds true for Piping and Instrumentation Diagrams (P& IDs), which are commonly serialized in the form of an XML document and may contain hundreds of interconnected equipment components and control devices.

[0010] In industrial documents, long identifiers, structured tags, and repeated syntax elements can lead to substantial tokenization overhead, which exacerbates the context-window constraint in computer-implemented processing. The token limit constitutes a technical constraint of the computer-implemented inference pipeline. In particular, when an input to a transformer-based language model exceeds the model’s context window, the input cannot be processed as a single sequence and the computer system is forced to apply intermediate processing that202404684 foreign version

[0011] 2

[0012] reduces the input size before a final inference call. Such intermediate processing directly affects technical resource usage, including memory consumption and processing time during inference, and may also affect data transfer volume when the language model is accessed via a network interface.

[0013] On the other hand, traditional retrieval approaches without the use of LLMs are limited in semantics and reasoning. They can only perform keyword- or vector-search. Combined approaches are called Retrieval-augmented generation (RAG), where keyword or vector search finds relevant document snippets first and feeds these into the LLM for final question answering. This can to some degree alleviate these problems, but the retrieval mechanism remains a significant bottleneck.

[0014] Summary of Invention

[0015] It is an object of the present invention to identify a problem in the prior art and to find a corresponding technical solution.

[0016] The invention is defined in the independent claims. Further advantageous arrangements and embodiments of the invention are set forth in the respective dependent claims.

[0017] According to the method for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit, the following operations are performed by components, wherein the components are hardware components and / or software components executed by one or more processors:

[0018] splitting an input document into multiple chunks, wherein each chunk fits into a token limit of a language model,

[0019] computing an essential information score for each token of each chunk, wherein for each chunk, the tokens of that chunk are provided as input to the language model and output is received from the language model that is used for computing the essential information scores,

[0020] selecting, for each chunk, informative tokens based on the highest essential information scores,

[0021] appending the informative tokens to a prompt containing a question and / or a task description, and sending the prompt to a large language model, and

[0022] receiving a response from the large language model.202404684 foreign version

[0023] 3

[0024] The system for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit, comprises a processor configured for:

[0025] splitting an input document into multiple chunks, wherein each chunk fits into a token limit of a language model,

[0026] computing an essential information score for each token of each chunk, wherein for each chunk, the processor provides the tokens of that chunk as input to the language model and receives output from the language model that is used for computing the essential information scores,

[0027] selecting, for each chunk, informative tokens based on the highest essential information scores,

[0028] appending the informative tokens to a prompt containing a question and / or a task description, and sending the prompt to a large language model, and

[0029] receiving a response from the large language model.

[0030] The following advantages and explanations are not necessarily the result of the object of the independent claims. Rather, they may be advantages and explanations that only apply to certain embodiments or variants.

[0031] The term "computer" should be interpreted as broadly as possible, in particular to cover all electronic devices with data processing properties. Computers can thus, for example, be personal computers, servers, clients, programmable logic controllers (PLCs), handheld computer systems, pocket PC devices, mobile radio devices, smartphones, or any other communication devices that can process data with computer support, for example processors or other electronic devices for data processing. Computers can in particular comprise one or more processors and memory units.

[0032] In at least some embodiments, the selection of informative tokens enables construction of a final prompt that satisfies the token limit of the large language model while preserving a bounded amount of information from each chunk. This provides a technical effect of enabling execution of a computer-implemented question-answering request under a fixed context window constraint, and of reducing the amount of data processed in the final inference call compared to forwarding the entire input document.

[0033] The method and system, or at least some of their embodiments, enable LLM-based inference on a long input document by decomposing the tokens of the input document with minimal loss202404684 foreign version

[0034] 4

[0035] of context. The method and system, or at least some of their embodiments, have the potential to significantly increase the number of industrial applications of LLMs, as well as to improve the efficiency and performance of LLM integration into industrial applications such as engineering assistance.

[0036] The method and system, or at least some of their embodiments, are fully LLM-based and limit the loss of context by substantially increasing any off-the-shelve LLM context length, in spite of the LLM's token limit.

[0037] The method and system, or at least some of their embodiments, are advantageous compared to a mere summarization of the input document, as summarization may overlook important details or nuances in the text.

[0038] The method and system, or at least some of their embodiments, are also advantageous compared to manual selection of relevant parts of the input document, as it requires human intervention and a prior understanding of the document contents.

[0039] Description of Embodiments

[0040] In an embodiment of the method and system, the computing operation is characterized by the input being passed through a decoder-only transformer part of the language model, the output from the language model providing a prediction probability for each token or enabling computation of a prediction probability for each token, and

[0041] computing the highest essential information scores for the tokens with the lowest prediction probability.

[0042] In an embodiment of the method and system, the output from the language model is a nexttoken probability distribution for each token. The prediction probability of each token is extracted from the next-token probability distribution for the previous token in the respective chunk for the respective token.

[0043] In an embodiment of the method and system, when computing the essential information score for each token, not only the prediction probability of the token is considered, but also a position of the token within the respective chunk.

[0044] In an embodiment of the method and system, the computing operation is characterized by202404684 foreign version

[0045] 5

[0046] for each chunk, passing the tokens of that chunk as input through an encoder-only transformer part of the language model, and receiving as the output from the language model an embedding of the chunk, and

[0047] for each token of the chunk,

[0048] - masking the token to form a masked chunk and providing the masked chunk as input to the encoder-only transformer part of the language model, and receiving as the output from the language model an embedding of the masked chunk,

[0049] - computing the essential information score for the token proportional to a shift between the embedding of the chunk and the embedding of the masked chunk.

[0050] In an embodiment of the method and system, the language model is

[0051] an encoder-decoder model, in particular a T5 large language model,

[0052] an encoder-only model, in particular a BERT model or a Sentence-BERT model, or a domain-specific language model.

[0053] In an embodiment of the method and system, the language model is the large language model or a different language model.

[0054] In an embodiment of the method and system, the selecting operation is adjusted or repeated until a token limit of the large language model is reached.

[0055] An embodiment of the method and system comprises the additional operation of outputting, by a user interface, the response, and / or feeding the response into an Al module for further processing.

[0056] An embodiment of the method and system comprises the additional operation of tokenizing, by a tokenizer, before or after the splitting operation each chunk to determine its tokens.

[0057] In an embodiment of the method and system, the tokenizer is a custom tokenizer that ensures that semantics are matched to tokens, in particular, by assigning each identifier of industrial equipment to a single token.

[0058] In an embodiment of the method and system, the input document is a single document or a set of documents.202404684 foreign version

[0059] 6

[0060] The computer program product comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method.

[0061] The provisioning device for the computer program product stores and / or provides the computer program product.

[0062] Brief Description of the Drawing

[0063] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, the drawings show embodiments that are presently preferred.

[0064] However, the invention is not limited to the specific instrumentalities disclosed. The embodiments may be combined with each other. Furthermore, the embodiments may be combined with any of the features described above. Unless stated otherwise, identical reference signs denote the same features or functionally identical elements between drawings. Included in the drawings are the following Figures:

[0065] Fig. 1 shows a first embodiment,

[0066] Fig. 2 shows another embodiment,

[0067] Fig. 3 shows a flowchart of a possible exemplary embodiment,

[0068] Fig. 4 shows an embodiment using a decoder-only transformer DOT, and

[0069] Fig. 5 shows another embodiment using a decoder-only transformer DOT.

[0070] In the following description, various aspects of the present invention and embodiments thereof will be described. However, it will be understood by those skilled in the art that embodiments may be practiced with only some or all aspects thereof. For purposes of explanation, specific numbers and configurations are set forth in order to provide a thorough understanding.

[0071] However, it will also be apparent to those skilled in the art that the embodiments may be practiced without these specific details.

[0072] The described components can each be hardware components or software components. For example, a software component can be a software module such as a software library; an202404684 foreign version

[0073] 7

[0074] individual procedure, subroutine, or function; or, depending on the programming paradigm, any other portion of software code that implements the function of the software component. A combination of hardware components and software components can occur, in particular, if some of the effects according to the invention are preferably exclusively implemented by special hardware (e.g., a processor in the form of an ASIC or FPGA) and some other part by software.

[0075] Devlin, Jacob, et al.: " BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", 2018, arXiv:1810.04805v2 [cs.CL], available on the internet at https: / / doi.org / 10.48550 / arXiv.1810.04805 on 26.02.2025, disclose the BERT language model. The entire contents of that document are incorporated herein by reference.

[0076] Reimers, Nils, and Iryna Gurevych: " Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", EMNLP 2019, arXiv:1908.10084v1, available on the internet at https: / / doi.org / 10.48550 / arXiv.1908.10084 on 26.02.2025, disclose the Sentence-BERT language model. The entire contents of that document are incorporated herein by reference.

[0077] Yao, Wenlin, et al.: " How do Words Contribute to Sentence Semantics? Revisiting Sentence Embeddings with a Perturbation Method", Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, disclose an algorithm to determine how words contribute to sentence semantics. The entire contents of that document are incorporated herein by reference.

[0078] Description of Examples

[0079] Fig. 1 shows a sample structure for computer-implementation of the invention which comprises:

[0080] (101) computer system

[0081] (102) processor

[0082] (103) memory

[0083] (104) computer program (product)

[0084] (105) user interface

[0085] In this embodiment of the invention the computer program 104 comprises program instructions for carrying out the invention. The computer program 104 is stored in the memory 103 of the computer system 101. The computer system 101 may carry out the invention by executing the program instructions of the computer program 104 by the processor 102. Results of the202404684 foreign version

[0086] 8

[0087] invention may be presented on the user interface 105. Alternatively, they may be stored in the memory 103 or on another suitable means for storing data.

[0088] Fig. 2 shows another sample structure for computer-implementation of the invention which comprises:

[0089] (201) provisioning device

[0090] (202) computer program (product)

[0091] (203) computer network / lnternet

[0092] (204) computer system

[0093] (205) mobile device / smartphone

[0094] In this embodiment the provisioning device 201 stores a computer program 202 which comprises program instructions for carrying out the invention. The provisioning device 201 provides the computer program 202 via a computer network / Internet 203. By way of example, a computer system 204 or a mobile device / smartphone 205 may load the computer program 202 and carry out the invention by executing the program instructions of the computer program 202.

[0095] In a variation of this embodiment, the provisioning device 201 is a computer-readable storage medium, for example a SD card, that stores the computer program 202 and is connected directly to the computer system 204 or the mobile device / smartphone 205 in order for it to load the computer program 202 and carry out the invention by executing the program instructions of the computer program 202.

[0096] Preferably, the embodiments shown in Figs. 3 to 5 can be implemented with a structure as shown in Fig. 1 or Fig. 2.

[0097] The embodiments described in the following provide novel inference capabilities for LLMs that allow processing documents that are too large to fit within the inherent token limit.

[0098] It is widely known from the state of the art that the transformer architecture originally consists of an encoder and a decoder. Due to its high potential, many variations of this architecture are in use today, including encoder-only transformer architectures, decoder-only transformer architectures, and encoder-decoder-transformer architectures. Some large language models such as those from the GPT family have a decoder-only transformer architecture, in other words, they only use the decoder part of the original transformer architecture.202404684 foreign version

[0099] 9

[0100] Fig. 3 shows the basic steps of the embodiments described in the following. The setting in which these embodiments operate assumes that a user prompts a large language model LLM with a question (wherein the question can also be or include a task) along with an input document ID containing necessary context for answering the question. An example could be the user asking the large language model LLM to identify all valves in a provided Process and Instrumentation Diagram (P& ID) as input document ID. The input document ID has a size that does not fit within the token limit of the large language model LLM. The input document ID can also be a set of multiple documents that, alone or together, do not fit within the token limit of the large language model LLM.

[0101] In a splitting operation 1, the input document ID is split into multiple chunks C (independent pieces) which are small enough to fit into a token limit of a language model LM. As will be described below with regard to various embodiments, the language model LM can either be the large language model LLM itself or a different language model. Preferably, the splitting logic of the splitting operation 1 is use-case-specific. For example, in the above-mentioned P& ID use case, a preferable embodiment can ensure to only split on completed XML tags, and not inbetween words.

[0102] After the splitting operation 1, each chunk C is tokenized by a tokenizer in a tokenizing operation 2. In a variation of the current embodiment, the tokenizing operation 2 can be executed before the splitting operation.

[0103] The vocabulary of current (large) language models often uses chunked words as tokens (e.g., tokens “cur” and “ious” for the word “curious”). However, this kind of tokenization can create problems when the input document ID contains complex and long identifiers for technical equipment, as is common in the industrial field. If a long identifier is broken down into many tokens, then the described embodiments could be hindered in identifying the most informative tokens, as the semantics of each token are not clear. Preferably, the tokenizing operation 2 is performed by a custom tokenizer that ensures that semantics are matched to tokens, for example, by assigning each long identifier to a single token. In case a conventional tokenizer is used, at least some of the embodiments can treat tokens belonging to the same semantic unit (e.g., a long identifier) as a group, as will be described below.

[0104] In a computing operation 3, a processor computes an essential information score EIS for each token of each chunk C, wherein for each chunk C, the processor provides the tokens of that202404684 foreign version

[0105] 10

[0106] chunk C as input to the language model LM and receives output from the language model LM. In other words, the tokens of each chunk C are passed through the language model LM. If the language model LM is not executed locally, then communication with the language model LM is performed by a network interface. Based on the output of the language model LM, the processor computes an essential information score EIS for each token of each chunk. This serves the aim of identifying tokens with the highest essential information score EIS.

[0107] In a selecting operation 4, for each chunk C, informative tokens IT are selected based on the highest essential information scores EIS calculated in the computing operation 3. These informative tokens IT contain the “essence” of the respective chunks C.

[0108] It is possible to control how much contextual information from the input document ID is inadvertently lost by increasing the number of informative tokens IT in the selecting operation 4. For example, the count of informative tokens IT can be increased until the token limit of the large language model LLM is reached, while considering an amount of tokens already consumed by the question itself.

[0109] In an appending operation 5, the informative tokens IT are appended to the user’s question (or task) and sent to the large language model LLM as a prompt. If the large language model LLM is not executed locally, then sending is performed by a network interface. Only keeping the informative tokens IT (as opposed to all tokens necessary for representing the chunks C) ensures that the final input fits within the token limit of the large language model LLM.

[0110] From a computer-implementation perspective, limiting the final prompt to the informative tokens reduces the number of tokens processed by the large language model in the final inference step. This can reduce memory usage and processing time associated with the final inference call. When the large language model is provided as a remote service, limiting the final prompt may also reduce the amount of data transmitted over a computer network to the service.

[0111] In a receiving operation 6, a response O is received from the large language model LLM. If the large language model LLM is not executed locally, then receiving is performed by a network interface.

[0112] For example, the received response O can be output by a user interface to a user or fed into an Al module for further processing.202404684 foreign version

[0113] 11

[0114] In an embodiment, the essential information score EIS indicates how difficult it is for the language model LM to predict the respective token. According to this embodiment, a high essential information score EIS indicates that the respective token contains essential information that cannot be derived from the rest of the tokens. If the respective token is hardest to predict, then it has the lowest prediction probability. The processor assigns the highest essential information scores EIS to the tokens with the lowest prediction probability.

[0115] As mentioned above, the language model LM can either be the large language model LLM itself or a different language model, including a different large language model. In case different models are used, the same tokenizer needs to be used with both models. For the current embodiment, the computing operation 3 is implemented by passing the tokens of each chunk C forward through a decoder-only transformer part of the language model LM, which outputs a prediction probability for each token. The language model LM may have a decoder-only transformer architecture or an encoder-decoder-transformer architecture, but in any case, only the decoder-only transformer part of the language model LM is used for these forward passes.

[0116] For example, if the language model LM is the large language model LLM itself and if the input document ID is split into n chunks C, then n+1 forward passes are performed through the large language model LLM: One for each chunk and one for the final prompt after the appending operation 5.

[0117] With regard to the computational costs of these forward passes, a smaller and / or cheaper language model LM may be used for the initial n forward passes for each chunk. In that case, only the final prompt after the appending operation 5 needs to be sent to the large language model LLM itself.

[0118] While the current embodiment requires n+1 forward passes for n chunks C and may lose some context from the input document ID (which can be controlled through an increase in the number of informative tokens IT preserved by the selecting operation 4), it can substantially increase the context length despite the given token limit, enabling use cases that would otherwise not be possible to address with the help of LLMs.

[0119] For example, when provided with an input document ID that represents a P& ID in a standardized format, the current embodiment is expected to only preserve details that are hard to predict given the surrounding context, such as identifiers and non-standard attributes &202404684 foreign version

[0120] 12

[0121] values. Repeated tokens (e.g., related to XML syntax) and easy to predict attributes are dropped.

[0122] With regard to the forward passes of the tokens of each chunk C through the decoder-only transformer part of the language model LM, k tokens of a given chunk C form an input sequence that is processed 1 -by-1 when passed through the decoder-only transformer part. With each token, an internal next-token probability distribution is created (k in total).

[0123] Conventional use of the decoder-only transformer part would only access the final probability distribution (to generate a k+1-th token). The current embodiment, however, accesses the nexttoken probability distribution for each token of the chunk C as a means of verifying whether the next observed token (the next token known to occur in the chunk C) is predictable by the language model LM.

[0124] Weighted Essential Information Score

[0125] The following section describes an advantageous implementation for computing the essential information score for the current embodiment.

[0126] Given a document chunk D = {x1,x2,...xt,

[0127]

[0128] made of n tokens and LLM vocabulary embedding matrix E.

[0129] A last layer token representation xtis used to predict a next token Xt+1(as random variable here) with last layer key, query and value matrices K, Q, V.

[0130] P(Xt+i\xt>xt-i> ■■■)=softma.x(EXt')

[0131]

[0132] If pXt+1= P(Xt+1= xt+^xt.x^^...), i.e. the probability of the observed next token, is high, it means that the preceding tokens contain all information and xt+1does not give new information.

[0133] If P(Xt+1= xt+1\xt, xt-,... ) is low, it means that xt+1adds new information to its predecessors and is therefore a good candidate for conveying essential document information.

[0134] Another consideration is the amount of preceding context that is available to every token. Since xt+1can attend to all prior tokens, the closer t+1 is to the end of the chunk the more information it can potentially contain.202404684 foreign version

[0135] So the advantageous implementation uses a weighted essential information score that considers low token probability and “closeness” to end of chunk:

[0136] E

[0137]

[0138] IscoreXt+i= (1 - pXt+i) * (1 - wpos(n- (t + 1)))

[0139] where wposmay be experimentally chosen.

[0140] More formally, Algorithm A contains pseudo code representing the current embodiment.

[0141] Algorithm A

[0142] 1: Initialize userPrompt and input_document

[0143] 2: if size of input_document > LM’s token limit then

[0144] 3: Split input_document into chunks and store in chunks

[0145] 4: Initialize an empty list informative_tokens to store the essential tokens

[0146] 5: for each chunk in chunks do

[0147] 6: Pass the tokens of chunk through the decoder-only transformer of LM and get an estimated probability for each observed next token, wherein the observed next token is the token following a current token,

[0148] 7: Obtain the k tokens with the highest weighted essential information score computed from the estimated probabilities and add them to informative_tokens

[0149] 8: end for

[0150] 9: Append informative_tokens to userPrompt to form finalinput

[0151] 10: Feed final Input into LLM to get response

[0152] 11: else

[0153] 12: Combine userPrompt and input_document as input and feed into LLM to get response 13: end if

[0154] 14: Return response as the result of the algorithm

[0155] Fig. 4 shows a simplified example. An input document ID consisting of eight tokens is divided into a first chunk C1 and a second chunk C2, four tokens each. Each chunk C1, C2 is fed into a decoder-only transformer DOT to identify informative tokens IT with lowest prediction probability P(x). The identified informative tokens IT are appended to two tokens that represent a user’s question QT to obtain a final response, using a final forward pass through the decoder-only transformer DOT.202404684 foreign version

[0156] 14

[0157] Fig. 5 shows a more specific implementation as compared to Fig. 4. Here, the decoder-only transformer DOT has an architecture as known from the state of the art. Therefore, it represents each input token with query vector Q, a key vector K, and a value vector V in a masked multihead attention module MMHA.

[0158] For example, each token can be encoded to an embedding using an embedding layer. A position encoding layer can encode the position of each token, for example, as a vector that is added to each token embedding to encode sequential information. As a result, each token embedding captures the meaning and position of the respective token. The query vector Q, key vector K, and value vector V are computed from each token embedding.

[0159] Processing in the masked multi-head attention module MMHA is followed by an add & normalize step AN, a feed forward layer FFN, for example a small multilayer perceptron consisting of two fully connected layers with a nonlinear activation function in between, and another add & normalize step AN. To put it simply, the decoder-only transformer DOT computes a prediction probability P(x) for each token. For a more detailed and accurate description of the actual processing and output, the above section Weighted Essential Information Score should be consulted.

[0160] Fig. 5 illustrates that when feeding the informative tokens IT together with the question QT into the large language model, the key vectors K and value vectors V of the informative tokens IT may be used in the multi-head attention module MMHA.

[0161] Returning to Fig. 3, another embodiment is now introduced where the essential information score EIS indicates how much an embedding shifts if the respective token is masked. Here, the language model LM is either the large language model LLM, or another language model, such as BERT, Sentence-BERT, or a domain-specific language model. The language model LM can be an encoder-decoder model, for example a T5 large language model, or an encoder-only model such as BERT or Sentence-BERT.

[0162] An encoder-only part of the language model LM encodes the chunks C and the essential information scores EIS are computed based on a perturbation analysis. When encoding each chunk C, the encoder-only part of the language model LM creates an embedding, which is a vector representing the entire chunk C. By masking individual tokens and hiding them from the language model LM, this embodiment estimates the contribution (importance) of the masked202404684 foreign version

[0163] 15

[0164] information to the final semantics of the chunk C. The underlying assumption is that the semantic importance of a word in a sentence is proportional to how much the new sentence representation drifts from the original sentence representation if that word is masked. The essential information score EIS of that word is given by the shift in the embedding space.

[0165] In other words, the current embodiment identifies and utilizes shifts in text embeddings for identifying the informative tokens IT (with the highest essential information scores EIS) in each chunk C. The informative tokens IT can be seen as a compressed version of the original text which can be processed by the large language model LLM with minimal loss of context.

[0166] The current implementation computes the essential information score EIS for a specific token using the following steps:

[0167] 1. Pass each chunk to the language model LM to obtain an embedding.

[0168] 2. Masking: Replace the token for which the essential information score EIS is to be computed with a mask token to form a masked chunk.

[0169] 3. Prediction: Pass the masked chunk to the language model (LM) to obtain an embedding.

[0170] 4. Comparison: Compare the embeddings generated in steps 1 and 3, for example, by computing their distance in the embedding space. In particular, the distance can be computed using a suitable metric, such as cosine similarity.

[0171] The following algorithm, written in pseudo-code, details the current implementation. In the pseudo-code the term "mask" is used to indicate the action of masking a particular token in a chunk, and the compute_shift function calculates the Euclidean distance between the original and the masked chunk representations:202404684 foreign version

[0172] 16

[0173] Algorithm B

[0174] 1: procedure \

[0175]

[0176] NFERENCE(document,question, LLM, LM,token_limit,n) 2: Initialize chunks to split document into pieces that fit within tokenjimit

[0177] 3: Initialize informative_tokens as an empty list

[0178] 4: for each chunk in chunks do

[0179] 5: encoded_chunk <— er\code(chunk)

[0180] 6: ► encode() is performed by encoder-only part of LM

[0181] 7: Initialize token_Elscores as an empty list

[0182] 8: for each token in chunk do

[0183] 9: masked_chunk <— mask(chunk, token)

[0184] 10: encoded_masked_chunk <— encode(masked_chunk)

[0185] 11: Elscore <— compute_shift(encoded_mas ed_c / 7L / n, encoded_chunk)

[0186] 12: ► compute_shift calculates the Euclidean distance between the original and the masked chunk representations.

[0187] 13: Append (token, Elscore) to token_Elscores

[0188] 14: end for

[0189] 15: Sort token_Elscores in descending order of Elscores

[0190] 16: Select top n tokens from token_Elscores and append to informative_tokens

[0191] 17: end for

[0192] 18: final nput <— apper\d(informative_tokens, question)

[0193] 19: final_response <— LLM_respond(f / na / _ / npt / f)

[0194] 20: return final_response

[0195] 21: end procedure

[0196] In case a group of tokens represents a single semantic unit, for example a long identifier of industrial equipment, then a variant of the current embodiment masks the entire group of tokens and assigns the determined essential information score to the entire group of tokens.

[0197] An example for a chunk C, here a long original sentence, and the resulting compressed version (formed by the informative tokens IT) could be:

[0198] Original Sentence (chunk C):

[0199] " During the manufacturing process of the chemical plant, the raw materials are initially passed through a grinder to ensure they are uniformly sized before being mixed in a large tank with water and a variety of other chemicals under controlled temperature and pressure conditions."202404684 foreign version

[0200] Reduced Sentence (subset of original, consisting of the informative tokens IT):

[0201] " Raw materials passed through grinder, mixed large tank with water other chemicals under controlled conditions."

[0202] In the reduced sentence, the key actions (grinding, mixing) and conditions (water, chemicals, specific temperature, and pressure) are preserved while removing some additional details to make the sentence shorter and more concise.

[0203] An example for a long (DEXPI) XML chunk describing an equipment used within a P& ID and what the resulting compressed version could look like:

[0204] Original XML chunk:

[0205] < Equipment ID=" XMP_5857 ">

[0206] < GenericAt tributes Set=" Properties ">

[0207] < GenericAttribute Name=" Description" Format="string" Value=" Self- operating pressure control" / >

[0208] < GenericAttribute Name=" Subclass " Format="string" Value=" Valve" / > < GenericAttribute Name=" Medium Group" Format="string" Value=" Flammable Gas" / >

[0209] < GenericAttribute Name=" Maximum Operating Temperature" Format="double" Value="50. 00" Units=" °C" / >

[0210] < GenericAttribute Name=" Minimum Operating Temperature" Format="double" Value="-15. 00" Units=" °C" / >

[0211] < GenericAttribute Name=" Pressure at Max. Op. Temp. " Format="double" Value="83. 00" Units="bar" / >

[0212] < GenericAttribute Name=" Sealing Water required" Format="bool" Value="0" / >

[0213] < GenericAttribute Name=" Flow Direction required" Forma t="bool" Value="l" / >

[0214] < / GenericAttributes>

[0215] < / Equipment>

[0216] Compressed version (informative tokens IT):

[0217] XMP 5837, Self-operating pressure valve, Flammable Gas, Operating Temperature -15 to 50, Flow direction required, Sealing Water not required202404684 foreign version

[0218] 18

[0219] Note that in the compressed version, the Boolean attributes are correctly preserved, despite the corresponding values (“0” I “1”) being relatively predictable (50%) given that the modality is known (Format=“bool”).

[0220] Retrieval Augmented Generation

[0221] The previously described embodiments also apply to retrieval augmented generation (RAG) use cases, where a set of retrieved documents is too large to fit into the token limit of the large language model LLM. Finding the Top-n tokens in a set of k retrieved documents can be achieved by finding the n / k tokens for each document individually. This likewise enables the final prompt to comply with the token limit of the large language model under a bounded token budget while reducing the amount of retrieved text forwarded to the large language model for the final inference step, which can reduce computational load and, where applicable, network transmission volume. With regard to the embodiments described above, the set of documents is acting as the input document ID.

[0222] For example, the method can be executed by one or more processors. Examples of processors include a microcontroller or a microprocessor, an Application Specific Integrated Circuit (ASIC), or a neuromorphic microchip, in particular a neuromorphic processor unit. The processor can be part of any kind of computer, including mobile computing devices such as tablet computers, smartphones or laptops, or part of a server in a control room or cloud.

[0223] The above-described method may be implemented via a computer program product including one or more computer-readable storage media having stored thereon instructions executable by one or more processors of a computing system. Execution of the instructions causes the computing system to perform acts corresponding to the operations of the method described above.

[0224] The instructions for implementing processes or methods described herein may be provided on computer-readable storage media or memories, such as a cache, buffer, RAM, FLASH, removable media, hard drive, or other computer readable storage media. Computer readable storage media include various types of volatile and non-volatile storage media. The functions, acts, or tasks illustrated in the figures or described herein may be executed in response to one or more sets of instructions stored in or on computer readable storage media. The functions, acts or tasks may be independent of the particular type of instruction set, storage media, processor or processing strategy and may be performed by software, hardware, integrated202404684 foreign version

[0225] 19

[0226] circuits, firmware, micro code, and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like.

[0227] The invention has been described in detail with reference to embodiments thereof and examples. Variations and modifications may, however, be effected within the spirit and scope of the invention covered by the claims. The phrase “A, B, and / or C” as an alternative expression may provide that one or more of A, B, and C may be used.

[0228] Independent of the grammatical term usage, individuals with male, female, or other gender identities are included within the term.

Claims

202404684 foreign version20Claims1. A computer implemented method for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit, wherein the following operations are performed by components, and wherein the components are hardware components and / or software components executed by one or more processors:splitting (1) an input document (ID) into multiple chunks (C), wherein each chunk fits into a token limit of a language model (LM),computing (3) an essential information score (EIS) for each token of each chunk (C), wherein for each chunk (C), the tokens of that chunk (C) are provided as input to the language model (LM) and output is received from the language model (LM) that is used for computing the essential information scores (EIS),selecting (4), for each chunk (C), informative tokens (IT) based on the highest essential information scores (EIS),appending (5) the informative tokens (IT) to a prompt containing a question and / or a task description, and sending the prompt to a large language model (LLM), andreceiving (6) a response (O) from the large language model (LLM).

2. The method of claim 1, wherein the computing operation (3) is characterized bythe input being passed through a decoder-only transformer (DOT) part of the language model (LM),the output from the language model (LM) providing a prediction probability for each token or enabling computation of a prediction probability for each token, andcomputing the highest essential information scores (EIS) for the tokens with the lowest prediction probability.

3. The method of claim 2, whereinthe output from the language model (LM) is a next-token probability distribution for each token, andthe prediction probability of each token is extracted from the next-token probability distribution for the previous token in the respective chunk (C) for the respective token.

4. The method of claim 3, whereinwhen computing the essential information score for each token, not only the prediction probability of the token is considered, but also a position of the token within the respective chunk.202404684 foreign version215. The method of claim 1, wherein the computing operation (3) is characterized byfor each chunk (C), passing the tokens of that chunk (C) as input through an encoder-only transformer part of the language model (LM), and receiving as the output from the language model (LM) an embedding of the chunk (C), andfor each token of the chunk (C),- masking the token to form a masked chunk and providing the masked chunk as input to the encoder-only transformer part of the language model (LM), and receiving as the output from the language model (LM) an embedding of the masked chunk,- computing the essential information score (EIS) for the token proportional to a shift between the embedding of the chunk (C) and the embedding of the masked chunk.

6. The method of claim 5, wherein the language model (LM) isan encoder-decoder model, in particular a T5 large language model,an encoder-only model, in particular a BERT model or a Sentence-BERT model, or a domain-specific language model.

7. The method according to any of the preceding claims,wherein the language model (LM) is the large language model (LLM) or a different language model.

8. The method according to any of the preceding claims,wherein the selecting operation (4) is adjusted or repeated until a token limit of the large language model (LLM) is reached.

9. The method according to any of the preceding claims, with the additional operation of outputting, by a user interface, the response (O), and / orfeeding the response (O) into an AI module for further processing.

10. The method according to any of the preceding claims, with the additional operation of tokenizing (2), by a tokenizer, before or after the splitting operation (1) each chunk (C) to determine its tokens.

11. The method according to claim 10,wherein the tokenizer is a custom tokenizer that ensures that semantics are matched to tokens, in particular, by assigning each identifier of industrial equipment to a single token.202404684 foreign version2212. The method according to any of the preceding claims,wherein the input document is a single document or a set of documents.

13. A system for enabling a large language model to respond to a question regarding an input document from an industrial domain that exceeds its token limit, comprising a processor configured for:splitting (1) an input document (ID) into multiple chunks (C), wherein each chunk fits into a token limit of a language model (LM),computing (3) an essential information score (EIS) for each token of each chunk (C), wherein for each chunk (C), the processor provides the tokens of that chunk (C) as input to the language model (LM) and receives output from the language model (LM) that is used for computing the essential information scores (EIS),selecting (4), for each chunk (C), informative tokens (IT) based on the highest essential information scores (EIS),appending (5) the informative tokens (IT) to a prompt containing a question and / or a task description, and sending the prompt to a large language model (LLM), andreceiving (6) a response (O) from the large language model (LLM).

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to one of the method claims.

15. A provisioning device for the computer program product according to the preceding claim, wherein the provisioning device stores and / or provides the computer program product.

Images