Cot-based agent-type multimodal LLM provision system

Abstract

The present invention relates to a multimodal large language model (LLM) that integrates and processes modality information, and the present invention comprises: an input module that receives, as an input, modality information comprising any one or more of text information, image information, audio information, and numerical information; an agent module that comprises a large language model (LLM), generates an execution step for stepwise inference in a chain-of-thought (CoT) manner by integrating aligned modality information, and calculates stepwise execution content; a tool management module that manages a tool and metadata for executing the execution content calculated by the agent module; and an execution module that executes the execution content by calling the tool or setting / changing the metadata, analyzes a result of the execution, and provides same to the agent module. According to the present invention as described above, the present invention has the effects of improving the error correction efficiency of an agent-type multimodal LLM and securing stability, by deriving, in each execution step through CoT-based stepwise inference, a stepwise inference result by using an external tool, analyzing an execution result in each step, and performing re-planning if the execution result in each step falls outside a planned range.

Description

COT-based agent-type multimodal LLM provision system

[0001] The present invention relates to a multimodal large language model (LLM) that integrates and processes multiple modality information including text, images, audio, and numerical information, and to an agent-type multimodal LLM providing system in which a Chain-of-Thought (CoT)-based agent performs step-by-step reasoning, autonomously selects and calls external tools, and performs replanning according to the execution results.

[0002]

[0003] Conventional Large Language Models (LLMs) were primarily limited to generating responses based on text-based inputs and had limitations in comprehensively processing various modalities, such as images, audio, and sensor data, required in complex real-world environments.

[0004] In addition, there was a problem in that structures that output simple inference results made it difficult to solve problems involving actual actions, such as external code execution, database access, document viewing, and physical device control.

[0005] Recently, the Chain of Sort (CoT) technique has been proposed, enabling the structuring of step-by-step inference within a model; however, systems that combine this CoT-based inference with actual execution steps and external tool integration to provide an iterative structure of plan, execution, verification, and replan have not yet been sufficiently presented.

[0006]

[0007] The present invention has been devised to solve the aforementioned conventional problems, and aims to provide an agent-type multimodal LLM model that integrates and processes multiple modality information.

[0008] Furthermore, the present invention aims to provide an agent-type multimodal LLM model that generates execution steps through CoT-based step-by-step reasoning and enables the autonomous selection / calling and execution of various tools, such as code execution, database access, document viewing, and physical device control.

[0009] Furthermore, the present invention aims to provide an agent-type multimodal LLM model that improves the efficiency of error correction by performing replanning when the step-by-step execution result deviates from the planned range.

[0010]

[0011] According to the features of the present invention for achieving the above-mentioned purpose, the present invention comprises an agent-type multimodal LLM providing system that receives modality information including text, image, audio, and numeric information and provides an inference result, the system comprising: an input module that receives modality information including one or more of text information, image information, audio information, or numeric information; an agent module composed of a massive language model (LLM) that integrates sorted modality information to generate execution steps for step-by-step inference in a Chain-of-Thought (CoT) manner and produces execution content for each step; a tool management module that manages tools and metadata for executing the execution content produced by the agent module; and an execution module that calls the tool or sets / changes the metadata to execute the execution content, analyzes the execution result, and provides it to the agent module.

[0012] At this time, the above tool may be configured to include a code execution module, a database access module, a document viewing module, and a physical device control module for executing the above execution content.

[0013] In addition, the above metadata may be configured to include function settings, input / output settings, and restrictions for the operation of the tool.

[0014] In addition, the present invention may further comprise a response module that provides the execution result of the final stage to the user by the agent module as modality information including one or more of text information, image information, audio information, or numeric information.

[0015] In addition, the agent module may revise the execution step if the step execution result provided by the execution module falls outside the planned range.

[0016] In addition, the agent module may provide the user with step-by-step execution results provided by the execution module, and may also receive the user's judgment result regarding whether the execution result deviates from the planned range.

[0017] In addition, the above agent module may derive cause analysis and solutions through Chain of Sort (CoT)-based reasoning when errors (misrecognition, incomplete data, etc.) are generated during the multimodal information inference process.

[0018] In addition, the input module may receive the modality information individually, tokenize it, and integrate it by aligning the distribution and scale.

[0019] And the input module may be configured to include a modality tokenizer that tokenizes the modality information; and a multimodal alignment interface that aligns the distribution and scale of the tokenized modality information.

[0020] In addition, the input module may also collect feature vectors from each sub-model and perform integrated inference on the modality information.

[0021] In addition, the tool management module can design the configuration of the tool as open so that the user can define the tool name, function, input / output format, permissions, etc. when registering a new tool.

[0022] Additionally, the input module may further include a modal weight providing module that assigns a weight indicating the degree of reflection during inference of the agent model for each category of the modality information.

[0023] And the above modal weights may also be set for each execution content.

[0024] In addition, the modal weight providing module learns the modal weights for each user and assigns the learned modal weights according to the user; the learning of the modal weights may be generated by reinforcement learning using the composition ratio of modality information provided when executing each execution content and additional modality information provided by the user during the inference process for the pre-set modal weights.

[0025]

[0026] The following effects can be expected from the CoT-based agent-type multimodal LLM providing system according to the present invention as described above.

[0027] In other words, the present invention not only allows for the derivation of sophisticated inference results by utilizing multiple modality information, but also has the effect of reducing the heterogeneity of the inference results and maximizing real-time interaction with the user by applying an omnimodel.

[0028] Furthermore, in this invention, step-by-step inference results are derived by utilizing external tools for each execution step through CoT-based step-by-step inference, and by analyzing the execution results of each step and performing replanning when the execution results deviate from the planned range, the error correction efficiency of the agent-type multimodal LLM model is improved and stability is ensured.

[0029] In addition, the present invention allows user intervention at each stage of the inference process for each execution stage, thereby enabling the user to perform the execution process together with the agent model.

[0030] In addition, the present invention assigns category-specific weights to multimodal information and adjusts the said weights to match the execution task and user characteristics, thereby having the effect of adjusting the degree of reflection of multimodal information according to the user's personal characteristics and the unique characteristics of the execution task.

[0031]

[0032] FIG. 1 is a block diagram illustrating the configuration of an agent-type multimodal LLM providing system according to a specific embodiment of the present invention.

[0033] FIG. 2 is a conceptual diagram illustrating the inference process of an agent module according to the present invention in comparison with the prior art.

[0034] FIG. 3 is a flowchart illustrating a specific embodiment of the inference process of an agent-type multimodal LLM providing system according to the present invention.

[0035] FIG. 4 is a flowchart illustrating another embodiment of the inference process of an agent-type multimodal LLM providing system according to the present invention.

[0036] FIG. 5 is an exemplary diagram illustrating an example of a weight applied to an agent-type multimodal LLM providing system according to the present invention.

[0037]

[0038] The best embodiment for implementing the present invention is an agent-type multimodal LLM providing system that receives modality information including text, image, audio, and numeric information and provides an inference result, comprising: an input module that receives modality information including one or more of text information, image information, audio information, or numeric information; an agent module composed of a massive language model (LLM) that integrates sorted modality information to generate execution steps for step-by-step inference in a Chain-of-Thought (CoT) manner and calculates execution content for each step; a tool management module that manages tools and metadata for executing the execution content calculated by the agent module; and an execution module that calls the tools or sets / changes metadata to execute the execution content, analyzes the execution result, and provides it to the agent module; wherein the tools may be configured to include a code execution module, a database access module, a document viewing module, and a physical device control module for executing the execution content.

[0039]

[0040] Hereinafter, we will examine an agent-type multimodal LLM providing system according to a specific embodiment of the present invention with reference to the attached drawings.

[0041] Before proceeding with the explanation, the effects, features, and methods for achieving the present invention will become clear from the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0042] In describing the embodiments of the present invention, if it is determined that a detailed description of known functions or configurations may unnecessarily obscure the essence of the invention, such detailed description will be omitted. Furthermore, the terms described below are defined considering the functions in the embodiments of the present invention, and these may vary depending on the intentions or conventions of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0043] Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and since these computer program instructions may be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram or each step of the flowchart.

[0044] Since these computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, the instructions stored in said computer-available or computer-readable memory may also be used to produce a manufactured item containing instruction means that perform the function described in each block of a block diagram or each step of a flowchart.

[0045] And since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0046] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specific logical functions, and in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order.

[0047] In other words, the two blocks or steps described can actually be performed substantially simultaneously, and can also be performed in the reverse order of their corresponding functions as needed.

[0048]

[0049] FIG. 1 is a block diagram illustrating the configuration of an agent-type multimodal LLM providing system according to a specific embodiment of the present invention, FIG. 2 is a conceptual diagram illustrating the inference process of an agent module according to the present invention in comparison with the prior art, FIG. 3 is a flowchart illustrating a specific embodiment of the inference process of an agent-type multimodal LLM providing system according to the present invention, FIG. 4 is a flowchart illustrating another embodiment of the inference process of an agent-type multimodal LLM providing system according to the present invention, and FIG. 5 is an illustrative diagram illustrating an example of a weight applied to an agent-type multimodal LLM providing system according to the present invention.

[0050] First, as illustrated in FIG. 1, the CoT-based agent-type multimodal LLM providing system according to the present invention is configured to include an input module (100), an agent module (200), a tool management module (300), an execution module (400), and a response module (500).

[0051] First, the input module (100) receives multimodal input data provided by a user or an external system and performs the function of preprocessing it into a form that can be processed by an agent module (200). The multimodal data refers to information of an automatic category that includes one or more of text information, image information, audio information, or numerical information.

[0052] At this time, the input module (100) may specifically be configured to include a modality tokenizer (110), a multimodal alignment interface (120), and a modal weight providing module (130).

[0053] The modality tokenizer (110) described above is a part that tokenizes input data of different formats, such as text, images, audio, and numerical information, in a manner suitable for each characteristic. For example, text information may be broken down into subwords or tokens, image information may be converted into patch-based embeddings or feature vectors, and audio information may be converted into frequency spectrums or time series features.

[0054] And the multimodal alignment interface (120) aligns the distribution, scale, and representation space of each tokenized modality information to minimize inference errors caused by information imbalance between different modalities. Through this, the agent module (200) can process multimodal information integrally on a single inference space.

[0055] Of course, the specific configuration of the input module (100) may be configured differently depending on the inference method of the agent module (200), and the inference method of the agent module (200) that adopts the CoT method may be a mono method and an omni method as shown in FIG. 2.

[0056] The above-described embodiment was explained using the omni-mode configuration of the input module (100) as an example, but the mono-mode configuration is also possible.

[0057] Meanwhile, the modal weight providing module (130) is a part that provides weights to adjust the weight of each modality information reflected in the inference process, and this will be described later with reference to FIG. 5.

[0058] Next, the agent module (200) is configured based on a large language model (LLM) and performs step-by-step reasoning based on Chain-of-Thought (CoT) through multimodal information provided from the input module (100).

[0059] That is, unlike a simple input-output method or a single prompt-based inference, the agent module (200) has a structure that internally generates a multi-stage inference path and verifies / supplements intermediate results at each stage.

[0060] Specifically, the agent module (200) sets a plurality of execution steps (plan) to achieve a goal and generates execution content for each execution step.

[0061] At this time, the execution details for each step may include the purpose of execution, the type of tool to be called, tool input parameters, and the expected result range.

[0062] In addition, the agent module (200) analyzes the step-by-step execution results provided by the execution module (400), determines whether the result falls within a pre-set plan range (expected result), and if it falls outside the plan range, can regenerate the execution step (Revise).

[0063] And the tool management module (300) performs the role of managing external tools to actually execute the execution content generated by the agent module (200).

[0064] To this end, the tool management module (300) may include a plurality of tools, such as a code execution module, a database access module, a document viewing module, and a physical device control module, and manages metadata for each tool together.

[0065] The above metadata may include a tool name, function description, input / output format, access rights, execution restriction conditions, etc., and the tool management module (300) is designed with an open structure so that a user can register a new tool and define metadata.

[0066] Meanwhile, the execution module (400) actually calls the tool registered in the tool management module (300) and performs execution according to the execution step generated by the agent module (200).

[0067] In addition, the execution module (400) collects success status, output values, error information, etc., as a result of performing each execution step, and provides this to the agent module (200).

[0068] In this case, if the execution result is incomplete or an error occurs, the relevant information may be used as a basis for replanning.

[0069] Next, the response module (500) provides the final execution result derived by the agent module (200) to the user.

[0070] At this time, it is desirable that the final execution result be provided in a multimodal form including one or more of text, images, audio, and numerical information.

[0071]

[0072] Below, we will examine specific embodiments of the inference process of the agent-type multimodal LLM providing system according to the present invention.

[0073] Figure 3 illustrates a specific example of the inference process of an agent-type multimodal LLM providing system according to the present invention as a flowchart.

[0074] As illustrated in FIG. 3, the agent-type multimodal LLM providing system according to the present invention starts with multimodal data (information) being input (S110).

[0075] At this time, modality information is received by fusing information from various categories (sources), is preprocessed by the input module (100), and transmitted to the agent module (200).

[0076] Afterwards, the agent module (200) distinguishes execution steps based on CoT according to modality information and calculates the execution content for each step (S120).

[0077] And the agent module (200) executes each execution step sequentially, and as described above, the execution method is such that the agent module (200) transmits the execution content to the execution module (400), and the execution module (400) calls each tool managed by the tool management module (300) and executes the execution content (S130).

[0078] Next, the execution module (140) analyzes the execution result and provides it to the agent module (200) (S140).

[0079] At this time, the agent module (200) determines whether the analyzed execution result is within the planned range (S150), and if the execution result is outside the planned range, modifies and supplements the execution steps and the execution details of each step.

[0080] On the other hand, if the above execution result does not exceed the planned range, the execution contents of each step are executed sequentially until the final step is reached (S160).

[0081] Subsequently, if the established final step is executed based on the judgment result of the above 160th step, the response module (500) provides the final inference result to the user (S170).

[0082] Of course, in this case, the above response module (500) can provide the final inference result in a multimodal form and can provide the inference process and execution results for each step together.

[0083]

[0084] Meanwhile, another embodiment of the inference process of the agent-type multimodal LLM providing system illustrated in FIG. 4 allows user intervention at each execution step in the embodiment illustrated in FIG. 3.

[0085] Specifically, in the above step 140, the execution module (140) provides the analyzed execution result to the agent module (200) and the user simultaneously (S141).

[0086] Afterwards, if there is user intervention, the agent module (200) modifies and supplements the execution steps and the execution details of each step, just as in the case where the analyzed execution result is outside the planned range (S143).

[0087] At this time, the term "user intervention" refers to the user requesting a reset or modification of the inference process after seeing the execution result of the relevant step.

[0088] Accordingly, in the present invention, when performing inference through the CoT method, corrections are made at incorrect stages, thereby enabling the inference process that satisfies the user to be performed together with the user.

[0089] Meanwhile, according to another embodiment of the present invention, as illustrated in FIG. 5, when processing multimodal information of various categories, category-specific weights can be assigned to the multimodal information so that the multimodal information for each category is optimized according to the characteristics of the execution task and the individual characteristics of the user.

[0090] That is, as illustrated in FIG. 1, the input module (100) is configured to further include a modal weight providing module (130) that assigns a weight indicating the degree of reflection during inference of the agent model for each category of the modality information.

[0091] And the modal weight providing module (130) sets weights for each category for each execution content as shown in FIG. 5, and the setting of weights can be distinguished by user.

[0092] In addition, the modal weight providing module (130) can learn the user's execution processes and optimize the set weights to match the user's characteristics.

[0093] At this time, the learning of the above modal weights can be optimized by reinforcement learning using the pre-set modal weights through the composition ratio of modality information provided during the execution of each execution content and modality information additionally provided by the user during the inference process.

[0094] For example, in image analysis tasks, the weights of image or video information may be set relatively high, while in control command execution tasks, the weights of text or numeric information may be set high. Additionally, weights may be dynamically updated using a reinforcement learning method based on user feedback.

[0095] Through this, the degree of reflection of multimodal information in the present invention is adjusted according to the user's personal characteristics and the unique characteristics of the execution task, thereby enabling the maximization of inference efficiency.

[0096] Next, we will examine the specific details of the reinforcement learning applied to optimize weights in the modal weight providing module (130) above.

[0097] The above reinforcement learning corrects pre-set modal weights based on the execution result of the corresponding step. Various reinforcement learning algorithms may be used, and by utilizing user feedback and the accuracy of the execution result as a reward (R), each modality ( The following formula can be applied to the algorithm for updating the weights of ).

[0098]

[0099]

[0100] Here,

[0101] is the reflection weight of modality Mi for task Tj at the k-th execution;

[0102] is the learning rate;

[0103] is the difference between the current reward and the average reward (Advantage);

[0104] represents the gradient of the log probability density function at the current weight.

[0105]

[0106] Accordingly, when the agent module calls a specific tool during the CoT inference process, the weight of the modality combination that produced a successful result is increased, and this serves as the basis for updating the user-customized weight table of Fig. 5 in real time.

[0107]

[0108] For example, in the case of providing a context-sensitive smart home service, if a user inputs a photo of the view outside the window along with a command saying, "Create a living room atmosphere that matches the current weather,"

[0109] The input module (100) collects voice (audio) and photo (image) data, and the distribution of each data is aligned through the modality tokenizer (110) and the multimodal alignment interface (120).

[0110] Subsequently, the modal weight providing module (130) provides the existing learned weights for the corresponding task (Tj). Assigns.

[0111] And the agent module (200) generates execution steps such as 'image analysis → weather determination → lighting control' through CoT.

[0112] Next, the execution module (400) calls the API of the tool management module (300) to execute the execution content of the execution step, and user satisfaction according to the execution result ( ) is substituted into the formula below to update weights in real time, thereby enhancing personalized performance.

[0113]

[0114]

[0115] The rights of the present invention are not limited to the embodiments described above but are defined by the claims, and it is obvious that a person skilled in the art may make various modifications and adaptations within the scope of the rights described in the claims.

[0116]

[0117] The present invention relates to a multimodal large language model (LLM) that integrates and processes modality information. According to the present invention, step-by-step inference results are derived by utilizing external tools for each execution step through CoT-based step-by-step inference, and by analyzing the execution results of each step and performing replanning when the execution results of each step deviate from the planned range, the error correction efficiency of the agent-type multimodal LLM model is improved and stability is ensured.

Claims

1. An agent-type multimodal LLM providing system that receives modality information including text, image, audio, and numeric information and provides inference results, An input module that receives modality information including one or more of text information, image information, audio information, or numeric information; An agent module composed of a Large Language Model (LLM) that integrates sorted modality information to generate execution steps for step-by-step reasoning in a Chain-of-Thought (CoT) manner and produces step-by-step execution content; A tool management module that manages tools and metadata for executing the execution content produced by the above agent module; and A CoT-based agent-type multimodal LLM providing system characterized by comprising: an execution module that calls the above tool or sets / changes metadata to execute the above execution content, analyzes the execution result, and provides it to the above agent module.

2. In Paragraph 1, The above tool is, A CoT-based agent-type multimodal LLM providing system characterized by comprising a code execution module, a database access module, a document viewing module, and a physical device control module for executing the above execution contents.

3. In Paragraph 2, The above metadata is, A CoT-based agent-type multimodal LLM providing system characterized by being configured to include function settings, input / output settings, and restrictions for the operation of the above tool.

4. In Paragraph 3, A CoT-based agent-type multimodal LLM providing system characterized by further comprising a response module that provides the execution result of the final stage to the user as modality information including one or more of text information, image information, audio information, or numerical information by the agent module above.

5. In Paragraph 4, The above agent module is, A CoT-based agent-type multimodal LLM providing system characterized by revisiting the execution step when the step execution result provided by the execution module deviates from the planned range.

6. In Paragraph 5, The above agent module is, A CoT-based agent-type multimodal LLM providing system characterized by providing step-by-step execution results provided from the above-mentioned execution module to the user, and receiving the user's judgment result regarding whether the execution results deviate from the planned range.

7. In Paragraph 5, The above agent module is, A CoT-based agent-type multimodal LLM providing system characterized by deriving cause analysis and solutions through Chain of Sort (CoT)-based reasoning when errors (misrecognition, incomplete data, etc.) are generated during the multimodal information inference process.

8. In any one of paragraphs 1 through 7, The above input module is, A CoT-based agent-type multimodal LLM providing system characterized by individually receiving the above modality information, tokenizing it, and integrating it by aligning its distribution and scale.

9. In Paragraph 8, The above input module is, A modality tokenizer that tokenizes the above modality information; A CoT-based agent-type multimodal LLM providing system characterized by being configured to include a multimodal alignment interface that aligns the distribution and scale of tokenized modality information.

10. In any one of paragraphs 1 through 7, The above input module is, A CoT-based agent-type multimodal LLM providing system characterized by collecting feature vectors from each sub-model and performing integrated inference on the above modality information.

11. In any one of paragraphs 1 through 7, The above tool management module is, A CoT-based agent-type multimodal LLM providing system characterized by designing the configuration of the above tool as open, allowing the user to define the tool name, functions, input / output formats, permissions, etc. when registering a new tool.

Images