Response generation method and system
The response generation method aligns AI models' responses with query characteristics by determining appropriate inference types and guides the response flow, enhancing reliability in critical environments.
Patent Information
- Application Number
- PCT/KR2026/096978
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2026-05-13
- Filing Date
- 2026-05-14
- Publication Date
- 2026-11-19
AI Technical Summary
Existing AI models struggle to generate responses based on reasoning methods suitable for the characteristics of diverse queries, particularly in critical environments like education, research, and business support, where response reliability is essential.
A response generation method and system that determines an appropriate inference type based on context information, inputs an inference guide, and adjusts the AI model's response flow to align with the query's characteristics, using a classification model to select and reflect specific reasoning methods.
This approach enhances response reliability by ensuring AI models generate responses aligned with query characteristics, reducing inappropriate inference types and improving inference quality without additional training or resource overhead.
Smart Images

Figure KR2026096978_19112026_PF_FP_ABST
Abstract
Description
Response generation method and system
[0001] The present invention relates to a response generation method and system, wherein the method and system generate data related to inference patterns from long text inference information generated by an artificial intelligence model, identify a target inference pattern suitable for a query based on the generated data, and enable the artificial intelligence model to generate a response according to the target inference pattern.
[0002] With the recent advancement of artificial intelligence technology, techniques for generating responses to various queries using AI models are being utilized across multiple fields. In particular, AI models can generate long-form reasoning information involving step-by-step reasoning processes to solve complex problems, thereby expanding their scope of application in diverse tasks such as mathematical problem solving, knowledge-based question answering, logical judgment, and safety assessment.
[0003] In environments utilizing long-text reasoning, not only the responses generated by AI models but also the reasoning patterns leading to those responses are recognized as important factors. Even for the same query, various reasoning patterns can emerge, such as stepwise analysis, exploration of multiple possibilities, judgment based on hypotheses, or judgment based on verification; differences in these reasoning patterns can influence the direction and content of the response.
[0004] Furthermore, as query types become more diverse, the reasoning patterns required for each query may also differ. For example, mathematical calculations, the evaluation of scientific evidence, safety assessments, and the identification of user intent may require different reasoning methods; consequently, there is an increasing need to consider reasoning patterns suitable for the characteristics of the query.
[0005] Furthermore, since long-text inference information can include multiple inference methods, it is becoming increasingly important to consider the relationship between the query and the inference pattern to utilize the AI model's response in a form that aligns with the query. In particular, in environments where response reliability is critical—such as education, research support, expert knowledge responses, safety judgments, and business support—the demand for generating responses based on inference patterns suitable for the query is gradually expanding.
[0006] Therefore, there is a need for technology that enables artificial intelligence models to generate responses based on reasoning methods suitable for the characteristics of the query.
[0007] The present invention is intended to provide a response generation method and system capable of generating a response from an artificial intelligence model based on a reasoning method suitable for the characteristics of a query.
[0008] More specifically, the present invention is intended to provide a response generation method and system capable of determining the inference method used to generate a response in an artificial intelligence model according to the characteristics of a query.
[0009] In addition, the present invention is intended to provide a response generation method and system capable of reflecting a reasoning method suitable for a query in a prompt.
[0010] Furthermore, the present invention aims to provide a response generation method and system capable of generating criteria for distinguishing different inference methods from the inference content of an artificial intelligence model.
[0011] Furthermore, the present invention aims to provide a response generation method and system in which different reasoning methods are distinguished according to certain criteria, and the distinguished reasoning methods are utilized to generate a response to a query.
[0012] To solve the problem described above, the response generation method, response generation system, and classification model learning method according to the present invention provide a data processing method that determines an appropriate inference type based on context information of an input query and inputs an inference guide corresponding to the determined inference type together with the artificial intelligence model, thereby allowing the response flow of the artificial intelligence model to be adjusted at the time of inference.
[0013] More specifically, a response generation method according to the present invention may include the steps of: obtaining at least one query; identifying context information related to at least one of the meaning and domain of the query; specifying a target inference type to be used for inference regarding the query based on the context information; generating input data including the query and an inference guide corresponding to the target inference type so that an artificial intelligence model performs inference according to the target inference type; inputting the input data to the artificial intelligence model; receiving a response to the query generated from the artificial intelligence model based on an inference result according to the target inference type; and providing a response to the query.
[0014] In an embodiment, the step of specifying the target inference type may include the step of selecting at least one inference type corresponding to the context information as the target inference type among a plurality of inference types corresponding to different inference methods.
[0015] In an embodiment, the inference guide may include at least one of name information of the target inference type, definition information of an inference method according to the target inference type, and guidance information that guides an inference procedure according to the inference method.
[0016] In an embodiment, the artificial intelligence model can perform the inference according to the target inference type based on the inference guide included in the input data, and generate the response to the query based on the inference result.
[0017] In an embodiment, the artificial intelligence model determines the inference order for the query based on the inference guide, performs inference according to the inference order to generate an intermediate inference result based on the inference, and can generate the response to the query based on the intermediate inference result.
[0018] In an embodiment, the intermediate inference result may include at least one of a partial inference result, partial conclusion information, and verification result included in a chain of thought.
[0019] In an embodiment, the plurality of inference types may correspond to each of a plurality of different reference queries, and the step of selecting the target inference type may include the step of calculating association information between the query and each of the plurality of reference queries, the step of selecting at least one specific reference query associated with the query among the plurality of reference queries based on the calculated association information, and the step of specifying the target inference type based on at least one inference type among the plurality of inference types that corresponds to the specific reference query.
[0020] In an embodiment, the step of specifying the target inference type may include the step of verifying response performance information corresponding to each of at least one inference type corresponding to the specific criterion query, and the step of specifying any one of the at least one inference type as the target inference type based on the response performance information.
[0021] In an embodiment, the response performance information may include at least one of accuracy information indicating the degree to which the response to which the at least one inference type is applied corresponds to the correct answer, safety information indicating the degree to which the response corresponds to a preset safety standard, and user feedback information regarding the response.
[0022] In an embodiment, the step of verifying the context information may include the step of extracting at least one of semantic information representing the meaning of the query and domain information representing the domain of the query from the query, and the step of generating the context information based on at least one of the extracted semantic information and domain information.
[0023] In an embodiment, the inference guide may be selected from among a plurality of pre-established inference guides as an inference guide corresponding to the target inference type.
[0024] In an embodiment, selecting the inference guide may involve selecting the inference guide corresponding to the target inference type based on a pre-established matching criterion between the plurality of inference guides and the plurality of inference types.
[0025] In an embodiment, the query may be input by a user, the inference guide may be determined based on the context information, and the input data may be generated by combining the query and the determined inference guide.
[0026] In an embodiment, the step of specifying the target inference type may include the step of inputting the context information into a pre-trained classification model and receiving the target inference type from the classification model, and the classification model may be a model that has learned the relationship between the context information of a reference query input as training data and the response performance information of the inference type corresponding to the reference query.
[0027] In an embodiment, the step of specifying the target inference type may further include the step of determining the output of the classification model as the target inference type when the confidence score of the target inference type output from the classification model is greater than or equal to a preset threshold, and the step of determining the target inference type based on a preset basic inference type or reference query matching when the confidence score is less than the preset threshold.
[0028] In an embodiment, the response generation method according to the present invention may further include the steps of: calculating response performance information for the response to which the target inference type is applied; receiving user feedback information for the response; updating the response performance information based on the user feedback information; and using the updated response performance information to determine the target inference type of a subsequent query.
[0029] Meanwhile, the response generation system according to the present invention may include one or more processors and a memory in which instructions executed by the one or more processors are stored, and the one or more processors may be configured to determine an inference type corresponding to the query among a plurality of inference types based on context information of the query, input an inference guide corresponding to the determined inference type together with the query into an artificial intelligence model, and provide a response output from the artificial intelligence model.
[0030] Meanwhile, the program according to the present invention is a program that can be stored on a computer-readable recording medium and may include instructions that are executed by one or more processors to perform the steps of: obtaining at least one query; verifying attribute information corresponding to the query; specifying a target inference pattern among a plurality of inference patterns corresponding to the query and to be used for inferring a response to the query based on the attribute information; configuring a prompt using at least one of the target inference pattern and the query so that an artificial intelligence model performs inference according to the target inference pattern; inputting the prompt to the artificial intelligence model; and receiving a response generated based on the target inference pattern from the artificial intelligence model.
[0031] Meanwhile, the classification model learning method according to the present invention is a method performed by a computer and may include: a step of extracting context information related to at least one of the meaning and domain of a reference query for each of a plurality of reference queries; a step of generating a response by applying a plurality of inference types to each of the plurality of reference queries and calculating response performance information for the generated response; a step of configuring a combination of the extracted context information, the applied inference type, and the calculated response performance information as training data; and a step of training a classification model that receives context information based on the training data and outputs an inference type corresponding to the context information.
[0032] In an embodiment, the classification model may include a plurality of binary classifiers corresponding to different inference method dimensions, and each of the plurality of binary classifiers may be trained to output one of two inference types in the corresponding inference method dimension.
[0033] As described above, the response generation method and system according to the present invention can identify a target inference pattern suitable for a query among a plurality of inference patterns based on attribute information corresponding to the query. Through this, an inference pattern that matches the characteristics of the query can be selected in an environment where various inference methods can be applied even to the same query, and the problem of an artificial intelligence model generating a response based on an inference method unsuitable for the query can be structurally reduced.
[0034] In addition, the response generation method and system according to the present invention can construct a prompt by combining inference instruction information corresponding to a target inference pattern with a query. Through this, it is possible to induce the reflection of a specific inference order, verification method, idea development method, and conclusion derivation method when an artificial intelligence model generates a response, and to generate a response that reflects an inference structure suitable for the query, rather than a simple result-oriented response.
[0035] Furthermore, the response generation method and system according to the present invention can generate classification criteria for distinguishing inference patterns based on long-text inference information generated by an artificial intelligence model. Through this, classification criteria for inference patterns can be generated based on the inference methods actually appearing in the long-text inference information, without relying on fixed criteria pre-set by humans, and can flexibly reflect differences in inference methods appearing in various queries and tasks.
[0036] Furthermore, the response generation method and system according to the present invention can generate embedding vectors for each of the classification criteria and perform clustering based on the similarity between the embedding vectors. Through this, semantically similar inference pattern classification criteria can be automatically grouped, and duplicate or similar criteria can be organized to derive inference pattern classification criteria that ensure representativeness.
[0037] Furthermore, the response generation method and system according to the present invention can generate rubric data for classifying inference patterns corresponding to representative classification criteria. Through this, the definition and characteristics of each inference pattern can be provided in a structured form, and different inference patterns can be interpreted and compared according to consistent criteria.
[0038] Furthermore, the response generation method and system according to the present invention can classify inference patterns corresponding to long-text inference information using rubric data. Through this, it is possible to determine which inference pattern the long-text inference information generated by an artificial intelligence model corresponds to, and to analyze the characteristics of the inference method included in the long-text inference information.
[0039] Furthermore, the response generation method and system according to the present invention can store representative classification criteria, rubric data, classification results of inference patterns corresponding to long-text inference information, and data related to the classification results, and utilize them for processing new queries. Through this, a target inference pattern for a new query can be identified based on the previously stored information, and it becomes possible to generate a response that reflects an inference pattern suitable for the query.
[0040] Furthermore, the response generation method and system according to the present invention can generate rubric data for classifying inference patterns corresponding to representative classification criteria. Through this, the definition and characteristics of each inference pattern can be provided in a structured form, and different inference patterns can be interpreted and compared according to consistent criteria.
[0041] Furthermore, the response generation method and system according to the present invention can classify inference patterns corresponding to long-text inference information using rubric data. Through this, it is possible to determine which inference pattern the response generated by the artificial intelligence model was derived based on, and to systematically analyze the suitability of the response and the characteristics of the inference method.
[0042] Furthermore, the response generation method and system according to the present invention can store representative classification criteria, rubric data, classification results of inference patterns corresponding to long-text inference information, and data related to the classification results, and utilize them for processing new queries. Through this, accumulated data can be reused to quickly identify target inference patterns for new queries, and it becomes possible to generate responses that reflect inference patterns suitable for queries without repetitive data generation.
[0043] Furthermore, the response generation method and system according to the present invention may have a structure in which a target inference type is determined from context information of a query, and an inference guide corresponding to the target inference type is input together into an artificial intelligence model. Through this structure, an inference procedure suitable for the characteristics of the query can be reflected in the response generation process of the artificial intelligence model without updating the parameters of the artificial intelligence model through additional learning, and consequently, an improvement in inference quality can be achieved without increasing the computational resources and storage capacity required for the training of the artificial intelligence model.
[0044] Furthermore, the response generation method and system according to the present invention can be configured so that a single artificial intelligence model can generate responses corresponding to multiple inference types without having to train or possess a separate artificial intelligence model for each query. As a result, the number of models in the entire system and the complexity of model management can be reduced compared to a structure that builds separate artificial intelligence models for each domain.
[0045] Furthermore, the response generation method and system according to the present invention may include a fallback mechanism in which a target inference type is determined based on a preset default inference type or reference query matching when the confidence score falls below a preset threshold. Through this, the risk of an inappropriate inference type being applied due to misclassification by the classification model can be reduced, and the stability and consistency of the response quality of the artificial intelligence model can be improved.
[0046] Furthermore, the response generation method and system according to the present invention may have a data processing structure that generates classification criteria based on long-text inference information generated by an artificial intelligence model, and clusters embedding vectors corresponding to the generated classification criteria to derive representative classification criteria. Through this data processing structure, inference methods actually observed in various queries or tasks can be reflected in the classification criteria compared to cases where fixed classification criteria are manually set in advance by humans; consequently, the suitability of the classification criteria to the operating environment can be improved, and the cost of rebuilding the classification criteria can be reduced when new inference methods are observed.
[0047] FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented.
[0048] FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an embodiment of a computing system in which the present invention can be implemented.
[0049] Figure 3 illustrates an example of a block diagram from another perspective of a computing device, which is one of the components of a computing system.
[0050] FIG. 4 is a conceptual diagram illustrating a response generation system according to the present invention.
[0051] FIG. 5 is a flowchart illustrating a response generation system according to the present invention.
[0052] FIG. 6 is a flowchart illustrating a data generation method of a response generation system according to the present invention.
[0053] FIGS. 7, FIGS. 8, FIGS. 9, FIGS. 10, FIGS. 11, FIGS. 12 and FIGS. 13 are conceptual diagrams for explaining a data generation method of a response generation system according to the present invention.
[0054] FIG. 14 is a flowchart illustrating a response generation method according to the present invention.
[0055] FIGS. 15, 16, 17, 18 and 19 are conceptual diagrams for explaining a response generation method according to the present invention.
[0056] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.
[0057] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0058] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0059] Singular expressions include plural expressions unless the context clearly indicates otherwise.
[0060] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0061] Meanwhile, FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. In this regard, the response generation system according to the present invention can be implemented through a computing device described below and can perform data processing related to the response generation method described in this specification.
[0062] Referring to FIG. 1, a computing system (10000) performing a response generation method according to one embodiment of the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multiprocessor computing device.
[0063] The components of at least one computing device of the present invention may include various hardware components such as one or more processors, memory, other hardware, and a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.
[0064] Meanwhile, at least one computing device included in the computing system (10000) that performs the response generation method may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.
[0065] Meanwhile, according to one embodiment of the present specification, when training a classification model according to the present invention, the data format of the training data (e.g., multiple-choice format and open-ended format) may affect the distribution of inference patterns determined by the trained classification model, and accordingly, the classification model training method according to the present invention can ensure consistency in determining inference patterns in an operating environment by diversifying the data format of the training data into at least one format. In addition, according to another embodiment, if there are multiple classification models or response generation models trained in different data formats, a model merging step may be additionally performed to combine the weights of the multiple models using weight merging or linear interpolation, thereby allowing the distribution of inference patterns to be flexibly adjusted without additional training.
[0066] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and a private network through a network interface or an adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method by which at least one computing device is connected to communicate through the network (1070) is not limited and may be connected to communicate in a manner different from the described examples.
[0067] Furthermore, at least one of other computer-type devices and systems not shown in FIG. 1 may interact technically with at least one computing device or other system through one or more connections to a network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).
[0068] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.
[0069] A computing system (10000) that performs a response generation method according to the present invention may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).
[0070] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and a memory (1012) for performing a response generation method. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).
[0071] At least one processor (1011) constituting the user computing device (1010) may include at least one general-purpose processor and at least one special-purpose processor. For example, at least one processor (1011) constituting the user computing device (1010) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and other electrical units for performing functions, or a plurality of electrically connected processors.
[0072] Furthermore, at least one processor (1011) may be configured to execute at least one of the computer-readable instructions contained in memory (1012) and other instructions described herein.
[0073] The memory (1012) constituting the user computing device (1010) according to the present invention may include at least one physically durable storage medium among volatile memory, non-volatile memory, fixed medium, removable medium, magnetic medium, optical medium, semiconductor medium and other types.
[0074] For example, memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs the storage function of memory on the internet. Such memory (1012) may store data and instructions necessary for the at least one processor (1011) to perform the operation of an application for inference pattern analysis and inference pattern-based response generation.
[0075] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.
[0076] A user can interact with a computing system (10000) including at least one computing device through at least one input / output of input text, touch, voice, movement, computer vision, gesture, and other forms using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface, and other user interface (UI) representations.
[0077] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on at least one of the user inputs received from the user interface and the network.
[0078] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.
[0079] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) for generating text inference information corresponding to a query or analyzing inference patterns included in text inference information, or other types of machine learning models including at least one of a non-linear model and a linear model, and may be composed of a combination thereof.
[0080] According to an embodiment of the present invention, a user computing device (1010) may perform a response generation method using a local or / and external machine learning model (1020). Alternatively, the user computing device (1010) may perform a response generation method using a machine learning model (1040) provided by a server.
[0081] Additionally, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) may provide at least one of an inference pattern analysis result, an inference pattern classification result, rubric data, and an inference pattern-based response generation result to the user computing device (1010) on at least one of an application and a web, in accordance with a request from a user received through the user computing device (1010).
[0082] In addition, according to another embodiment of the present invention, at least a part of the user computing device (1010) and the server computing system (1030) are linked together to perform a response generation method, thereby providing the user with a response generation result based on an inference pattern or an inference pattern analysis result.
[0083] Additionally, according to various embodiments of the present invention, at least one of a user computing device (1010) and a server computing system (1030) can learn a machine learning model (1020, 1040) used in a data generation method or a response generation method through interaction with a training computing system (1050) that is communicationally connected via a network (1070). In this case, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be part of the server computing system (1030) or part of the user computing device (1010).
[0084] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).
[0085] The memory (1032) constituting the server computing system (1030) according to the present invention may include at least one of volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.
[0086] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the at least one processor (1031) to perform the operation of an application for inference pattern analysis and inference pattern-based response generation may be stored.
[0087] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.
[0088] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on at least one machine learning model.
[0089] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.
[0090] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.
[0091] The training data (1061) of the present invention may include at least one of input data and data previously output from at least one machine learning model (e.g., recursive learning feedback).
[0092] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, the model connection between different machine learning models may include or represent at least one of model parameters and relationships between models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities. The present invention does not limit the machine learning parameters described according to the embodiments, and a single machine learning model may further include a plurality of model parameters.
[0093] Meanwhile, FIG. 2 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.
[0094] As illustrated in FIG. 2, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing a machine learning-based data generation method or a response generation method. The at least one application included in the computing device (1100) may communicate with the sensor, context manager, device state manager, or additional component(s) within the computing device (1100) via an Application Programming Interface (API). In one embodiment, the at least one application may interface with device components, such as receiving sensor data or state data or transmitting prediction results to an output device via a public or private API.
[0095] Meanwhile, FIG. 3 illustrates an example of a block diagram in another aspect of a computing device (1200), which is one of the components of a computing system (10000) that performs a response generation method according to an embodiment of the present invention.
[0096] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).
[0097] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.
[0098] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store at least one of the following: a query stored within the computing device (1200), long text inference information, a response generated based on the long text inference information, correct answer information, inference pattern classification criteria, embedding vector, cluster information, representative inference pattern classification criteria, rubric data, inference pattern classification result, and target inference pattern information, and provide this as input data required for the data generation method and the response generation method. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) through a private API, etc.
[0099] The technology described herein may be composed of a single or multiple computing devices, and a machine learning model performing a data generation method or a response generation method may be executed sequentially or in parallel on one component or multiple distributed components. Data storage, machine learning models, and applications may be distributed and operated locally or over a network, and these configurations can be flexibly applied to various system architectures.
[0100] Key terms used in the claims and description of the invention described herein shall be interpreted based on their meanings commonly accepted in the technical field to which the invention pertains; however, if explicitly defined with a different meaning in this specification, such definition shall be understood to prevail. Furthermore, key terms used in this specification may refer to the same technical concept using various expressions; accordingly, this specification explicitly declares the equivalent expressions (synonyms or equivalent terms) referred to by each key term through subsequent paragraphs. This declaration of equivalence is intended to clarify that the terms described in the claims are intended to be compatible with the text of the specification or other expressions describing the same invention.
[0101] In this specification, "context information" may refer to data representing at least one of the semantic characteristics, domain characteristics, formal characteristics, and intentional characteristics of a query as an analysis result derived from the query. More specifically, the context information may include at least one of semantic information, domain information, type information, format information, difficulty information, and intent information. The "context information" in this specification may also be referred to as "attribute information," "analysis information," "feature information," "query analysis data," "query embedding," "query feature vector," "context information," "query attribute," or "query analysis data," and all such expressions may be understood as referring to the same concept as "context information" used in this specification.
[0102] In this specification, "reasoning type" may refer to a category of thought flow that can be applied in the process of an artificial intelligence model deriving a response to a query, and may include predefined reasoning categories distinguished according to at least one classification criterion among analysis perspective, scope of search, classification of reasoning types, mode of idea development, verification method, and clarification approach. In this specification, "reasoning type" may also be referred to as "reasoning pattern," "reasoning method," "reasoning strategy," "reasoning mode," "thinking pattern," and "reasoning style," and all such expressions may be understood as referring to the same concept as "reasoning type" used in this specification.
[0103] Meanwhile, according to one embodiment of the present specification, the scope of exploration dimension is a classification dimension that distinguishes the width of candidate inference paths considered by an artificial intelligence model for generating a response, and can be implemented as a binary classifier that outputs either of two inference methods (e.g., a narrow and focused exploration method and a broad and extensive exploration method) within the same dimension. More specifically, a narrow and focused exploration method may refer to a method in which the artificial intelligence model derives a response by focusing on the single most probable inference path or a small number of candidate inference paths, and a broad and extensive exploration method may refer to a method in which the artificial intelligence model derives a response by simultaneously considering multiple candidate inference paths and selecting an appropriate path among them.
[0104] Meanwhile, according to one embodiment of the present specification, the clarification approach dimension is a classification dimension that distinguishes the conclusion clarification method until the artificial intelligence model derives a response, and can be implemented as a binary classifier that outputs either of two inference methods (e.g., iterative refinement inference method and immediate conclusion inference method) within the same dimension. More specifically, the iterative refinement inference method may refer to an inference method in which the artificial intelligence model derives a tentative conclusion and then progressively supplements and refines it to reach a final conclusion, and the immediate conclusion inference method may refer to an inference method in which the artificial intelligence model derives a final conclusion in a single step or a minimum step based on an input query and an inference guide.
[0105] In this specification, "reasoning guide" refers to information for guiding an artificial intelligence model to perform reasoning according to a specific reasoning type, and may include at least one of name information of the reasoning type, definition information of the reasoning method, and guidance information of the reasoning procedure. In this specification, "reasoning guide" may also be referred to as "reasoning instruction," "reasoning guidance," "thought flow guidance data," "reasoning prompt component," and "reasoning directive," and all of these expressions may be understood as referring to the same concept as "reasoning guide" as used in this specification.
[0106] In this specification, "input data" may refer to data input into an artificial intelligence model to enable the model to generate a response, and may consist of at least a combination of a query and an inference guide, and may additionally include context information or a part thereof. In this specification, "input data" may also be referred to as "prompt," "input prompt," "model input," "query prompt," and "input prompt," and all such expressions are understood to refer to the same concept as "input data" as used in this specification.
[0107] In this specification, "intermediate reasoning result" may refer to an output derived by an artificial intelligence model while proceeding with a thought flow according to a reasoning guide, and may include at least one of a partial reasoning result, partial conclusion information, verification result, and inference hypothesis information included in a chain of thought. In this specification, "intermediate reasoning result" may also be referred to as "partial reasoning result," "chain-of-thought output," "thinking trajectory," "reasoning trace," and "intermediate inference output," and all of the above expressions may be understood as referring to the same concept as "intermediate reasoning result" used in this specification.
[0108] In this specification, "response performance information" refers to information indicating the evaluation results for a response to which a specific type of inference has been applied, and may include at least one of accuracy information indicating the degree to which the response corresponds to the correct answer, safety information indicating the degree to which the response corresponds to a preset safety standard, and user feedback information regarding the response. In this specification, "response performance information" may also be referred to as "performance score," "evaluation score," "quality metric," "reward score," "response quality," and "evaluation metric," and all of these expressions may be understood as referring to the same concept as "response performance information" used in this specification.
[0109] In this specification, "reference query" may refer to a pre-registered query used as reference material for determining the inference type, and may be stored mapped to at least one inference type and corresponding response performance information. In this specification, "reference query" may also be referred to as "reference query," "anchor query," "sample query," "registered query," and "reference example," and all such expressions are understood to refer to the same concept as "reference query" as used in this specification.
[0110] In this specification, the term "classification model" may refer to an artificial intelligence model that receives context information as input and outputs an inference type corresponding to said context information, and may be implemented in the form of a single multi-class classifier or in the form of multiple binary classifiers corresponding to each dimension of different inference methods. In this specification, the term "classification model" may also be referred to as a "routing model," "selector model," "decision model," "pattern classifier," and "reasoning router," and all such expressions are understood to refer to the same concept as the term "classification model" used in this specification. Meanwhile, general terms such as "query," "artificial intelligence model," and "response" described in the claims in this specification shall be interpreted in their ordinary meaning in the technical field to which the present invention belongs, unless otherwise defined. Furthermore, formal limitations such as "characterized by" described in the claims are intended to specify the essence of the claimed invention and should not thereby limit the claimed invention to the specific embodiments described in this specification.
[0111] In this specification, "LLM booster" may refer to a data processing configuration that improves (boosts) the response performance of an artificial intelligence model, including a large-scale language model (LLM), by inducing the model to follow a more appropriate inference flow for an input query. More specifically, the LLM booster may be understood as a data processing configuration that performs at least one of the following operations: checking context information of a query; identifying a target inference type suitable for the query among a plurality of inference types based on the context information; generating input data by combining an inference guide corresponding to the target inference type with the query; and inputting the input data into the artificial intelligence model to obtain a response based on an inference result according to the target inference type. That is, the LLM booster may function as an external augmentation configuration that improves at least one of the response accuracy and safety of the artificial intelligence model relative to a baseline by adding an inference guide at the query stage without changing the weights of the artificial intelligence model itself. In this specification, "LLM Booster" may also be referred to as "reasoning booster," "reasoning enhancement module," "reasoning guide addition module," "response performance improvement module," "reasoning booster," "LLM augmentation module," "response quality booster," and "inference enhancement framework," and all such expressions are understood to refer to the same concept as "LLM Booster" as used in this specification. The response generation system (100), response generation method, and corresponding program described in this specification may be understood as an embodiment implementing the LLM Booster.In other words, the present invention can be said to relate to an LLM booster that improves the response performance of an artificial intelligence model.
[0112] Meanwhile, according to one embodiment of the present specification, the reasoning method dimensions that can be adopted by the classification model according to the present invention include: (i) a dimension of analytical perspective in which either a top-down reasoning method or a bottom-up reasoning method is selected; (ii) a dimension of scope of exploration in which either a narrow and focused search method or a broad search method is selected; (iii) a dimension of reasoning type in which either an inductive reasoning method or a deductive reasoning method is selected; (iv) a dimension of idea development in which either a sequential development method or a parallel development method is selected; (v) a dimension of verification focus in which either a data-driven verification method or a hypothesis-driven verification method is selected; and (vi) a dimension of clarification approach in which an iterative refinement method or an immediate conclusion It may include at least one of the dimensions for selecting one of the methods. Each of the six dimensions can be defined as two opposing inference methods, and thus the classification model of the present invention can be implemented in the form of a plurality of binary classifiers that output one of the two opposing methods for each dimension with respect to input context information.
[0113] As used herein, the "default reasoning type" may also be referred to as "default reasoning type," "fallback reasoning type," "default thought pattern," or "default pattern." All such expressions should be understood to mean a safety fallback reasoning type adopted by the response generation system according to the present invention when the confidence score of the classification model falls below a preset threshold or when post-processing verification of the output of the classification model is not satisfied. In other words, the fallback mechanism according to the present invention can function as a safety device to reduce the risk of an inappropriate reasoning type being applied due to misclassification by the classification model.
[0114] Meanwhile, according to one embodiment of the present specification, a response generation method according to the present invention may be composed of a series of steps performed by a computer, wherein the steps may include: obtaining at least one query; verifying context information related to at least one of the semantics and domains of the query; specifying a target inference type to be used for inference regarding the query based on the context information; generating input data including the query and an inference guide corresponding to the target inference type so that an artificial intelligence model performs inference according to the target inference type; inputting the input data to the artificial intelligence model; receiving a response to the query generated from the artificial intelligence model based on the inference result according to the target inference type; and providing a response to the query. More specifically, in the step of verifying the context information, the response generation system may extract semantic and domain characteristics inherent in the query using at least one of natural language analysis, semantic embedding, or a domain classifier. In the step of specifying the target inference type, the response generation system may select at least one inference type that best matches the context information from among multiple inference types corresponding to different inference methods (e.g., top-down analysis pattern, bottom-up analysis pattern, stepwise arithmetic inference pattern, hypothesis testing pattern, or safety review pattern, etc.) as the target inference type. The selection in this step may be implemented in a deterministic manner that outputs a single candidate, or in a probabilistic manner that calculates a score for each of the multiple candidates. Additionally, in the step of generating the input data, the response generation system may combine the query itself with an inference guide corresponding to the target inference type to configure an input structure so that the artificial intelligence model performs inference following a thought flow according to the target inference type.The response generation system can receive the response from the artificial intelligence model and provide it to the input subject.
[0115] The present invention aims to provide a response generation method and system capable of generating a response from an artificial intelligence model based on an inference pattern suitable for a query. More specifically, the response generation system according to the present invention generates classification criteria for distinguishing inference patterns from long-text inference information generated by an artificial intelligence model, and generates pair data corresponding to each of said classification criteria, and can identify a target inference pattern suitable for a query based on the generated classification criteria. Subsequently, the response generation system can perform response generation by the artificial intelligence model using a prompt that reflects the target inference pattern.
[0116] In the present invention, a query may include a question, command, problem, request, or a combination thereof that is input into an artificial intelligence model and becomes the subject of response generation. For example, the query may be an input related to at least one of mathematical problem solving, knowledge-based question answering, logical judgment, safety judgment, identification of user intent, document analysis, business support, and expert knowledge response. However, the types of queries described above are merely examples, and the query in the present invention may further include various forms of input for which a response can be generated using an artificial intelligence model.
[0117] In the present invention, the long-text inference information may include at least one of the development, review, verification, judgment, and conclusion derivation content of the inference generated by an artificial intelligence model in response to a query. Such long-text inference information may be included in the data used in the data generation method, and the data may include at least one of a query, long-text inference information, a response generated based on the long-text inference information, and correct answer information corresponding to the query.
[0118] Meanwhile, the response generation system according to the present invention generates classification criteria and pair data corresponding to each of the classification criteria from long text inference information, performs embedding on the pair data to generate embedding vectors corresponding to each of the classification criteria, and can perform clustering of the classification criteria based on semantic similarity between the embedding vectors. In addition, the response generation system according to the present invention identifies representative classification criteria based on the clustering results, generates rubric data for classifying inference patterns corresponding to the representative classification criteria, and can classify a plurality of inference patterns related to the long text inference information to correspond to at least one of the representative classification criteria using the rubric data.
[0119] In addition, the response generation system according to the present invention can generate data related to the classification results of the plurality of inference patterns. Here, the data related to the classification results may include at least one of the classification results of inference patterns corresponding to long-text inference information and an inference pattern analysis report explaining the classification results.
[0120] In the present invention, the artificial intelligence model may include at least one model that can be used to generate a response to a query or to generate classification criteria based on long-text inference information. For example, the artificial intelligence model may include at least one of a machine learning model, a deep learning model, a deep neural network model, a language model, a large-scale language model, a generative artificial intelligence model, a transformer-based model, a natural language processing model, and a multimodal model. However, the artificial intelligence model described above is merely an example, and the artificial intelligence model in the present invention may further include various models capable of processing at least one of a query, long-text inference information, an inference pattern, or a prompt.
[0121] The present invention aims to generate a response from an artificial intelligence model based on an inference pattern suitable for a query. Below, we will examine the response generation system according to the present invention in more detail together with the attached drawings. Figure 4 is a conceptual diagram illustrating the response generation system according to the present invention.
[0122] Meanwhile, as illustrated in FIG. 4, the response generation system (100) according to the present invention may include at least one of an input unit (101), an output unit (102), a communication unit (110), a storage unit (120), a data collection unit (130), a reference processing unit (140), an embedding unit (150), a clustering unit (160), a rubric generation unit (170), an inference pattern classification unit (180), a prompt configuration unit (190), and a control unit (195).
[0123] Meanwhile, according to one embodiment of the present specification, a response generation system according to the present invention may be implemented in the form of a computing system comprising one or more processors and a memory in which instructions executed by the one or more processors are stored. More specifically, the memory may store at least a plurality of inference types and an inference guide corresponding to each of the plurality of inference types, along with instructions for performing a response generation operation of the present invention. The one or more processors may be configured to determine an inference type corresponding to the query among the plurality of inference types stored in the memory based on context information of the query by executing instructions stored in the memory, input an inference guide corresponding to the determined inference type together with the query into an artificial intelligence model, and provide a response output from the artificial intelligence model.
[0124] Meanwhile, in this specification, the control unit may refer to a configuration that controls and coordinates the operation of each component (e.g., context analysis unit, inference type determination unit, input data generation unit, response receiving unit, response providing unit, etc.) within a response generation system or data generation system according to the present invention, and may be understood as referring to a concept corresponding to 'one or more processors' described in the claims. In this specification, the operation of the control unit may be defined by instructions stored in memory, and said instructions may be stored on a computer-readable recording medium and distributed in the form of a program according to the present invention.
[0125] Meanwhile, according to one embodiment of the present specification, the response generation method according to the present invention may be implemented in the form of a program that can be stored on a computer-readable recording medium. The terms "attribute information," "inference pattern," and "prompt" used in this paragraph are intended to refer to the same concept as "context information," "inference type," and "input data" used in other parts of the present specification, respectively, and such lexical diversification is intended to clarify that the other independent claims of the claims and the present independent claim are different lexical expressions of the same invention. The "computer-readable recording medium" described in the present specification may include at least one of ROM, RAM, flash memory, hard disk drive, solid-state drive, optical disk, magnetic disk, and any non-transient storage medium.
[0126] Meanwhile, according to one embodiment of the present specification, the input data of the present invention may be generated by combining data derived from different sources. More specifically, the query may be input by a user, and the inference guide may be determined by the response generation system based on the context information, and the input data may be generated by combining the query input by the user and the inference guide determined by the response generation system. That is, according to the present invention, the user only needs to input their query, and since the inference guide for it is determined independently by the response generation system by analyzing the context information, the user can enjoy the effects of the present invention without being aware of the existence or form of the inference guide. This source separation structure can function as a key difference distinguishing the present invention from conventional prompt engineering methods.
[0127] The above response generation system (100) acquires data including query and long text inference information, generates classification criteria for distinguishing multiple inference patterns related to long text inference information based on the data, and can generate pair data consisting of at least one classification criterion among the classification criteria and an inference pattern corresponding to the classification criterion. Additionally, the above response generation system (100) performs embedding on the pair data to generate embedding vectors corresponding to each of the classification criteria, performs clustering on the classification criteria based on the embedding vectors, and can identify representative classification criteria that represent each of the clusters among the classification criteria included in the clusters according to the clustering results based on a preset criterion.
[0128] Additionally, the response generation system (100) can perform a series of data processing flows, including generating rubric data for classifying inference patterns corresponding to the representative classification criteria, using the rubric data to classify a plurality of inference patterns related to the long text inference information so as to correspond to at least one of the representative classification criteria, and generating data related to the classification results of the plurality of inference patterns.
[0129] Additionally, the response generation system (100) can specify a target inference pattern based on attribute information corresponding to a query, construct a prompt using at least one of the target inference pattern and the query, and input the prompt into an artificial intelligence model to perform a data processing flow that receives a response based on the target inference pattern.
[0130] However, the components of the response generation system (100) according to the present invention are not necessarily limited thereto and may further include various hardware components that perform the same or similar roles as described in the description of the present specification.
[0131] Although not illustrated, the response generation system (100) according to the present invention may include one or more processors, and such processors may include one or more general-purpose processors and one or more special-purpose processors, for example, a digital signal processor, a tensor processing unit, a graphics processing unit, a neural network processing unit, an application integrated circuit, an application semiconductor, a field programmable gate array, a quantum processing unit, etc.
[0132] One or more processors may be configured to execute instructions stored in the storage unit (120), computer-readable instructions, and other instructions described herein. The response generation method and system according to the present invention may perform data processing described below by the cooperation of memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (120). Such memory may store at least one or more artificial intelligence models utilized to perform the response generation method according to the present invention.
[0133] Meanwhile, the input unit (101) can be configured in various ways as a means of data input. For example, the input unit (101) can be configured to receive user input. The input unit (101) can be configured to receive user input from a user terminal. Here, “receiving input” may mean receiving an input signal or a selection signal corresponding to the user’s input based on input made by the user through the configuration of the input unit provided in the user terminal.
[0134] Here, the user terminal may include at least one of a mobile phone, smartphone, notebook computer, portable computer, slate PC, tablet PC, ultrabook, desktop computer, digital broadcasting terminal, PDA, PMP, navigation, wearable device, for example, a watch-type terminal, a glass-type terminal, and an HMD.
[0135] In addition, the input unit (101) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.
[0136] The input unit (101) may also be referred to as a user interface module. The input unit (101) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (101).
[0137] Here, user input may include documents, text, images or videos, voice, etc. In this case, the response generation system (100) may further include a module that converts voice into text.
[0138] Next, the output unit (102) can output information through an output unit configuration provided in a user terminal linked to the response generation system (100) according to the present invention, such as a display unit, a touch screen, a speaker, etc. For example, the output unit (102) can output at least one page or service page linked to the response generation system (100) according to the present invention to the display unit of the user terminal. In addition, the output unit (102) does not necessarily mean a hardware means, but can be understood as a channel for outputting results to a user.
[0139] Next, the communication unit (110) may be connected via a wireless or wired network to a user terminal, a server, e.g., a central server, an external server, etc., a device, and at least one network, etc., to receive or transmit overall data and information necessary for the operation of the response generation system (100) according to the present invention.
[0140] The communication unit (110) can support various communication methods according to the communication standards of the communicating device. The processor can transmit and receive data with an external server and a user terminal using the communication unit.
[0141] For example, the communication unit (110) may be configured to communicate with a communication target using at least one of WLAN, Wi-Fi, Wi-Fi Direct, DLNA, WiBro, WiMAX, HSDPA, HSUPA, LTE, LTE-A, 5G, Bluetooth, RFID, infrared communication, UWB, ZigBee, NFC, Wi-Fi Direct, and Wireless USB technologies.
[0142] Next, the storage unit (120, or memory) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and accessed by at least one of one or more processors.
[0143] One or more computer-readable storage media may include volatile and non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (120) may be implemented using a single physical device, e.g., one optical, magnetic, organic, or other memory or disk storage device, whereas in other examples, the storage unit (120) may be implemented using two or more physical devices.
[0144] The storage unit (120) may include computer-readable instructions and additional data. The storage unit (120) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and at least some of the functions of the device and network.
[0145] Furthermore, at least a portion of the storage unit (120) may be a cloud storage or a cloud server. The storage unit (120) may store at least a portion of the data corresponding to user input received from the input unit (101) and the training data or training data set.
[0146] That is, the storage unit (120) is sufficient as a space where information necessary for the operation of the response generation system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.
[0147] Furthermore, the storage unit (120) can store a computer program including computer program instructions. Furthermore, the storage unit (120) can store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (195) when loaded into the processor of the system (100).
[0148] In one embodiment, the storage unit (120) may store at least one of a query, text inference information, a response generated based on text inference information, correct answer information, classification criteria, embedding vector, cluster information, representative classification criteria, rubric data, a classification result of an inference pattern corresponding to text inference information, data related to the classification result, attribute information, a target inference pattern, and a prompt.
[0149] Additionally, the storage unit (120) can store correspondence relationships between multiple queries and multiple inference patterns, and the information stored in the storage unit (120) can be used to specify a target inference pattern corresponding to a new query or to configure a prompt to be input into an artificial intelligence model.
[0150] Additionally, in the present invention, the database or the previously specified storage (300) may be included in the storage unit (120) or implemented as an external storage linked to the storage unit (120). The database (300) may store at least one of a query, long text inference information, a response generated based on the long text inference information, correct answer information, classification criteria, embedding vector, cluster information, representative classification criteria, rubric data, classification result of an inference pattern corresponding to the long text inference information, data related to the classification result, attribute information, target inference pattern, and prompt.
[0151] Additionally, the database (300) can store correspondence relationships between multiple queries and multiple inference patterns, and can be used to specify a target inference pattern corresponding to a new query or to configure a prompt to be input into an artificial intelligence model.
[0152] Next, the data collection unit (130) may be configured to perform the role of acquiring data including a query and long text inference information corresponding to the query.
[0153] For example, the data collection unit (130) may obtain a query from at least one of a user terminal, a server, or a database (300), and obtain long text inference information generated by an artificial intelligence model in response to the query. Additionally, the data collection unit (130) may further obtain at least one of a response generated in response to the query and correct answer information corresponding to the query.
[0154] The data collection unit (130) can obtain one long text inference information corresponding to one query, and can also obtain multiple long text inference information corresponding to each of multiple queries.
[0155] Additionally, the data collection unit (130) can acquire multiple long text inference information generated by different artificial intelligence models in response to the same query, thereby enabling comparison of inference patterns included in different long text inference information.
[0156] At this time, the data may include at least one of a query, long text inference information generated by an artificial intelligence model in response to the query, a response generated based on the long text inference information, and correct answer information corresponding to the query.
[0157] In the present invention, long-text inference information may refer to text data comprising a plurality of judgments, reviews, verifications, idea developments, and conclusion derivations performed by an artificial intelligence model to generate a response to a query. Such long-text inference information may include the development of inferences distinct from the final response and may serve as a subject for analyzing inference patterns.
[0158] Furthermore, the data collection unit (130) can organize the data such that at least one of the query, long text inference information, response, and correct answer information included in the data corresponds to each other. For example, the data collection unit (130) can organize one query and at least one of the long text inference information, response, and correct answer information corresponding to said one query into one data unit.
[0159] In the present invention, the data collection unit (130) may also be named a data acquisition unit, a data generation target acquisition unit, an inference information collection unit, or a long text inference information acquisition unit.
[0160] Next, the standard processing unit (140) may be configured to perform the role of generating classification criteria for distinguishing inference patterns based on long text inference information included in the data.
[0161] For example, the standard processing unit (140) can use data obtained through the data collection unit (130) to identify at least one of the thought process, judgment order, verification method, information utilization method, and conclusion derivation method included in the long text inference information. Based on the characteristics of the identified long text inference information, the standard processing unit (140) can generate classification criteria to distinguish different inference patterns.
[0162] Additionally, the reference processing unit (140) can generate pair data including at least one classification criterion and at least one inference pattern distinguished according to the classification criterion. For example, the reference processing unit (140) can generate pair data corresponding to a specific classification criterion, and the pair data may include a specific classification criterion and at least one inference pattern distinguished according to the specific classification criterion.
[0163] In the present invention, the classification criterion may refer to a criterion for distinguishing the inference method included in the long-text inference information. The classification criterion may be a criterion for distinguishing how the long-text inference information interprets a problem, utilizes information, develops a judgment, performs verification, and derives a conclusion.
[0164] Additionally, in the present invention, an inference pattern may refer to a type of long text inference information distinguished according to a classification criterion. At least one inference pattern distinguished according to the classification criterion may correspond to a single classification criterion, and the classification criterion and at least one inference pattern may be composed of pair data.
[0165] Furthermore, the standard processing unit (140) can generate classification criteria from multiple long text inference information. In this case, the standard processing unit (140) can standardize inference methods that appear repeatedly in different long text inference information or inference methods that are distinct from each other. Additionally, the standard processing unit (140) is not limited to the expressions included in the long text inference information itself, but can generate classification criteria based on the structure and judgment method of the long text inference information.
[0166] The classification criteria generated by the criteria processing unit (140) may include semantically similar criteria, and these classification criteria may be subject to clustering by the embedding unit (150) and the clustering unit (160).
[0167] In the present invention, the reference processing unit (140) may also be named a reference generation unit, an inference pattern reference generation unit, a classification reference generation unit, a pattern reference processing unit, or an inference reference generation unit.
[0168] Next, the embedding unit (150) may perform the role of generating embedding vectors corresponding to each of the classification criteria by performing embedding on the pair data generated by the reference processing unit (140).
[0169] For example, the embedding unit (150) may configure the pair data into a standard representation in the form of text and embedding the standard representation to generate embedding vectors corresponding to each of the classification criteria. In one embodiment, the pair data may include at least one of a classification criterion, at least one inference pattern corresponding to the classification criterion, and explanatory information related to the inference pattern.
[0170] Additionally, the embedding unit (150) can generate an embedding vector by using pair data including one classification criterion and at least one inference pattern distinguished according to the one classification criterion as the embedding target. Accordingly, the embedding vector may not merely represent the name of the criterion, but may also include semantic information related to the classification criterion and at least one inference pattern distinguished according to the classification criterion.
[0171] Meanwhile, according to one embodiment of the present specification, the step of verifying the context information may include extracting at least one of semantic information representing the meaning of the query and domain information representing the domain of the query from the query. More specifically, the semantic information may refer to data representing at least one of the core intent, requested action, and answer format being asked by the query, and may be extracted from the query through a natural language processing model or a semantic embedding model. The domain information may refer to data representing the subject area to which the query belongs (e.g., mathematics, coding, medical, legal, everyday conversation, etc.), and may be extracted from the query through a pre-trained domain classification model. The response generation system may generate the context information based on at least one of the extracted semantic information and domain information, and the generated context information may be used as input specific to a target inference type in a subsequent step.
[0172] In the present invention, an embedding vector may refer to vector data that numerically expresses the meaning of a classification criterion. The embedding vector may be used to calculate semantic similarity or difference between classification criteria, and may be used by the clustering unit (160) described below to classify semantically similar classification criteria into a single cluster.
[0173] Furthermore, the embedding unit (150) can generate multiple embedding vectors corresponding to each of the classification criteria and can place the generated multiple embedding vectors in the same semantic space. In this case, classification criteria having similar meanings can be placed at relatively close positions in the semantic space, and classification criteria having different meanings can be placed at relatively far positions in the semantic space.
[0174] Additionally, the embedding unit (150) may further generate an embedding vector for at least one of a query, long text inference information, rubric data, and a classification result of an inference pattern corresponding to the long text inference information. In this case, the embedding vector generated by the embedding unit (150) may be used to verify the correspondence between the query and the inference pattern, to calculate the similarity between a new query and an existing query, or to search for a classification result of an inference pattern corresponding to the long text inference information.
[0175] In the present invention, the embedding unit (150) may also be named an embedding vector generation unit, a vector generation unit, a vector conversion unit, an embedding generation unit, or a semantic expression generation unit.
[0176] Next, the clustering unit (160) may be configured to perform clustering for the classification criteria based on semantic similarity between a plurality of embedding vectors generated by the embedding unit (150).
[0177] For example, the clustering unit (160) receives an embedding vector corresponding to each classification criterion, calculates the distance or semantic similarity between the embedding vectors, and can classify classification criteria that are similar to each other into the same cluster based on the calculated semantic similarity. At this time, the clustering unit (160) can generate clusters using at least one method among merged hierarchical clustering, distance-based clustering, density-based clustering, or similarity-based clustering.
[0178] Additionally, the clustering unit (160) may calculate cluster evaluation values to determine the number of clusters and select an appropriate cluster structure based on the cluster evaluation values. Here, the cluster evaluation values may include at least one of the cohesion within the cluster, the separation between clusters, the silhouette score, and the distance distribution between clusters.
[0179] Furthermore, the clustering unit (160) may select a representative classification criterion that represents the cluster from among the classification criteria included in each cluster. For example, the clustering unit (160) may analyze the semantic similarity between embedding vectors within the cluster and specify a criterion with high semantic similarity to other embedding vectors, a criterion located at the center of the cluster, or a criterion having representative features within the cluster as a representative classification criterion.
[0180] At this time, embedding vectors corresponding to classification criteria may be used as input data, and cluster information, a list of classification criteria by cluster, and a representative classification criterion corresponding to each cluster may be generated as output data.
[0181] In the present invention, clustering can be understood as data processing for grouping semantically similar classification criteria into a single cluster and specifying representative classification criteria for each cluster. Through this, representative classification criteria used to generate rubric data or classify inference patterns can be generated.
[0182] Additionally, the clustering unit (160) can configure hierarchical relationships between clusters as needed, thereby expressing the relationship between the classification criteria of the upper concept and the classification criteria of the lower concept.
[0183] In the present invention, the clustering unit (160) may also be named a reference clustering unit, a pattern clustering unit, a clustering unit, or a similarity-based grouping unit.
[0184] Next, the rubric generation unit (170) may be configured to perform the role of generating rubric data to distinguish inference patterns corresponding to representative classification criteria derived by the clustering unit (160).
[0185] For example, the rubric generation unit (170) may receive an inference pattern corresponding to a representative classification criterion and generate rubric data including at least one of definition information for the inference pattern, feature information for distinguishing the inference pattern, and example information for the inference pattern. At this time, the rubric generation unit (170) may configure the rubric so that at least one inference pattern corresponding to a representative classification criterion can be distinguished.
[0186] Additionally, the rubric generation unit (170) may generate rubric data to include explanatory information corresponding to each inference pattern, and the explanatory information may include at least one of the inference structure, information utilization method, verification procedure, error correction method and conclusion derivation method included in the long text inference information.
[0187] At this time, the rubric generation unit (170) can configure the rubric data in a form that includes at least one of definition information, feature information, and example information for each inference pattern so as to determine whether the text inference information corresponds to a specific inference pattern.
[0188] Additionally, the rubric generation unit (170) can generate rubric data for distinguishing inference patterns by using at least one of a representative classification criterion, cluster information, and an inference pattern corresponding to the representative classification criterion.
[0189] In the present invention, rubric data can be understood as data including a judgment criterion for distinguishing inference patterns corresponding to a specific representative classification criterion. By including at least one of a definition, characteristic, and example of an inference pattern, the rubric data can be used as a criterion to determine which inference pattern the long text inference information corresponds to.
[0190] Additionally, the rubric generation unit (170) may generate rubric data that further includes example information corresponding to each inference pattern as needed. In this case, the example information may include examples of long text inference information to which a specific inference pattern is applied, thereby improving the understanding of the inference pattern.
[0191] Furthermore, the rubric generation unit (170) can generate multiple rubric data corresponding to each of the multiple representative classification criteria, and can generate rubric data including various inference patterns according to different classification criteria.
[0192] In the present invention, the rubric generation unit (170) may also be named a rubric generation unit, an inference pattern judgment criterion generation unit, or a pattern distinction criterion generation unit.
[0193] Next, the inference pattern classification unit (180) may be configured to perform the role of classifying inference patterns corresponding to long text inference information included in the data using rubric data generated by the rubric generation unit (170).
[0194] For example, the inference pattern classification unit (180) receives long text inference information and, using at least one of the definition information, feature information, and example information included in the rubric data, can determine an inference pattern corresponding to the long text inference information among a plurality of inference patterns.
[0195] Additionally, the inference pattern classification unit (180) can identify an inference pattern that satisfies a selection condition among a plurality of inference patterns defined in the rubric data, and, if necessary, may determine that a plurality of inference patterns are applied simultaneously to a single long text inference information. For example, a portion of the long text inference information may be classified as corresponding to a first inference pattern, and another portion as corresponding to a second inference pattern.
[0196] At this time, the inference pattern classification unit (180) can generate at least one of the classification result of an inference pattern corresponding to the long text inference information, inference pattern identification information, feature information for each inference pattern, and difference information between inference patterns by using at least one of the long text inference information and rubric data.
[0197] In the present invention, inference pattern classification may refer to determining which type among a plurality of inference patterns an inference structure or way of thinking included in long-text inference information corresponds to. Such inference pattern classification can be understood as a process of interpreting the structural characteristics of long-text inference information and correlating them with a standardized inference pattern system, through which differences between different long-text inference information can be structurally compared or analyzed.
[0198] Additionally, the inference pattern classification unit (180) can analyze at least one of the distribution of inference patterns corresponding to a specific query type, the transition relationship between inference patterns, the relationship between inference patterns and responses, and the correspondence relationship between inference patterns and response performance information by repeatedly performing inference pattern classification on multiple long text inference information.
[0199] Furthermore, the inference pattern classification unit (180) can generate an inference pattern analysis report that explains the characteristics of the long text inference information based on the classification result of the inference pattern. For example, the inference pattern analysis report may include at least one of name information of the inference pattern corresponding to the long text inference information, basis information classified as an inference pattern, section information corresponding to the inference pattern in the long text inference information, and information explaining the characteristics of the inference pattern.
[0200] In the present invention, the inference pattern classification unit (180) may also be named a pattern classification unit, an inference structure analysis unit, a pattern determination unit, a rubric-based classification unit, or an inference pattern analysis unit.
[0201] Next, the prompt configuration unit (190) may be configured to perform the role of configuring a prompt to be input into an artificial intelligence model by reflecting a target inference pattern corresponding to the query.
[0202] For example, the prompt configuration unit (190) may receive a target inference pattern specified by the control unit (195) and a query obtained through the input unit (101) or the data collection unit (130), and may generate a prompt that instructs inference according to the target inference pattern. At this time, the prompt configuration unit (190) may configure the prompt to include instruction information or explanatory information that reflects at least one of an inference method, an idea development method, a verification procedure, and a conclusion derivation method corresponding to the target inference pattern.
[0203] Additionally, the prompt configuration unit (190) can configure a prompt that instructs an artificial intelligence model to perform inference according to a target inference pattern by using rubric data generated by the rubric generation unit (170) or inference pattern information derived by the inference pattern classification unit (180). For example, the prompt configuration unit (190) can generate an instruction sentence that includes an explanation related to an inference method or inference instruction information corresponding to a specific inference pattern and include it in the prompt.
[0204] At this time, the prompt configuration unit (190) can configure a prompt that reflects a target inference pattern suitable for the query based on at least one of attribute information of the query, association information between the query and a previously stored reference query, or a correspondence relationship between a target inference pattern and the query.
[0205] At this time, at least one of a query, target inference pattern information, rubric data, a classification result of an inference pattern corresponding to long text inference information, and attribute information of the query may be used as input data, and prompt data that can be input to an artificial intelligence model may be generated as output data.
[0206] In the present invention, a prompt can be understood as input data for enabling an artificial intelligence model to perform inference in a specific manner, comprising instructional information, explanatory information, or constraints provided along with a query. Such a prompt can serve the role of enabling the artificial intelligence model to generate a response based on inference according to a target inference pattern.
[0207] Additionally, the prompt configuration unit (190) can generate multiple prompts that reflect different target inference patterns for the same query, and can be used to generate or compare responses according to different inference methods.
[0208] Furthermore, the prompt configuration unit (190) can transmit the generated prompt to an artificial intelligence model through the communication unit (110) so that the artificial intelligence model can generate a response based on a target inference pattern.
[0209] In the present invention, the prompt configuration unit (190) may also be named a prompt generation unit, an inference control input generation unit, a pattern reflection input configuration unit, an artificial intelligence model input configuration unit, or an inference induction input generation unit.
[0210] Next, the control unit (195) can perform the role of controlling the overall operation of the response generation system (100) related to the present invention. The control unit (195) can process signals, data, information, etc. that are input or output through the components of the response generation system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (195) can be physically implemented by the processor described above.
[0211] Meanwhile, as illustrated in FIG. 5, the control unit (195) can generate classification criteria based on text inference information related to a query and structure inference patterns corresponding to the classification criteria to generate data related to classification results used for data generation and response generation.
[0212] More specifically, the control unit (195) can obtain data including long-form inference information related to at least one query. For example, as illustrated in FIG. 5, the long-form inference information may include the development, review, and conclusion derivation of inferences generated by an artificial intelligence model in response to a query, such as “The question says ...”, “let me make sure ...”, and “The answer is ...”. The control unit (195) can obtain basic data for distinguishing inference patterns using this long-form inference information.
[0213] The control unit (195) can generate classification criteria to distinguish multiple inference patterns related to long text inference information based on data. For example, as illustrated in FIG. 5, the control unit (195) can generate classification criteria such as analysis perspective, verification focus, clarification method, inference type, and conclusion derivation method in the long text inference information. Here, the classification criteria may mean criteria for distinguishing the inference method included in the long text inference information. More specifically, the classification criteria may mean criteria for distinguishing the method of interpreting the problem, the method of utilizing information, the method of developing judgments, the method of performing reviews, and the method of presenting conclusions in the long text inference information.
[0214] The control unit (195) can generate pair data consisting of at least one classification criterion among classification criteria and an inference pattern corresponding to said classification criterion. In the present invention, pair data may refer to information composed of at least one classification criterion and at least one inference pattern distinguished according to said classification criterion as a single data unit. For example, as shown in FIG. 5, a top-down approach and a bottom-up approach may correspond to a classification criterion called analysis perspective, and an explicit review method and an implicit review method may correspond to a classification criterion called verification focus. However, these classification criteria and inference patterns are merely examples, and the control unit (195) can generate pair data including different classification criteria and inference patterns depending on the content of the long text inference information.
[0215] The control unit (195) can perform embedding on the pair data to generate embedding vectors corresponding to each of the classification criteria. For example, as illustrated in FIG. 5, the control unit (195) can place the classification criteria and inference patterns in an embedding space and calculate the semantic similarity between the embedding vectors. Here, the embedding vector may refer to data that expresses the meaning of the classification criteria and inference patterns in vector form. By using the entire pair data as the embedding target, the control unit (195) can reflect not only the name of the classification criteria but also the meaning of the classification criteria and the corresponding inference pattern.
[0216] The control unit (195) can perform clustering of classification criteria based on embedding vectors. For example, as illustrated in FIG. 5, the control unit (195) can hierarchically cluster inference patterns or classification criteria placed in the embedding space and process them so that semantically similar classification criteria are included in the same cluster. Here, clustering may mean data processing to group semantically similar classification criteria into one cluster and to specify a representative classification criterion for each cluster.
[0217] The control unit (195) can specify representative classification criteria that represent each of the clusters among the classification criteria included in the clusters according to the clustering results based on preset criteria. For example, as illustrated in FIG. 5, the control unit (195) can specify a classification criterion that represents one cluster in the embedding space as a representative classification criterion. In this case, the control unit (195) can specify a representative classification criterion using at least one of semantic similarity between embedding vectors within the cluster, distance from the cluster center, and frequency of occurrence within the cluster.
[0218] The control unit (195) can generate rubric data for classifying inference patterns corresponding to representative classification criteria. For example, as illustrated in FIG. 5, the control unit (195) can generate rubric data to distinguish between a top-down approach and a bottom-up approach corresponding to an analysis perspective, which is a representative classification criterion. The rubric data may include at least one of definition information for an inference pattern, feature information for distinguishing an inference pattern, and example information for an inference pattern. For example, the rubric data may include descriptions for distinguishing each inference pattern, such as “A top-down approach is an inference pattern that shows a clear top-level approach,” and “A bottom-up approach is an inference pattern that focuses on specific content.”
[0219] The control unit (195) can classify multiple inference patterns related to long text inference information using rubric data so that they correspond to at least one of representative classification criteria. For example, as illustrated in FIG. 5, the control unit (195) can classify long text inference information as corresponding to a top-down approach when it establishes an overall structure and then develops detailed content, and as corresponding to a bottom-up approach when it accumulates detailed facts to reach a conclusion.
[0220] The control unit (195) can generate data related to the classification results of a plurality of inference patterns. For example, as illustrated in FIG. 5, the control unit (195) can generate an inference pattern report. Here, the data related to the classification results may include at least one of the classification results of inference patterns corresponding to long text inference information and an inference pattern analysis report explaining said classification results. Additionally, the inference pattern analysis report may include at least one of the name information of the inference pattern corresponding to the long text inference information, the basis information classified as an inference pattern, the interval information corresponding to the inference pattern in the long text inference information, and the feature description information of the inference pattern.
[0221] In this way, the control unit (195) can generate classification criteria and pair data from the text inference information, and perform embedding, clustering, identification of representative classification criteria, generation of rubric data, and classification of inference patterns for the pair data. Accordingly, the control unit (195) can structure the inference method included in the text inference information into classification criteria, inference patterns, rubric data, and data related to the classification results. This structured data can be used to identify a target inference pattern for a new query or to construct a prompt according to the target inference pattern.
[0222] Meanwhile, as described above, the present invention provides a response generation system (100) that generates a response from an artificial intelligence model based on an inference pattern suitable for a query. The response generation system (100) according to the present invention generates classification criteria and pair data for distinguishing a plurality of inference patterns from long text inference information, and can store rubric data generated based on the classification criteria and data related to the classification result. In addition, the response generation system (100) can identify a target inference pattern corresponding to the query among a plurality of inference patterns based on attribute information corresponding to the query, and can generate a response to the query using a prompt that reflects the target inference pattern.
[0223] Meanwhile, in the present invention, “collection” may mean receiving or obtaining a query from an external device, user terminal, server, database, or storage unit. Accordingly, “collecting at least one query” may be understood as having the same meaning as or corresponding to “obtaining at least one query.”
[0224] In the present invention, “specific” and “determine” may refer to a process of selecting or identifying one or at least one of a plurality of candidates based on input information. Accordingly, “specificating attribute information corresponding to a query” can be understood as having the same meaning as or equivalent to “verifying attribute information corresponding to a query.” Furthermore, “determining a target inference pattern” can be understood as having the same meaning as or equivalent to “specificating a target inference pattern.”
[0225] In the present invention, “pre-set classification criteria” may refer to reference information that is stored or generated in advance and used to distinguish a plurality of inference patterns from one another. Here, the pre-set classification criteria may include at least one of a classification criterion generated based on long-text inference information, a representative classification criterion, reference information included in rubric data, and reference information for distinguishing inference patterns.
[0226] In the present invention, “pre-set verification conditions” may refer to condition information that is pre-defined or set and used to determine whether data generated in each processing step included in the data generation method of the present invention is suitable for use in a subsequent processing step. Here, the pre-set verification conditions may refer to condition information that must satisfy at least one of pair data, embedding vector (600), cluster, representative classification criteria (800), rubric data (900), inference pattern classification result, and inference pattern analysis report, and may include at least one of data composition requirements, data item inclusion requirements, semantic correspondence relationship requirements, and inference structure correspondence requirements. In the present invention, “pre-set verification conditions” are information having a meaning distinct from “pre-set criteria” described in the claims, and should be understood as being distinguished from “pre-set criteria” for specifying representative classification criteria (800).
[0227] In the present invention, “a plurality of inference patterns configured to have different inference characteristics” may refer to a plurality of types of inference methods that can be used to generate responses to the same query or different queries. More specifically, the plurality of inference patterns may have inference characteristics that are distinguished from one another in at least one of an analysis perspective, a verification focus, an idea development method, an inference type, a clarification approach, a conclusion drawing method, and a calculation execution method.
[0228] In the present invention, “input data” may refer to data input into an artificial intelligence model to induce the generation of a response. Accordingly, “input data” in the program clause may be understood as having the same meaning as or a corresponding meaning to “prompt” in the response generation method. For example, the input data may include at least one of a query, a target inference pattern, inference instruction information, information combining the query and the target inference pattern, and prompt data that can be input into the artificial intelligence model.
[0229] In the present invention, “receiving a response” may mean that a system or electronic device receives a response generated by an artificial intelligence model. Therefore, “receiving a response from an artificial intelligence model” may be understood as having the same meaning as or corresponding to “receiving a response from an artificial intelligence model.”
[0230] The terms “instruction,” “operation,” and “execution” described in the claims of this specification may mean that each process described in the response generation method is implemented to be executable by one or more processors. Accordingly, the processes described in the response generation method, such as query acquisition, attribute information verification, target inference pattern identification, prompt configuration, prompt input, and response receiving, may be implemented as processes performed by a processor of the system or as operations executed by instructions of a program.
[0231] Hereinafter, together with the attached drawings, we will examine in more detail the data generation method included in the response generation system (100) according to the present invention. FIG. 6 is a flowchart for explaining the data generation method of the response generation system according to the present invention, and FIGS. 7, FIGS. 8, FIGS. 9, FIGS. 10, FIGS. 11, FIGS. 12 and FIGS. 13 are conceptual diagrams for explaining the data generation method of the response generation system according to the present invention.
[0232] Meanwhile, as illustrated in FIG. 6, the data generation method according to the present invention may include the steps of: acquiring data including long text inference information related to at least one query (S610); generating classification criteria for distinguishing a plurality of inference patterns related to long text inference information based on the data (S620); generating pair data consisting of at least one classification criterion among the classification criteria and an inference pattern corresponding to the classification criterion (S630); performing embedding on the pair data to generate embedding vectors corresponding to each of the classification criteria (S640); performing clustering on the classification criteria based on the embedding vectors (S650); specifying representative classification criteria representing each of the clusters among the classification criteria included in the clusters according to the clustering result based on a preset criterion (S660); generating rubric data for classifying inference patterns corresponding to the representative classification criteria (S670); and using the rubric data to classify a plurality of inference patterns related to long text inference information so as to correspond to at least one of the representative classification criteria (S680). There is. Subsequently, the present invention may proceed with the step (S690) of generating data related to the classification results of a plurality of inference patterns.
[0233] In the present invention, there may be various methods for acquiring data including the long text inference information.
[0234] For example, as illustrated in FIG. 7, the control unit (195) may obtain data (450) comprising at least one query (411, 412, 413) from at least one of various sources and at least one long text inference information (410, 420, 430) corresponding to each of the queries. In this case, the data (450) may represent input data for performing inference pattern analysis based on long text inference information.
[0235] In the present invention, a query (411, 412, 413) may refer to natural language-based request information received by an artificial intelligence model to generate a response. More specifically, the query may include at least one of a problem-solving request, a calculation request, an explanation request, a comparison request, an inference request, a summary request, a safety-related request, and a request to generate a long response.
[0236] In addition, in the present invention, long text inference information (410, 420, 430) may refer to information including inference content performed by an artificial intelligence model while generating a response to a query. More specifically, long text inference information may refer to inference information in the form of a long text that includes at least one of thought flow information (or thinking trajectory), long text inference information (or long chain-of-thought), intermediate inference information, inference basis information, calculation execution information, verification execution information, and conclusion derivation information.
[0237] Here, thought flow information may refer to information containing multiple inference contents performed by an artificial intelligence model to generate a response to a query, arranged in a temporal or logical order. Additionally, long-text inference information may refer to long-text inference information containing multiple inference sentences, multiple inference structures, and multiple verification structures.
[0238] In addition, in the present invention, the inference structure may refer to the configuration of the inference development method, explanation linkage method, verification execution method, and conclusion derivation method included in the long text inference information. More specifically, the inference structure may include at least one of a sequential development structure, a parallel review structure, a verification-centered structure, a conclusion-first structure, and a stepwise calculation structure.
[0239] The present invention does not use simple response results or accuracy itself as the subject of analysis, but rather uses the inference content and inference structure included in the long text inference information itself as the subject of analysis. That is, the present invention can obtain data (450) for analyzing the inference structure used when an artificial intelligence model generates a response to a query.
[0240] In one embodiment, the control unit (195) may obtain a response generated from an artificial intelligence model and long text inference information used to generate the response. In another embodiment, the control unit (195) may obtain data (450) from a public data set containing long text inference information. In yet another embodiment, the control unit (195) may obtain data (450) from an external server, database, benchmark providing device, or long text inference generating device.
[0241] For example, the control unit (195) can obtain data (450) from at least one of a mathematical problem-solving data set, a logical reasoning data set, a scientific problem-solving data set, a safety-related query data set, a question-and-answer data set, a code generation data set, and an interactive response data set. More specifically, the control unit (195) can obtain data (450) from a data set including GPQA-Diamond, MMLU-Redux, MATH-500, Arena-Hard, XSTest, and WildGuard.
[0242] However, the present invention is not limited to a specific data set. In one embodiment, the control unit (195) may acquire data (450) including query and long-text inference information related to a specific domain. In another embodiment, the control unit (195) may acquire data corresponding to a plurality of domains by integrating them. In yet another embodiment, the control unit (195) may construct data (450) using user query logs and response logs of an artificial intelligence model.
[0243] Additionally, in the present invention, data (450) may include at least one of a query, text inference information, an answer generated based on text inference information, and correct answer information corresponding to the query.
[0244] For example, as illustrated in FIG. 7, the data may include a first query (411) and first long-word inference information (410) corresponding to the first query (411), a second query (412) and second long-word inference information (420) corresponding to the second query (412), and a third query (413) and third long-word inference information (430) corresponding to the third query (413). That is, the present invention can obtain data (450) in a structure in which long-word inference information corresponding to each query is connected.
[0245] In one embodiment, the data (450) may include a query, text inference information, and a response. In another embodiment, the data (450) may include all of the query, text inference information, response, and correct answer information. In yet another embodiment, the data (450) may further include at least one of model identification information corresponding to the text inference information, response generation time information, used prompt information, and model type information.
[0246] Here, the correct answer information may refer to standard response information for a query. More specifically, the correct answer information may include at least one of a standard response verified by a human evaluator, a standard response included in a benchmark data set, and a standard response based on multiple verification results. In the present invention, the correct answer information may be used as information for evaluating the response quality itself, as comparison standard information for calculating response performance information by inference pattern, or as at least one of information for analyzing the correspondence between long-text inference information and inference structures.
[0247] Additionally, the control unit (195) can collect data (450) using the data collection unit (130). Here, the data collection unit (130) may refer to a component for collecting data including a query, text inference information, a response, and correct answer information.
[0248] In one embodiment, the control unit (195) may exclude data that does not contain long sentence inference information from the analysis target. For example, the control unit (195) may exclude response data that contains only simple short sentence responses and does not contain long sentence inference information from the analysis target. In another embodiment, the control unit (195) may exclude data in which the length of the long sentence inference information is less than a preset standard length from the analysis target.
[0249] Additionally, the control unit (195) may store the acquired data (450) in the storage unit (120) or a previously specified storage unit (300). Here, the previously specified storage unit (300) may refer to a storage area for storing data used for long-text inference information-based inference pattern analysis.
[0250] In one embodiment, the control unit (195) may store the query and long-text inference information in the form of a single data sample. In another embodiment, the control unit (195) may store the query, long-text inference information, response, and correct answer information in a connected structure. In yet another embodiment, the control unit (195) may store the data (450) by separating it by domain, model, query type, or inference structure of the long-text inference information.
[0251] In addition, acquiring data (450) in the present invention can be understood as collecting base data for analyzing the inference content and inference structure contained in the long-text inference information itself, and configuring the reasoning behavior contained in the long-text inference information into an analyzable form. In particular, by using the long-text inference information itself as the subject of analysis rather than simple response results, the present invention can establish a data base for analyzing differences in the inference structure and inference method of an artificial intelligence model.
[0252] Subsequently, the control unit (195) can generate classification criteria (500) for distinguishing multiple inference patterns related to long text inference information based on the data (450) (see S620 in FIG. 6).
[0253] For example, as illustrated in FIG. 8, the control unit (195) can generate a classification criterion (500) for distinguishing a plurality of inference patterns (550) included in the long-text inference information by using data (450) including at least one query (411, 412, 413) and long-text inference information (410, 420, 430). Here, the classification criterion (500) may mean a criterion information for distinguishing differences in the inference structures included in the long-text inference information.
[0254] In the present invention, the classification criterion (500) may be a concept distinct from the inference pattern itself. The classification criterion (500) may refer to a perspective or standard for distinguishing the inference structure included in the long text inference information, and the inference pattern may refer to a specific type of inference method distinguished according to the classification criterion (500).
[0255] For example, depending on the analytical perspective, fact-based analysis-centered reasoning and overall structure-based analysis-centered reasoning can be distinguished as different reasoning patterns. Additionally, depending on the verification focus criteria, data-based verification-centered reasoning and hypothesis-based verification-centered reasoning can be distinguished as different reasoning patterns.
[0256] Additionally, in the present invention, the classification criterion (500) may be a concept distinct from simple topic classification information or response type classification information. More specifically, the classification criterion (500) may refer to reference information for distinguishing the difference between at least one of the inference structure, explanation structure, verification structure, judgment structure, and conclusion derivation structure included in the long text inference information.
[0257] Here, the inference structure may refer to the configuration of the reasoning development method, explanation connection method, verification execution method, and conclusion derivation method included in the long-text inference information. The explanation structure may refer to the method in which evidence, examples, comparisons, and conclusions are connected within the long-text inference information. Additionally, the verification structure may refer to the method in which intermediate judgment verification, error possibility review, and alternative review are performed within the long-text inference information. Furthermore, the judgment structure may refer to the method in which evidence is selected and connected to reach a specific conclusion within the long-text inference information.
[0258] For example, the control unit (195) can analyze the inference structure contained in the text inference information itself to generate at least one classification criterion (500) among an analysis perspective criterion, an access scope criterion, an inference type criterion, an idea development criterion, a verification focus criterion, and a clarification access criterion.
[0259] In one embodiment, the control unit (195) may generate an analysis perspective criterion to distinguish whether the long-text inference information develops details after establishing an overall structure or reaches a conclusion by accumulating details. In another embodiment, the control unit (195) may generate an access scope criterion to distinguish whether the long-text inference information is organized around a single solution path or examines multiple solution paths together. In yet another embodiment, the control unit (195) may generate an inference type criterion to distinguish whether the long-text inference information derives a general conclusion from individual cases or applies general principles to specific cases.
[0260] Additionally, in one embodiment, the control unit (195) may generate an idea development criterion to distinguish whether the long text inference information develops a single idea sequentially or examines multiple ideas together. In another embodiment, the control unit (195) may generate a verification focus criterion to distinguish whether the long text inference information performs a judgment based on data or evidence or performs a judgment by setting a hypothesis and examining the hypothesis. In yet another embodiment, the control unit (195) may generate a clarification approach criterion to distinguish whether the long text inference information repeatedly resolves uncertainty during inference or is structured around drawing a direct conclusion.
[0261] Additionally, the control unit (195) may generate at least one classification criterion (500) among safety priority criteria, ethical consideration criteria, content processing criteria, risk assessment criteria, and content intent criteria for the text inference information of a safety-related query. For example, the control unit (195) may generate a safety priority criterion to distinguish whether the text inference information is structured around risk prevention or around directly processing risk factors. In another embodiment, the control unit (195) may generate an ethical consideration criterion to distinguish whether the text inference information is centered on technical description or ethical judgment.
[0262] Additionally, the control unit (195) may additionally generate at least one classification criterion (500) among a map strategy criterion and an idea development criterion for the long-form inference information of a safety-related query. Here, the map strategy criterion may refer to criterion information for distinguishing whether the long-form inference information is configured to directly instruct a user to perform a specific action or to provide information so that the user can make a judgment.
[0263] In addition, content processing criteria may refer to standard information for distinguishing whether long-text inference information is structured to block potentially dangerous content or to publicly review potentially dangerous content.
[0264] Additionally, the control unit (195) can generate a plurality of inference patterns (550) corresponding to the content intent criteria, such as a benevolent intent inference pattern intended to generate a response that is helpful to the user and a malicious intent inference pattern intended to generate a response that is likely to cause harm to the user or a third party.
[0265] Additionally, the control unit (195) may generate a plurality of inference patterns (550) corresponding to safety priority criteria, such as a risk prevention-centered inference pattern that prioritizes risk prevention and a risk direct processing-centered inference pattern that directly accesses risk factors. Additionally, the control unit (195) may generate a high-risk recognition inference pattern that infers by assuming a high risk probability and a low-risk assumption inference pattern that infers by assuming a low risk probability, such as a plurality of inference patterns (550) corresponding to risk evaluation criteria.
[0266] As such, the classification criteria (500) corresponding to the text inference information of the safety-related query in the present invention may include a map strategy criterion, an ethical consideration criterion, a safety priority criterion, an idea development criterion, a content processing criterion, a risk assessment criterion, and a content intent criterion, and a plurality of inference patterns (550) corresponding to each classification criterion (500) may have different response performance information when generating a safety-related response.
[0267] Additionally, the control unit (195) can generate user understanding criteria for long-text inference information regarding instructions to be executed. Here, the user understanding criteria may refer to criteria information for distinguishing whether the long-text inference information reflects at least one of the user intent, constraints, request format, and response purpose included in the query.
[0268] For example, as illustrated in FIG. 8, the control unit (195) may generate a first classification criterion (510), a second classification criterion (520), and a third classification criterion (530). In one embodiment, the first classification criterion (510) may be a criterion for distinguishing differences in the perspective of analysis. In another embodiment, the second classification criterion (520) may be a criterion for distinguishing differences in the focus of verification. In yet another embodiment, the third classification criterion (530) may be a criterion for distinguishing differences in the method of developing ideas.
[0269] Here, the first classification criterion (510), the second classification criterion (520), and the third classification criterion (530) may be criteria for distinguishing different inference structures. That is, the present invention may not analyze long text inference information using only one criterion, but may generate multiple classification criteria (500) for analyzing the inference structures included in the long text inference information from multiple different perspectives.
[0270] In addition, in the present invention, the classification criteria (500) may be generated based on the long text inference information itself, rather than being generated solely from a predefined set of criteria. More specifically, the control unit (195) may generate the classification criteria (500) by analyzing the inference structure, explanation structure, verification structure, judgment structure, and conclusion derivation structure that appear repeatedly in the long text inference information.
[0271] In one embodiment, the control unit (195) may analyze a plurality of sentences included in the long text inference information and extract inference structure features based on the connection structure between sentences. In another embodiment, the control unit (195) may generate a classification criterion (500) by analyzing at least one of a judgment expression, a calculation expression, a comparison expression, a verification expression, and a conclusion expression included in the long text inference information. In yet another embodiment, the control unit (195) may extract common inference features based on inference structure features that are repeatedly included in the plurality of long text inference information and generate a classification criterion (500) based on the extracted common inference features.
[0272] Additionally, the control unit (195) can generate a classification criterion (500) using an artificial intelligence model. The control unit (195) can generate a classification criterion (500) by configuring a prompt that includes at least one of query and long text inference information and inputting the prompt into an artificial intelligence model.
[0273] In this case, the control unit (195) can configure the prompt to generate multiple inference structures included in the long text inference information in a form that is distinguishable from one another. For example, the control unit (195) can configure the prompt to standardize the inference structures actually included in the long text inference information by the artificial intelligence model and to limit the generation of semantically redundant standards.
[0274] In one embodiment, the control unit (195) may configure a prompt so that the name of the classification criterion (500) reflects the inference structure included in the long text inference information. In another embodiment, the control unit (195) may configure a prompt so that a plurality of inference patterns corresponding to the classification criterion (500) include an inference structure that distinguishes them from one another. In yet another embodiment, the control unit (195) may configure a prompt so that a plurality of inference patterns corresponding to the classification criterion (500) include different explanation structures or verification structures. Such a prompt may be configured as in the following embodiment (see Table 1 below).
[0275] You are tasked with analyzing the reasoning patterns used in the following response. The response includes the thought process for solving a problem. Your goal is to extract and describe patterns based on various criteria that characterize the model's problem-solving strategy.Please follow these guidelines:1. Identify multiple *meaningful criteria* that differentiate reasoning patterns. Each criterion should have a clear and descriptive name that reflects a real aspect of the reasoning process. **Do not use generic placeholders like 'Criterion 1'.**2. For each criterion, describe two contrasting *pattern types* (e.g., *Step-by-step* vs. *Outcome-first*, or *Concrete* vs. *Abstract*).3. Present your analysis in the following format, using and tags to enclose the list:Descriptive Criterion Name (Pattern A vs. Pattern B)Descriptive Criterion Name (Pattern A vs. Pattern B)...Descriptive Criterion Name (Pattern A vs. Pattern B)4. Do not include any explanations or commentary within the<patterns>tags.5. The example format above is only a guide. You are encouraged to define your own diverse and insightful pattern criteria based on the given response.Response: {answer}
[0276] In addition, generating a classification criterion (500) in the present invention can be understood as generating a criterion for analyzing differences in the inference structure included in the long-text inference information itself, rather than classifying the long-text inference information based on a simple topic criterion.
[0277] In particular, the present invention generates a classification criterion (500) based on the inference structure included in the text inference information itself, and can also analyze the inference structure appearing in a new query, a new model, or a new domain.
[0278] In addition, the present invention can generate a classification criterion (500) for distinguishing different inference patterns based on a plurality of inference structures included in the long text inference information. Accordingly, the control unit (195) can generate criterion information for analyzing the difference in inference structures included in the long text inference information, rather than the response result itself.
[0279] Additionally, the control unit (195) may store the generated classification criteria (500) in the storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of the classification criteria (500), classification criteria description information, inference structure information, and inference pattern distinction information.
[0280] Additionally, the control unit (195) is not limited to deriving only a few criteria through the generation of a single classification criterion (500), but can derive multiple candidate classification criteria by repeatedly performing the operation of generating classification criteria (500) for multiple long text inference information (410, 420, 430, etc.). In one embodiment, the control unit (195) can accumulate candidate classification criteria corresponding to different perspectives by repeatedly performing input to an artificial intelligence model for the same data (450). In another embodiment, the control unit (195) can derive candidate classification criteria from each of the long text inference information corresponding to different queries (411, 412, 413, etc.) and integrate the derived candidate classification criteria to form a set of candidate classification criteria.
[0281] In this case, the number of candidate classification criteria included in the set of candidate classification criteria may range from tens to hundreds, and the set of candidate classification criteria may include candidate classification criteria that express semantically overlapping or similar inference structures. In the present invention, constructing a set of candidate classification criteria can be understood as broadly collecting various inference structures contained within the long text inference information itself in the form of candidates, without relying on a limited set of criteria defined in advance.
[0282] Additionally, the control unit (195) may perform a subsequent operation to ensure that candidate classification criteria that express semantically overlapping or similar inference structures among the candidate classification criteria included in the set of candidate classification criteria are included in the same group. More specifically, the control unit (195) may identify a small number of representative classification criteria (800) that express inference structures distinct from each other from the set of candidate classification criteria by performing embedding and clustering on the candidate classification criteria included in the set of candidate classification criteria. In one embodiment, the number of representative classification criteria (800) identified from the set of candidate classification criteria may be less than the number of candidate classification criteria included in the set of candidate classification criteria, and may be a number corresponding to the analysis perspective, access scope, inference type, idea development, verification focus, and clarification approach.
[0283] In this way, the present invention specifies representative classification criteria (800) through a process that reduces semantic redundancy from a set of candidate classification criteria, thereby allowing for the extensive collection of various inference structures included in long text inference information, while also organizing the number of criteria used for inference pattern classification to a level that is interpretable by the user.
[0284] Subsequently, the control unit (195) can generate pair data consisting of at least one classification criterion (500) among the classification criteria (500) and an inference pattern corresponding to the classification criterion (500) (see S630 in FIG. 6).
[0285] For example, as illustrated in FIG. 8, the control unit (195) can generate a plurality of inference patterns (550) corresponding to each of the first classification criterion (510), the second classification criterion (520), and the third classification criterion (530), and generate pair data connecting the classification criterion (500) and the plurality of inference patterns (550).
[0286] In the present invention, pair data may refer to structured data formed by connecting a classification criterion (500) and an inference pattern distinguished according to the classification criterion (500) in order to analyze the inference structure included in the long text inference information. More specifically, the pair data may include at least one classification criterion (500) and a plurality of inference patterns (550) distinguished from each other according to the classification criterion (500).
[0287] Here, the classification criterion (500) may refer to reference information for distinguishing differences in inference structures included in long text inference information, and the inference pattern may refer to a type of specific inference method distinguished according to the classification criterion (500). Additionally, multiple inference patterns (550) may be configured to have different inference structures while corresponding to the same classification criterion (500).
[0288] That is, the present invention can generate pair data by connecting different inference patterns corresponding to a single classification criterion (500), rather than simply listing multiple inference patterns.
[0289] For example, multiple inference patterns (550) corresponding to the analysis perspective criteria may be detailed fact-based analysis-centered inference and overall structure-based analysis-centered inference. Additionally, multiple inference patterns (550) corresponding to the access scope criteria may be restricted search-centered inference and extended search-centered inference. Furthermore, multiple inference patterns (550) corresponding to the inference type criteria may be inductive inference and deductive inference.
[0290] Additionally, multiple inference patterns (550) corresponding to the idea development criteria may be sequential development-centered inference and parallel review-centered inference. Additionally, multiple inference patterns (550) corresponding to the verification focus criteria may be data-based verification-centered inference and hypothesis-based verification-centered inference. Additionally, multiple inference patterns (550) corresponding to the clarification approach criteria may be iterative clarification-centered inference and direct conclusion-drawing-centered inference.
[0291] Additionally, a plurality of inference patterns (550) generated according to classification criteria (500) corresponding to safety-related long-form inference information may be risk prevention-centered inference and risk direct processing-centered inference. In another embodiment, the plurality of inference patterns (550) may be technical description-centered inference and ethical judgment-centered inference.
[0292] For example, as illustrated in FIG. 8, a plurality of inference patterns (550) corresponding to a first classification criterion (510) may include a first inference pattern (551) and a second inference pattern. In one embodiment, the first classification criterion (510) may be a criterion for distinguishing differences in analysis perspectives, the first inference pattern (551) may be an inference centered on detailed fact-based analysis, and the second inference pattern may be an inference centered on overall structure-based analysis.
[0293] Additionally, the second classification criterion (520) may be a criterion for distinguishing differences in the focus of verification, and the plurality of inference patterns (550) corresponding to the second classification criterion (520) may be data-based verification-centered inference and hypothesis-based verification-centered inference. Additionally, the third classification criterion (530) may be a criterion for distinguishing differences in the method of developing ideas, and the plurality of inference patterns (550) corresponding to the third classification criterion (530) may be sequential development-centered inference and parallel review-centered inference.
[0294] Additionally, the control unit (195) can configure a prompt using at least one of the query and long text inference information, and input the prompt into an artificial intelligence model to generate pair data.
[0295] For example, the control unit (195) can configure a prompt to generate multiple inference patterns (550) that are distinct from each other based on an inference structure included in the text inference information itself. In this case, the control unit (195) can configure a prompt such that multiple inference patterns (550) corresponding to a single classification criterion (500) have different inference structures.
[0296] In one embodiment, the control unit (195) may configure a prompt to generate a plurality of inference patterns (550) based on an inference structure included in the long text inference information. In another embodiment, the control unit (195) may configure a prompt so that the names of the inference patterns reflect the inference structure included in the long text inference information. In yet another embodiment, the control unit (195) may configure a prompt to limit the generation of inference patterns that are semantically overlapping with each other and to generate inference patterns that are distinct from each other.
[0297] Additionally, the control unit (195) can generate multiple inference patterns (550) based on inference structures that are repeatedly included in multiple long text inference information. For example, the control unit (195) can generate inference patterns based on inference structures that are commonly included in multiple long text inference information, rather than being limited to specific long text inference information.
[0298] In one embodiment, the control unit (195) may generate a stepwise calculation-centered inference pattern when the calculation execution order is repeatedly included in the long text inference information. In another embodiment, the control unit (195) may generate a verification-centered inference pattern when the intermediate verification structure is repeatedly included in the long text inference information. In yet another embodiment, the control unit (195) may generate a conclusion-first centered inference pattern when the structure that presents a conclusion first and then explains details is repeatedly included in the long text inference information.
[0299] Additionally, the control unit (195) can check whether the generated pair data satisfies a preset verification condition. For example, the control unit (195) can check whether a plurality of inference patterns (550) corresponding to a single classification criterion (500) include different inference structures. In another embodiment, the control unit (195) can check whether the correspondence relationship between the classification criterion (500) and the plurality of inference patterns (550) satisfies a preset verification condition. In yet another embodiment, the control unit (195) can check whether the plurality of inference patterns (550) can distinguish differences in inference structures included in the long text inference information.
[0300] In addition, generating pair data in the present invention can be understood as structuring the inference structure contained in the long text inference information itself into a form distinguishable from one another, and connecting a plurality of inference patterns (550) corresponding to a single classification criterion (500).
[0301] In particular, the present invention can generate pair data for analyzing differences in reasoning behavior and reasoning structure included in the long text reasoning information itself, rather than simply classifying response types. Additionally, by generating a plurality of different reasoning patterns (550), the present invention can analyze differences in reasoning structures included in the long text reasoning information in an explainable form.
[0302] Additionally, the control unit (195) may store the generated pair data in the storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of a classification criterion (500), a plurality of inference patterns (550), a first inference pattern (551), inference pattern description information, inference structure information, and inference pattern distinction information.
[0303] Afterward, the control unit (195) can perform embedding on the pair data to generate embedding vectors (600) corresponding to each of the classification criteria (500) (see S640 in FIG. 6).
[0304] For example, as illustrated in FIG. 8, the control unit (195) can perform embedding on pair data consisting of a classification criterion (500) and a plurality of inference patterns (550) and generate a plurality of embedding vectors (610, 620, 630).
[0305] In the present invention, embedding may refer to a process of converting semantic information included in a classification criterion (500) and a plurality of inference patterns (550) corresponding to the classification criterion (500) into information in the form of a vector. More specifically, embedding may refer to a process of expressing at least one of an inference structure feature, an explanation structure feature, a verification structure feature, and a judgment structure feature included in the classification criterion (500) and the plurality of inference patterns (550) in a vector space.
[0306] Here, the embedding vector (600) may refer to information in which semantic information included in a classification criterion (500) and a plurality of inference patterns (550) corresponding to the classification criterion (500) is expressed in a vector form. Additionally, the embedding vector (600) may refer to vector information for analyzing semantic similarity or inference structure similarity between different classification criteria (500).
[0307] For example, embedding vectors corresponding to the analysis perspective criterion can reflect semantic information included in detail-fact-based analysis-centric inference and overall structure-based analysis-centric inference. Additionally, embedding vectors corresponding to the verification focus criterion can reflect semantic information included in data-based verification-centric inference and hypothesis-based verification-centric inference.
[0308] In addition, embedding vectors corresponding to idea development criteria may also reflect semantic information included in sequential development-centered reasoning and parallel review-centered reasoning.
[0309] For example, as illustrated in FIG. 8, the control unit (195) may generate a first embedding vector (610) by performing embedding on a first classification criterion (510) and a plurality of inference patterns (550) corresponding to the first classification criterion (510). Additionally, the control unit (195) may generate a second embedding vector (620) by performing embedding on a second classification criterion (520) and a plurality of inference patterns (550) corresponding to the second classification criterion (520). Additionally, the control unit (195) may generate a third embedding vector (630) by performing embedding on a third classification criterion (530) and a plurality of inference patterns (550) corresponding to the third classification criterion (530).
[0310] Here, the first embedding vector (610), the second embedding vector (620), and the third embedding vector (630) may include semantic information corresponding to different classification criteria (500). That is, the present invention can analyze semantic relationships between multiple classification criteria (500) by expressing multiple inference structures included in long text inference information in vector form.
[0311] Additionally, the control unit (195) can perform embedding by using the name information of the classification criterion (500) and a plurality of inference patterns (550) corresponding to the classification criterion (500) together. For example, the control unit (195) can generate an embedding vector by using the name information of "verification focus criterion" together with data-based verification-centered inference and hypothesis-based verification-centered inference.
[0312] In one embodiment, the control unit (195) may perform embedding using only the name information of the classification criteria (500). In another embodiment, the control unit (195) may perform embedding using only the descriptive information included in the plurality of inference patterns (550). In yet another embodiment, the control unit (195) may perform embedding using both the name information of the classification criteria (500) and the descriptive information included in the plurality of inference patterns (550).
[0313] Additionally, the control unit (195) can generate an embedding vector (600) using the embedding unit (150). Here, the embedding unit (150) may refer to a component for converting semantic information included in the classification criteria (500) and the plurality of inference patterns (550) into a vector form.
[0314] In one embodiment, the embedding unit (150) may generate an embedding vector (600) using a text embedding model. In another embodiment, the embedding unit (150) may generate an embedding vector (600) using a transformer-based language model. In yet another embodiment, the embedding unit (150) may generate an embedding vector (600) using a sentence embedding model or a semantic embedding model.
[0315] Additionally, the control unit (195) can perform embedding such that the semantic information included in the embedding vector (600) reflects the inference structure included in the long text inference information. For example, the control unit (195) can generate the embedding vector (600) based on at least one of the explanation structure, verification structure, judgment structure, and conclusion derivation structure included in the long text inference information.
[0316] In one embodiment, the control unit (195) may generate an embedding vector (600) based on a computation execution structure included in the long text inference information. In another embodiment, the control unit (195) may generate an embedding vector (600) based on a verification execution structure included in the long text inference information. In yet another embodiment, the control unit (195) may generate an embedding vector (600) based on a conclusion derivation structure included in the long text inference information.
[0317] Additionally, the control unit (195) can check whether the generated embedding vector (600) satisfies a preset verification condition. For example, the control unit (195) can check whether classification criteria (500) containing identical or similar inference structures are represented by embedding vectors (600) that are similar to each other. In another embodiment, the control unit (195) can check whether classification criteria (500) containing different inference structures are represented by embedding vectors (600) that are distinct from each other. In yet another embodiment, the control unit (195) can check the embedding results based on distance information or similarity information between embedding vectors (600).
[0318] Additionally, generating an embedding vector (600) in the present invention can be understood as expressing the inference structure contained in the long text inference information itself as semantic information in the form of a vector. In particular, the present invention can analyze semantic similarity and inference structure similarity between classification criteria (500) by expressing the inference structure contained in the long text inference information in the form of an embedding vector (600).
[0319] In addition, the present invention can express the inference structure included in the long text inference information in a form that can be analyzed in a vector space by converting the semantic information included in the classification criteria (500) and the plurality of inference patterns (550) into an embedding vector (600).
[0320] Additionally, the control unit (195) may store the generated embedding vector (600) in the storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of the embedding vector (600), the first embedding vector (610), the second embedding vector (620), the third embedding vector (630), the classification criteria (500), the plurality of inference patterns (550), and embedding-related meta-information.
[0321] Afterwards, the control unit (195) can perform clustering on classification criteria (500) based on embedding vectors (600) (see S650 in FIG. 6).
[0322] For example, as illustrated in FIGS. 9 and 10, the control unit (195) may perform clustering on a plurality of embedding vectors (600) including a first embedding vector (610), a second embedding vector (620), and a third embedding vector (630), and generate a first cluster (710), a second cluster (720), and a third cluster (730).
[0323] In the present invention, clustering may refer to a process of configuring classification criteria (500) that include similar inference structures to be included in the same group based on semantic similarity between a plurality of embedding vectors (600).
[0324] Here, semantic similarity may mean at least one similarity among semantic information, inference structure information, explanation structure information, verification structure information and judgment structure information included in the classification criteria (500) or multiple inference patterns (550).
[0325] For example, embedding vectors containing semantic information related to the verification execution structure may be included in the same cluster. Additionally, embedding vectors containing semantic information related to the sequential expansion structure may be included in the same cluster.
[0326] In addition, embedding vectors containing semantic information related to parallel review structures may be included in the same cluster.
[0327] For example, as illustrated in FIGS. 9 and 10, the control unit (195) may include embedding vectors related to verification focus criteria in the first cluster (710). Additionally, the control unit (195) may include embedding vectors related to explanation structures in the second cluster (720). Additionally, the control unit (195) may include embedding vectors related to idea development structures in the third cluster (730).
[0328] Here, the first cluster (710), the second cluster (720), and the third cluster (730) may contain semantic information related to different inference structures. That is, the present invention can analyze classification criteria (500) having similar inference structures in groups by using a plurality of embedding vectors (600).
[0329] Additionally, the control unit (195) can perform clustering using the clustering unit (160). Here, the clustering unit (160) may refer to a component for performing similarity analysis and cluster generation for a plurality of embedding vectors (600).
[0330] In one embodiment, the clustering unit (160) may perform hierarchical clustering. In another embodiment, the clustering unit (160) may perform density-based clustering. In yet another embodiment, the clustering unit (160) may perform center-based clustering.
[0331] Additionally, the control unit (195) may use a preset evaluation metric to determine the number of clusters generated when performing clustering. Here, the evaluation metric may refer to information for quantitatively expressing at least one of the degree to which embedding vectors (600) are aggregated within a cluster and the degree to which they are separated between clusters.
[0332] In one embodiment, the control unit (195) may use a silhouette score as an evaluation indicator. Here, the silhouette score may represent a value that quantifies the degree to which an arbitrary embedding vector (600) is close to other embedding vectors in the cluster to which it belongs and far from embedding vectors in other clusters. More specifically, the control unit (195) may perform clustering by changing the number of clusters according to candidate values and determine the number of clusters based on the silhouette score calculated for each candidate value.
[0333] In another embodiment, the control unit (195) may determine the number of clusters by using at least one of cluster internal dispersion information, inter-cluster distance information, and distribution density information of embedding vectors (600) as an evaluation indicator. In yet another embodiment, the control unit (195) may determine the number of clusters by repeating agglomerative clustering until a predetermined cluster distance threshold is reached.
[0334] In addition, in the present invention, the number of clusters is not limited to a fixed value specified in advance by the user, but can be dynamically determined based on the semantic distribution of candidate classification criteria included in the set of candidate classification criteria. Accordingly, even when performing analysis on a new data set, a new artificial intelligence model, or a new domain, the present invention can derive a number of representative classification criteria (800) suitable for the data.
[0335] Additionally, the control unit (195) can perform clustering based on distance information or similarity information between embedding vectors (600). For example, the control unit (195) can include embedding vectors (600) in the same cluster if the distance between them is less than or equal to a preset reference distance. In another embodiment, the control unit (195) can include embedding vectors (600) in the same cluster if the cosine similarity between them is greater than or equal to a preset reference similarity.
[0336] Additionally, the control unit (195) can perform clustering so that classification criteria (500) containing similar inference structures are included in the same cluster. For example, the control unit (195) can include embedding vectors related to iterative verification structures, intermediate judgment verification structures, and error review structures in the same cluster.
[0337] In another embodiment, the control unit (195) may include embedding vectors related to a sequential expansion structure, a stepwise calculation structure, and a sequential explanation structure in the same cluster. In yet another embodiment, the control unit (195) may include embedding vectors related to a structure that compares multiple perspectives and a structure that examines multiple judgments together in the same cluster.
[0338] Additionally, the control unit (195) can check whether the generated cluster satisfies preset verification conditions. For example, the control unit (195) can check whether the embedding vectors (600) included in the same cluster contain similar inference structures. In another embodiment, the control unit (195) can check whether the embedding vectors (600) included in different clusters contain different inference structures. In yet another embodiment, the control unit (195) can check the clustering results based on distance information or similarity information between clusters.
[0339] Additionally, the control unit (195) can generate multiple clusters based on multiple inference structures included in the text inference information. For example, the control unit (195) can generate clusters related to analysis perspectives, clusters related to verification focus, clusters related to idea development structures, and clusters related to clarification approaches.
[0340] In one embodiment, the control unit (195) may perform clustering on long text inference information corresponding to a specific domain. In another embodiment, the control unit (195) may perform integrated clustering on long text inference information corresponding to a plurality of domains. In yet another embodiment, the control unit (195) may perform individual clustering on long text inference information for each artificial intelligence model.
[0341] Additionally, performing clustering in the present invention can be understood as grouping inference structures included in long text inference information based on semantic similarity. In particular, the present invention can analyze classification criteria (500) containing similar inference structures together by grouping inference structures included in long text inference information based on embedding vectors (600).
[0342] In addition, the present invention can configure the inference structure included in the long text inference information into a form that can be analyzed in group units by generating clusters based on semantic information included in a plurality of classification criteria (500).
[0343] Additionally, the control unit (195) may store the generated clustering result in a storage unit (120) or a previously specified storage (300). At this time, the information stored may include at least one of the clustering result, a first cluster (710), a second cluster (720), a third cluster (730), an embedding vector (600), a classification criterion (500), and cluster-related meta-information.
[0344] Subsequently, the control unit (195) can identify representative classification criteria (800) representing each of the clusters among the classification criteria (500) included in the clusters according to the clustering result (700) based on a preset standard (see S660 in FIG. 6).
[0345] For example, as illustrated in FIG. 9, the control unit (195) can analyze the classification criteria (500) included in the first cluster (710), the second cluster (720), and the third cluster (730), and identify the first representative classification criteria (810), the second representative classification criteria (820), and the third representative classification criteria (830) representing each of the clusters.
[0346] In the present invention, representative classification criteria (800) may refer to a classification criterion that represents a cluster among a plurality of classification criteria (500) included in a single cluster. More specifically, representative classification criteria (800) may refer to a classification criterion that represents semantic information included in a cluster among a plurality of classification criteria (500) included in the same cluster.
[0347] Here, the classification criteria (500) may refer to reference information for distinguishing differences in inference structures included in long text inference information, and the representative classification criteria (800) may refer to reference information representing the classification criteria (500) included in the cluster.
[0348] Additionally, the pre-established criteria in the present invention may refer to criteria information used to specify representative classification criteria (800). More specifically, the pre-established criteria may include at least one of distance information between embedding vectors, similarity information, cluster center information, frequency information, and density information.
[0349] For example, the control unit (195) may specify a classification criterion (500) corresponding to the embedding vector that is closest in distance to other embedding vectors within a cluster as a representative classification criterion. In another embodiment, the control unit (195) may specify a classification criterion (500) having semantic information that is included with the highest frequency within a cluster as a representative classification criterion. In yet another embodiment, the control unit (195) may specify a classification criterion (500) corresponding to the embedding vector closest to the cluster center as a representative classification criterion.
[0350] For example, as illustrated in FIG. 9, the control unit (195) may specify a classification criterion representing semantic information related to a verification structure among a plurality of classification criteria (500) included in a first cluster (710) as the first representative classification criterion (810). Additionally, the control unit (195) may specify a classification criterion representing semantic information related to an explanation structure among a plurality of classification criteria (500) included in a second cluster (720) as the second representative classification criterion (820). Additionally, the control unit (195) may specify a classification criterion representing semantic information related to an idea development structure among a plurality of classification criteria (500) included in a third cluster (730) as the third representative classification criterion (830).
[0351] Additionally, the control unit (195) can select a representative embedding vector representing a cluster among the embedding vectors (600) and specify a classification criterion (500) corresponding to the representative embedding vector as representative classification criteria (800).
[0352] Here, the representative embedding vector may refer to vector information representing semantic information of a cluster among a plurality of embedding vectors (600) included in a cluster.
[0353] In one embodiment, the control unit (195) may select a representative embedding vector based on average distance information with other embedding vectors within the cluster. In another embodiment, the control unit (195) may select a representative embedding vector based on distance information with respect to the center of the cluster. In yet another embodiment, the control unit (195) may select a representative embedding vector based on similarity information between embedding vectors.
[0354] Additionally, the control unit (195) may use at least one of a plurality of selection criteria for selecting a representative embedding vector. In one embodiment, the control unit (195) may use a medoid-based selection criterion. Here, the medoid-based selection criterion may mean a criterion for selecting the embedding vector with the smallest sum of distances from other embedding vectors among a plurality of embedding vectors (600) included in the cluster as the representative embedding vector.
[0355] In another embodiment, the control unit (195) may use frequency-based selection criteria. Here, the frequency-based selection criteria may refer to criteria for selecting an embedding vector as a representative embedding vector that corresponds to the region containing the most candidate classification criteria having the same or similar semantic information among a plurality of classification criteria (500) included in the cluster.
[0356] In another embodiment, the control unit (195) may use a density-based selection criterion. Here, the density-based selection criterion may mean a criterion for selecting an embedding vector corresponding to the region with the highest distribution density of the embedding vectors (600) included in the cluster as the representative embedding vector.
[0357] In another embodiment, the control unit (195) may use a silhouette-based selection criterion. Here, the silhouette-based selection criterion may mean a criterion for selecting the embedding vector with the highest silhouette score among the embedding vectors (600) included in the cluster as the representative embedding vector.
[0358] Additionally, the control unit (195) may specify representative classification criteria (800) using at least one of the above-mentioned medoid-based selection criteria, frequency-based selection criteria, density-based selection criteria, and silhouette-based selection criteria, and when two or more selection criteria are used together, the classification criterion (500) corresponding to the embedding vector corresponding to the same or closest position among the representative embedding vectors derived for each selection criterion may be specified as the representative classification criterion.
[0359] Additionally, the control unit (195) can specify representative classification criteria (800) using the clustering unit (160) and the criteria processing unit (140). Here, the criteria processing unit (140) may refer to a component for analyzing semantic information included in the cluster and specifying a classification criterion (500) representing the cluster.
[0360] Additionally, the control unit (195) can check whether the representative classification criteria (800) satisfy a preset verification condition. For example, the control unit (195) can check whether the representative classification criteria (800) can represent a plurality of classification criteria (500) included in the cluster. In another embodiment, the control unit (195) can check whether the embedding vector corresponding to the representative classification criteria (800) is located at or below a preset reference distance from the cluster center. In yet another embodiment, the control unit (195) can check whether the representative classification criteria (800) distinguish different inference structures.
[0361] Additionally, the control unit (195) can specify a plurality of representative classification criteria (800) based on a plurality of inference structures included in the text inference information. For example, the control unit (195) can specify a representative classification criterion related to the analysis perspective, a representative classification criterion related to the verification focus, a representative classification criterion related to the idea development structure, and a representative classification criterion related to the clarification approach.
[0362] In one embodiment, the control unit (195) may specify representative classification criteria (800) for long text inference information corresponding to a specific domain. In another embodiment, the control unit (195) may specify common representative classification criteria (800) for long text inference information corresponding to a plurality of domains. In yet another embodiment, the control unit (195) may specify different representative classification criteria (800) for long text inference information for each artificial intelligence model.
[0363] Additionally, specifying representative classification criteria (800) in the present invention can be understood as selecting criterion information that represents the cluster among a plurality of classification criteria (500) included in the cluster. In particular, by specifying representative classification criteria (800) based on semantic information included in the embedding vector (600), the present invention can analyze the inference structure included in the long text inference information based on the representative criteria.
[0364] Meanwhile, according to one embodiment of the present specification, a training data generation process for learning a classification model according to the present invention may include: (a) a candidate extraction step of extracting a number of thought criterion candidates in an automated manner from the output responses of various artificial intelligence models (according to one embodiment, thousands of thought criterion candidates (e.g., about 4,057) may be extracted); (b) a semantic space embedding step of converting the extracted thought criterion candidates into semantic vectors through a pre-trained embedding model (e.g., text-embedding-3-large); (c) a hierarchical clustering step of grouping the embedded thought criteria based on semantic similarity to remove duplicates and selecting representative criteria (medoids) within the cluster; (d) an evaluation rubric generation step of defining two contrasting thought patterns for each classification dimension and generating an evaluation rubric capable of distinguishing the two thought patterns through a pre-trained large-scale language model (e.g., an inference model such as GPT-4o); and (e) an automatic analysis report generation step of automatically classifying the responses of the artificial intelligence models based on the evaluation rubric and generating an inference pattern analysis report indicating which inference pattern the responses follow. There is. The terms "analysis report" or "classification report" as used in this paragraph may be understood to refer to the same concept as the "inference pattern analysis report" as used in other parts of this specification.
[0365] In addition, by specifying representative classification criteria (800) for each cluster, the present invention can configure the inference structure included in the long text inference information into a form that can be distinguished by representative criteria units.
[0366] Additionally, the control unit (195) may store representative classification criteria (800) in a storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of the representative classification criteria (800), a first representative classification criterion (810), a second representative classification criterion (820), a third representative classification criterion (830), representative embedding vector information, and meta information related to the representative criteria.
[0367] Afterward, the control unit (195) can generate rubric data (900) for classifying inference patterns corresponding to representative classification criteria (800) (see S670 in FIG. 6).
[0368] For example, as illustrated in FIG. 11, the control unit (195) can generate first rubric data (910), second rubric data (920), and third rubric data (930) corresponding to the first representative classification criterion (810), the second representative classification criterion (820), and the third representative classification criterion (830), respectively.
[0369] In the present invention, rubric data (900) may refer to reference data for classifying inference patterns included in long text inference information to correspond to representative classification criteria (800). More specifically, rubric data (900) may include at least one of definition information, feature information, and example information of an inference pattern corresponding to the representative classification criteria (800).
[0370] Here, definition information may refer to information describing what inference structure a specific inference pattern possesses. Additionally, feature information may refer to information describing at least one feature among the explanatory structure, verification structure, judgment structure, and inference development structure identified in the specific inference pattern. Furthermore, example information may refer to example sentences, example inference content, or example inference structures of long-text inference information corresponding to the specific inference pattern.
[0371] Additionally, in the present invention, rubric data (900) may be a concept distinct from a simple keyword list or a simple classification label. Rubric data (900) may refer to structured reference data for distinguishing multiple inference patterns (550) based on an inference structure included in the long text inference information itself.
[0372] For example, if the first representative classification criterion (810) is a criterion for distinguishing differences in verification structures, the first rubric data (910) may include at least one of whether an intermediate judgment review is performed, whether iterative verification is performed, whether the possibility of error is confirmed, and whether an alternative review is performed.
[0373] Additionally, if the second representative classification criterion (820) is a criterion for distinguishing differences in explanation structures, the second rubric data (920) may include at least one of sequential explanation, example-based explanation, comparative explanation, and a method for deriving conclusions.
[0374] Additionally, if the third representative classification criterion (830) is a criterion for distinguishing differences in the inference development structure, the third rubric data (930) may include at least one of whether sequential development is performed, whether parallel review is performed, whether iterative clarification is performed, and whether conclusion-first development is performed.
[0375] Additionally, the rubric data (900) corresponding to the safety-related long-form inference information may include at least one of whether the inference is centered on risk prevention, whether ethical considerations are performed, whether the possibility of user harm is reviewed, and whether the safety-first explanation structure is structured.
[0376] Additionally, the rubric data (900) corresponding to the long text inference information related to instruction execution may include at least one of whether the user's intention is reflected, whether constraints are maintained, whether the request format is maintained, and whether the response purpose matches.
[0377] Additionally, the control unit (195) can generate rubric data (900) using the rubric generation unit (170). Here, the rubric generation unit (170) may represent a component for generating at least one of definition information, feature information, and example information of an inference pattern corresponding to representative classification criteria (800).
[0378] In one embodiment, the control unit (195) may generate rubric data (900) using representative classification criteria (800) and a plurality of inference patterns (550). In another embodiment, the control unit (195) may generate rubric data (900) based on an inference structure that is repeatedly verified in the long text inference information itself. In yet another embodiment, the control unit (195) may generate rubric data (900) based on an explanation structure or a verification structure that is commonly included in a plurality of long text inference information.
[0379] Additionally, the control unit (195) can configure a prompt including at least one of query and long text inference information, and input the prompt into an artificial intelligence model to generate rubric data (900).
[0380] For example, the control unit (195) can configure a prompt to generate definition information, feature information, and example information of an inference pattern corresponding to representative classification criteria (800). Additionally, the control unit (195) can configure a prompt so that the rubric data (900) reflects the inference structure contained within the long text inference information itself.
[0381] In one embodiment, the control unit (195) may configure a prompt so that definition information included in the rubric data (900) explains the characteristics of the inference structure. In another embodiment, the control unit (195) may configure a prompt so that example information included in the rubric data (900) corresponds to actual long text inference information. In yet another embodiment, the control unit (195) may configure a prompt so that the rubric data (900) can distinguish between a plurality of different inference patterns (550). Such a prompt may be configured as in the following embodiment (see Table 2 below).
[0382] Create a concise rubric for the following reasoning pattern criterion:[CRITERION NAME]: [PATTERN A] vs. [PATTERN B]For each pattern, provide:1. A clear, concise definition (2-3 sentences) that captures the essence of this reasoning pattern2. 3-4 key characteristics that distinguish this pattern3. 2 concrete examples of responses that demonstrate this pattern (keep examples brief, about 2-3 sentences each)Focus on making the distinctions between patterns clear and easily identifiable. The definitions and examples should help evaluators quickly categorize model responses without ambiguity.
[0383] Additionally, the control unit (195) can check whether the rubric data (900) satisfies a preset verification condition. For example, the control unit (195) can check whether the rubric data (900) includes at least one of definition information, feature information, and example information. In another embodiment, the control unit (195) can check whether the rubric data (900) can distinguish inference patterns corresponding to representative classification criteria (800). In yet another embodiment, the control unit (195) can check whether the example information included in the rubric data (900) corresponds to an inference structure included in the long text inference information.
[0384] Additionally, the control unit (195) can generate rubric data (900) based on the inference structure included in the long text inference information. For example, the control unit (195) can generate rubric data (900) based on at least one of the verification execution expression, comparison execution expression, calculation execution expression, and explanation expansion expression that are repeatedly included in the long text inference information.
[0385] In one embodiment, the control unit (195) may generate rubric data (900) for verification-centered inference based on an intermediate judgment review structure included in the long-text inference information. In another embodiment, the control unit (195) may generate rubric data (900) for sequential expansion-centered inference based on a step-by-step explanation structure included in the long-text inference information. In yet another embodiment, the control unit (195) may generate rubric data (900) for parallel review-centered inference based on a multiple judgment comparison structure included in the long-text inference information.
[0386] Additionally, generating rubric data (900) in the present invention can be understood as generating reference data including at least one of definition information, feature information, and example information of an inference pattern in order to classify an inference pattern corresponding to representative classification criteria (800).
[0387] In particular, the present invention can classify the inference patterns included in the long-text inference information into an explainable form by generating rubric data (900) based on the inference structure included in the long-text inference information itself.
[0388] In addition, the present invention can analyze the difference in inference structures included in long text inference information based on reference data by generating rubric data (900) for distinguishing a plurality of different inference patterns (550).
[0389] Additionally, the control unit (195) may store the generated rubric data (900) in the storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of the rubric data (900), the first rubric data (910), the second rubric data (920), the third rubric data (930), definition information, feature information, and example information.
[0390] Subsequently, the control unit (195) can use rubric data (900) to classify a plurality of inference patterns (550) related to long text inference information so as to correspond to at least one of representative classification criteria (800) (see S680 in FIG. 6).
[0391] For example, as illustrated in FIG. 12, the control unit (195) can analyze a plurality of inference patterns (550) included in the long text inference information using the first rubric data (910), the second rubric data (920), and the third rubric data (930), and classify the plurality of inference patterns (550) to correspond to representative classification criteria (800).
[0392] In the present invention, classifying a plurality of inference patterns (550) may mean analyzing at least one of the inference structure, explanation structure, verification structure, judgment structure, and conclusion derivation structure included in the long text inference information itself based on rubric data (900), and determining whether the long text inference information corresponds to at least one of the representative classification criteria (800).
[0393] Here, representative classification criteria (800) may mean representative criterion information for distinguishing differences in inference structures included in long text inference information, and rubric data (900) may include at least one of definition information, feature information and example information for distinguishing inference patterns corresponding to the representative classification criteria (800).
[0394] Additionally, in the present invention, a plurality of inference patterns (550) may refer to a plurality of types of inference methods having different inference structures according to a single representative classification criterion. For example, if a single representative classification criterion is a criterion for distinguishing differences in analysis perspectives, the plurality of inference patterns (550) may include detailed fact-based analysis-centered inference and overall structure-based analysis-centered inference. As another example, if a single representative classification criterion is a criterion for distinguishing differences in idea development methods, the plurality of inference patterns (550) may include sequential development-centered inference and parallel review-centered inference.
[0395] In one embodiment, the control unit (195) may determine the inference structure of the long-word inference information using definition information included in the rubric data (900). In another embodiment, the control unit (195) may determine the explanation structure or verification structure of the long-word inference information using feature information included in the rubric data (900). In yet another embodiment, the control unit (195) may determine an inference pattern having an inference structure corresponding to the long-word inference information using example information included in the rubric data (900).
[0396] For example, as illustrated in FIG. 12, the control unit (195) can analyze a first inference pattern (551) among a plurality of inference patterns (550) related to long text inference information based on the first rubric data (910) and classify the first inference pattern (551) into an inference pattern corresponding to a first representative classification criterion (810).
[0397] In one embodiment, the first rubric data (910) may include definition information for distinguishing differences in verification structures, iterative verification feature information, and intermediate judgment review feature information. In this case, if the long text inference information includes an expression for performing iterative verification and an intermediate judgment review structure, the control unit (195) may classify the first inference pattern (551) as an inference pattern corresponding to verification-centered inference.
[0398] In another embodiment, the second rubric data (920) may include definition information for distinguishing differences in explanation structures, sequential explanation feature information, and stepwise calculation feature information. In this case, if the text inference information includes sequential explanation structures and stepwise calculation structures, the control unit (195) may classify the inference pattern corresponding to the text inference information as sequential expansion-centered inference.
[0399] In another embodiment, the third rubric data (930) may include definition information for distinguishing differences in idea development methods, multiple judgment comparison feature information, and multiple perspective analysis feature information. In this case, if the long text inference information includes an explanation structure that compares multiple judgment structures, the control unit (195) may classify the inference pattern corresponding to the long text inference information as a parallel review-centered inference.
[0400] Additionally, regarding safety-related text inference information, the control unit (195) can classify whether the text inference information corresponds to risk prevention-centered inference or risk direct processing-centered inference by using rubric data (900) corresponding to safety priority criteria. In another embodiment, the control unit (195) can classify whether the text inference information corresponds to technical explanation-centered inference or ethical judgment-centered inference by using rubric data (900) corresponding to ethical consideration criteria.
[0401] Additionally, regarding the long text inference information related to instruction execution, the control unit (195) can determine whether the long text inference information reflects at least one of the user intent, constraints, request format, and response purpose by using rubric data (900) corresponding to the user understanding criteria.
[0402] Additionally, the control unit (195) can perform inference pattern classification for long text inference information using the inference pattern classification unit (180). Here, the inference pattern classification unit (180) may refer to a component for analyzing the inference structure of long text inference information using rubric data (900) and determining which representative classification criteria (800) the long text inference information corresponds to.
[0403] Additionally, the control unit (195) may classify inference patterns by individually analyzing multiple inference sections included in the long text inference information. Here, an inference section may refer to a part of a sentence unit, paragraph unit, or semantic unit containing a specific inference structure among the long text inference information.
[0404] For example, the control unit (195) can classify the inference patterns of each section of the long text inference information when verification-centered inference is confirmed in a specific section of the long text inference information and sequential expansion-centered inference is confirmed in another section.
[0405] In one embodiment, the control unit (195) may analyze the inference structure by sentence unit, paragraph unit, or semantic unit included in the long text inference information. In another embodiment, the control unit (195) may analyze different inference structures for the beginning, middle, and conclusion parts of the long text inference information. In yet another embodiment, the control unit (195) may classify the inference pattern by section based on the inference structure that is repeatedly included in a specific section of the long text inference information.
[0406] Additionally, the control unit (195) can analyze the inference structure of the long text inference information based on the degree of correspondence with a plurality of representative classification criteria (800). Here, the degree of correspondence may refer to how much the inference structure of the long text inference information corresponds to the definition information, feature information, and example information included in the rubric data (900).
[0407] In one embodiment, the control unit (195) may analyze the semantic similarity between the long text inference information and the definition information included in the rubric data (900). In another embodiment, the control unit (195) may analyze the structural similarity between the long text inference information and the example information included in the rubric data (900). In yet another embodiment, the control unit (195) may analyze the correspondence between the explanatory structure included in the long text inference information and the feature information included in the rubric data (900).
[0408] Additionally, the control unit (195) can check whether the inference pattern classification result satisfies a preset verification condition. For example, the control unit (195) can check whether the inference pattern classification result was generated based on at least one of definition information, feature information, and example information included in the rubric data (900). In another embodiment, the control unit (195) can check whether the inference pattern classification result corresponds to the inference structure of the long text inference information. In yet another embodiment, the control unit (195) can check whether the inference pattern classification result corresponds to at least one of the representative classification criteria (800).
[0409] Additionally, the control unit (195) may classify a single long text inference information as corresponding to a plurality of representative classification criteria (800) when the single long text inference information includes a plurality of inference structures together. In this case, the control unit (195) may generate the degree of correspondence or the correspondence relationship by segment of the long text inference information for each representative classification criterion.
[0410] Additionally, classifying a plurality of inference patterns (550) in the present invention can be understood as matching an inference structure included in the long text inference information itself to at least one of representative classification criteria (800) based on rubric data (900).
[0411] In particular, the present invention can classify the inference patterns of long-text inference information into an explainable form by comparing and analyzing the actual inference structure and explanation structure included in the long-text inference information itself with rubric data (900).
[0412] In addition, the present invention is not limited to classifying the entire text inference information into only one single type, but can classify multiple inference patterns (550) by analyzing multiple inference structures included in the text inference information by section.
[0413] Additionally, the control unit (195) may store the inference pattern classification result in the storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of a plurality of inference patterns (550), a first inference pattern (551), representative classification criteria (800), rubric data (900), inference pattern classification result by long text inference information, and inference pattern information by section.
[0414] Afterwards, the control unit (195) can generate data related to the classification results of a plurality of inference patterns (550) (see S690 in FIG. 6).
[0415] For example, as illustrated in FIG. 13, the control unit (195) can generate an inference pattern analysis report based on the inference pattern classification results for long text inference information. Here, the inference pattern analysis report may refer to analysis information that organizes at least one of the inference structure, explanation structure, verification structure, judgment structure, and conclusion derivation structure included in the long text inference information itself into an explainable form.
[0416] In the present invention, data related to the classification result may include at least one of a classification result of an inference pattern corresponding to long text inference information and an inference pattern analysis report explaining the classification result.
[0417] Here, the classification result of the inference pattern may mean information including which of the representative classification criteria (800) the long text inference information corresponds to, and which of the plurality of inference patterns (550) distinguished according to the said criteria it corresponds to.
[0418] Additionally, the inference pattern analysis report may include at least one of name information of the inference pattern corresponding to the long text inference information, basis information classified as an inference pattern, interval information corresponding to the inference pattern in the long text inference information, and feature description information of the inference pattern.
[0419] In the present invention, nominal information may refer to information for identifying inference patterns corresponding to textual inference information. For example, nominal information may include at least one of verification-centered inference, sequential expansion-centered inference, parallel review-centered inference, inductive inference, deductive inference, data-based verification-centered inference, hypothesis-based verification-centered inference, and conclusion-first-centered inference.
[0420] In the present invention, the basis information may refer to information explaining the reason why the long text inference information is classified into a specific inference pattern. More specifically, the basis information may refer to information explaining the reason why at least one of the expression, sentence structure, explanation structure, verification structure, judgment structure, and conclusion derivation structure included in the long text inference information corresponds to at least one of the definition information, feature information, and example information of the rubric data (900).
[0421] In the present invention, segment information may refer to location information containing an inference structure corresponding to a specific inference pattern among long text inference information. More specifically, segment information may include location information corresponding to at least one of a sentence unit, a paragraph unit, or a semantic unit of long text inference information.
[0422] Additionally, feature description information may refer to information that explains the characteristics of the inference structure included in the long-text inference information. More specifically, feature description information may include information regarding whether the long-text inference information unfolds sequentially, whether multiple perspectives are examined together, whether intermediate judgments are repeatedly verified, whether judgments are made based on data, whether judgments are made based on hypotheses, or whether a conclusion is presented first followed by detailed evidence.
[0423] For example, as illustrated in FIG. 13, the control unit (195) can generate name information of an inference pattern corresponding to the long text inference information. In one embodiment, if the long text inference information includes an iterative verification structure, the control unit (195) can generate name information corresponding to the long text inference information as verification-centered inference. In another embodiment, if the long text inference information includes a sequential explanation structure, the control unit (195) can generate name information corresponding to the long text inference information as sequential expansion-centered inference. In yet another embodiment, if the long text inference information includes a structure that compares multiple judgment structures, the control unit (195) can generate name information corresponding to the long text inference information as parallel review-centered inference.
[0424] Additionally, the control unit (195) may generate supporting information explaining the reason why the text inference information is classified into a specific inference pattern. For example, the control unit (195) may generate supporting information explaining the reason why the text inference information is classified as verification-centered inference if at least one of a sentence confirming an intermediate judgment, a sentence reviewing the possibility of error, and a sentence comparing alternative judgments is included in the text inference information. In another embodiment, the control unit (195) may generate supporting information explaining the reason why the text inference information is classified as sequential expansion-centered inference if the text inference information includes a sequential calculation structure or a sequential explanation structure. In yet another embodiment, the control unit (195) may generate supporting information explaining the reason why the text inference information is classified as parallel review-centered inference if the text inference information includes a structure that reviews multiple perspectives or multiple candidates together.
[0425] Additionally, the control unit (195) can generate segment information corresponding to a specific inference pattern in the long text inference information. In one embodiment, the control unit (195) can generate segment information for the beginning part when a conclusion-first central inference structure is included in the beginning part of the long text inference information. In another embodiment, the control unit (195) can generate segment information for the middle part when an iterative verification structure is included in the middle part of the long text inference information. In yet another embodiment, the control unit (195) can generate segment information for the conclusion part when a comparison explanation structure is included in the conclusion part of the long text inference information.
[0426] Additionally, the control unit (195) can generate feature description information for long text inference information. For example, the control unit (195) can generate feature description information that explains whether specific long text inference information has a structure that accumulates detailed facts to reach a conclusion, a structure that explains details after establishing an overall structure, a structure that performs iterative verification, a structure that examines multiple perspectives together, or a structure that presents a conclusion first and then explains the grounds.
[0427] Additionally, the control unit (195) may generate analysis information including a correspondence relationship between the text inference information and a plurality of representative classification criteria (800). For example, the control unit (195) may generate analysis information including which of the first representative classification criterion (810), the second representative classification criterion (820), and the third representative classification criterion (830) each inference segment included in the text inference information corresponds to.
[0428] Additionally, the control unit (195) may configure a prompt to generate an inference pattern analysis report and input the prompt into an artificial intelligence model. For example, the control unit (195) may configure a prompt including text inference information, rubric data (900), inference pattern classification results, and representative classification criteria (800), and input the prompt into an artificial intelligence model to generate an inference pattern analysis report.
[0429] In one embodiment, the control unit (195) may configure a prompt to explain the basis on which the text inference information is classified into a specific inference pattern. In another embodiment, the control unit (195) may configure a prompt to explain a section containing a specific inference structure in the text inference information. In yet another embodiment, the control unit (195) may configure a prompt to organize a plurality of inference structures included in the text inference information into an explainable form. Such a prompt may be configured as in the following embodiment (see Table 3 below).
[0430] You are an expert at analyzing reasoning patterns in model responses. You'll be provided with: 1. A rubric describing two distinct reasoning patterns (Pattern A and Pattern B) 2. A model response to analyzeYour task is to create a detailed analysis report that determines which pattern the response exhibits.Analysis Process:1. Carefully examine the response against both pattern definitions in the rubric2. Identify specific elements, structures, and linguistic features in the response that align with either pattern3. Note any mixed signals or elements that span both patterns4. Determine which pattern (A or B) the response most closely matchesReport Structure:1. **Initial Observations** (2-3 sentences summarizing key features of the reasoning approach)2. **Evidence for Pattern A**:- If applicable, quote 1-2 specific segments from the response that demonstrate Pattern A- Explain how these segments match characteristics described in the rubric3.**Evidence for Pattern B**:- If applicable, quote 1-2 specific segments from the response that demonstrate Pattern B- Explain how these segments match characteristics described in the rubric4. **Pattern Determination**:- Explain which pattern (A or B) is most dominant and why- Address any aspects that show characteristics of both patterns5. **Conclusion**:- Clearly state the final pattern determination using the format: "Final pattern determination:[PATTERN NAME]"Focus on concrete evidence from the response that matches specific elements from the rubricpatterns.Rubric: rubricResponse to analyze: response.
[0431] Additionally, the control unit (195) can check whether the generated inference pattern analysis report satisfies pre-set verification conditions. For example, the control unit (195) can check whether the inference pattern analysis report includes at least one of name information, basis information, interval information, and feature description information. In another embodiment, the control unit (195) can check whether the basis information included in the inference pattern analysis report corresponds to the inference structure of the long text inference information. In yet another embodiment, the control unit (195) can check whether the interval information included in the inference pattern analysis report corresponds to the sentence unit, paragraph unit, or semantic unit of the long text inference information.
[0432] Additionally, the control unit (195) can compare and analyze inference pattern analysis reports for multiple long text inference information. For example, the control unit (195) can analyze differences in inference structures of long text inference information generated by different artificial intelligence models. In another embodiment, the control unit (195) may analyze inference pattern features that are repeatedly included in a specific query type. In yet another embodiment, the control unit (195) may analyze inference patterns that are repeatedly included in a specific data set or a specific domain.
[0433] Additionally, the control unit (195) may store data related to the generated classification result in the storage unit (120) or a previously specified storage unit (300). For example, the control unit (195) may store at least one of representative classification criteria (800), rubric data (900), classification result of inference pattern corresponding to long text inference information, data related to the classification result, and query in the previously specified storage unit (300).
[0434] In one embodiment, the control unit (195) may store the results of classifying inference patterns by long text inference information. In another embodiment, the control unit (195) may store the results of analyzing inference patterns by section of the long text inference information. In yet another embodiment, the control unit (195) may store the inference pattern analysis report and the long text inference information in a connected structure.
[0435] Additionally, the control unit (195) may store at least one of the inference pattern distribution information by query type, inference pattern distribution information by domain, inference pattern distribution information by model, and inference pattern information corresponding to response performance.
[0436] Here, the inference pattern information corresponding to the response performance may refer to information including the correspondence between the inference pattern of the long text inference information and whether the response generated based on the long text inference information is correct or safe. The inference pattern information corresponding to the response performance may be used as reference information to specify a target inference pattern (1002) corresponding to a new query by calculating at least one of the response correct answer ratio information or safe response ratio information by inference pattern, and may also be used as information to analyze an inference structure available in a specific query or a specific query type.
[0437] For example, the control unit (195) may store inference pattern information associated with a high accuracy rate in a specific type of query. In another embodiment, the control unit (195) may store inference pattern feature information that is repeatedly included in a specific artificial intelligence model. In yet another embodiment, the control unit (195) may store inference pattern information that is related to the generation of safe responses.
[0438] Additionally, the control unit (195) can perform an operation to calculate inference pattern information corresponding to the response performance. More specifically, the control unit (195) can calculate response performance information by inference pattern using the classification result of the inference pattern corresponding to each of the plurality of long text inference information and information on whether the response generated based on the long text inference information satisfies the correct answer information or safe response criteria.
[0439] In one embodiment, the control unit (195) can calculate correct answer ratio information corresponding to the first inference pattern (551) based on the number of long text inference information classified by a first inference pattern (551) of a specific representative classification criterion (810, 820, 830, etc.) among a plurality of long text inference information, and the number of long text inference information classified by the first inference pattern (551) in which the response matches the correct answer information. In another embodiment, the control unit (195) can calculate safe response ratio information corresponding to the second inference pattern based on the number of long text inference information classified by a second inference pattern among a plurality of long text inference information in which the response satisfies the safe response criterion.
[0440] Here, correct response rate information and safe response rate information may be examples of response performance information. In the present invention, response performance information may refer to conditional information corresponding to the extent to which a correct response or a safe response is derived when an inference pattern is used for response generation.
[0441] Additionally, the control unit (195) can identify an inference pattern among the plurality of inference patterns (550) that is relatively more suitable for generating a response by comparing the response performance information calculated for each of the plurality of inference patterns (550) corresponding to the same representative classification criterion. In one embodiment, when response performance information for detailed fact-based analysis-centered inference and overall structure-based analysis-centered inference is calculated for the representative classification criterion corresponding to the analysis perspective, the control unit (195) can identify the inference pattern with higher response performance information among the two inference patterns as the inference pattern that is relatively more suitable for generating a response.
[0442] Meanwhile, according to one embodiment of the present specification, the response generation method according to the present invention can predict the optimal inference type (or optimal thought pattern) for a given query using only the context information of the input query, and the accuracy and safety of the response can be improved by inputting an inference guide corresponding to the predicted optimal inference type together with the artificial intelligence model. According to one embodiment, in an evaluation environment using an inference benchmark dataset such as GPQA-Diamond, the accuracy of a response to which the inference type determination method according to the present invention is applied can be improved compared to the baseline, and the safety score of the response can also be improved. However, the above figures correspond to examples of one embodiment, and the effects of the present invention are not limited to the above range of figures.
[0443] Additionally, the control unit (195) can calculate response performance information by classifying it into at least one of a data set unit, a domain unit, a query type unit, and an artificial intelligence model unit. In one embodiment, the control unit (195) can calculate the correct answer response rate information by inference pattern in a math problem solving data set and the safe response rate information by inference pattern in a safety-related data set separately from each other. In another embodiment, the control unit (195) may calculate response performance information by query unit, reflecting the fact that the inference pattern with the highest response performance information may differ for each query (411, 412, 413, etc.) even within the same data set.
[0444] Additionally, the control unit (195) can store the calculated response performance information in the storage unit (120) or the previously specified storage (300) and use it as information to specify a target inference pattern (1002) corresponding to a new query (1001) in a subsequent step.
[0445] Additionally, the control unit (195) can analyze how the inference structure included in the long text inference information varies depending on the artificial intelligence model, dataset, domain, or query type, based on data related to the generated classification result. In one embodiment, the control unit (195) can compare the distribution of inference patterns included in the long text inference information of different artificial intelligence models. In another embodiment, the control unit (195) may compare the distribution of inference patterns included in the same artificial intelligence model in different datasets. In yet another embodiment, the control unit (195) may analyze whether the distribution of inference patterns included in the long text inference information varies depending on the format difference of the training data.
[0446] Furthermore, in the present invention, generating data related to a classification result can be understood as generating at least one of a classification result of an inference pattern corresponding to long-text inference information and an inference pattern analysis report explaining said classification result. In particular, the present invention can generate an inference pattern analysis report that organizes the inference structure and inference features contained in the long-text inference information itself into an explainable form.
[0447] In addition, the present invention analyzes the inference structure of long text inference information in segment units and generates segments containing inference features and classification grounds together, thereby enabling the inference structure of long text inference information to be analyzed into an interpretable form.
[0448] In addition, the present invention can construct a data structure for analyzing inference patterns by query type, analyzing inference patterns based on response performance, analyzing inference structures by artificial intelligence model, analyzing inference structures by dataset, and generating inference instruction information based on the generated inference pattern classification results and inference pattern analysis reports.
[0449] Additionally, the present invention can be utilized as a data basis for specifying a target inference pattern corresponding to a query based on representative classification criteria (800), rubric data (900), inference pattern classification results, and an inference pattern analysis report, and for configuring a prompt (1003) including inference instruction information (1004) corresponding to the target inference pattern.
[0450] Meanwhile, according to one embodiment of the present specification, the inference guide is data that may be pre-constructed or dynamically determined in correspondence with the target inference type, and may include at least one of name information of the target inference type, definition information of an inference method according to the target inference type, and guidance information that guides the inference procedure according to the inference method. More specifically, the name information may be a label in a form identifiable by a human or artificial intelligence model, such as "step-by-step arithmetic calculation inference type," "hypothesis testing inference type," or "comparative analysis inference type." The definition information may be information describing the essential characteristics of the inference method in natural language or a structured form, and may be expressed in a form such as, "The step-by-step arithmetic calculation inference type is an inference method that derives a final result by decomposing numerical operations step-by-step and specifying intermediate results." The guidance information may be information describing a step-by-step procedure or formal guideline for following the inference method, and may be expressed in a form such as, "First, the input numerical values are separated and listed, partial results are calculated for each step, and finally, the partial results are summed to output the final result." The artificial intelligence model described above can perform inference according to the target inference type based on the inference guide included in the input data, and can generate the response to the query based on the inference result. That is, the artificial intelligence model recognizes which inference method should be applied from the name information, understands the essential characteristics of the inference method from the definition information, and proceeds with inference by following specific procedures from the guidance information. This operation can provide the effect of enabling the adjustment of the response flow at the time of inference without additionally training the weights of the artificial intelligence model.
[0451] Meanwhile, the term 'reasoning instruction information' (or reference numeral 1004) used in this specification is intended to refer to a concept equivalent to the 'reasoning guide' described in the claims, and may also be referred to in other parts of this specification as 'reasoning guidance information', 'thought flow guidance data', 'reasoning prompt component', 'reasoning directive', etc., and all such expressions should be understood as meaning information for guiding an artificial intelligence model to perform reasoning according to a specific reasoning type or reasoning pattern.
[0452] Meanwhile, the term 'prompt' (or reference numeral 1003) used in this specification is intended to refer to the same concept as 'input data' in the claims, and may also be referred to in other parts of this specification as 'input prompt', 'model input', 'query prompt', 'input prompt', etc., and all such expressions should be understood as meaning data that is input into an artificial intelligence model to cause the artificial intelligence model to generate a response.
[0453] That is, the data structure generated in the present invention can be used to specify a target inference pattern (1002) suitable for a query (1001) in an inference process, and to configure a prompt (1003) so that a response generation according to the target inference pattern (1002) is performed.
[0454] Below, we will examine in more detail the response generation method according to the present invention, which generates a response (1110) by configuring a prompt (1003) to perform inference according to the target inference pattern (1002) based on the generated inference pattern classification result, representative classification criteria (800), and rubric data (900). FIG. 14 is a flowchart for explaining the response generation method according to the present invention, and FIG. 15, FIG. 16, FIG. 17, FIG. 18 and FIG. 19 are conceptual diagrams for explaining the response generation method according to the present invention.
[0455] Meanwhile, as illustrated in FIG. 14, the response generation method according to the present invention may include the steps of obtaining at least one query (S710), verifying attribute information corresponding to the query (S720), specifying a target inference pattern among a plurality of inference patterns based on the attribute information that corresponds to the query and is used for inferring a response to the query (S730), configuring a prompt using at least one of the target inference pattern and the query so that the artificial intelligence model performs inference according to the target inference pattern (S740), and inputting the prompt to the artificial intelligence model (S750). Subsequently, the step of receiving a response generated based on the target inference pattern from the artificial intelligence model (S760) may be performed.
[0456] For example, as illustrated in FIG. 15, the control unit (195) can receive (or obtain) a query (1001) from a user through a service page (1000) linked with a user terminal. Here, the service page (1000) may refer to a user interface for providing at least one of a question-and-answer service, an inference support service, a problem-solving service, a conversational response service, and a user request-based response generation service using an artificial intelligence model (200). Additionally, the service page (1000) may be provided in the form of at least one of a web page, an application execution screen, a conversational interface, a chat interface, and an input screen provided to the user terminal.
[0457] For example, as illustrated in FIG. 15, an input area for entering a query (1001) may be provided on the service page (1000) of the user terminal, and the user may enter a query (1001) such as “calculate the result of adding 23 and 47 step by step” through the input area. The control unit (195) may receive the query (1001) from the user terminal through the input unit (101) or the communication unit (110).
[0458] In the present invention, a query (1001) may refer to input information for generating a response using an artificial intelligence model (200). More specifically, the query (1001) may refer to natural language-based input information including at least one of information, calculation, judgment, explanation, summary, analysis, and problem-solving requests desired by a user. Additionally, the query (1001) may be input information in the form of a short sentence, input information in the form of a long sentence consisting of multiple sentences, and may be configured to include at least one of text, voice, image description information, and structured request information.
[0459] In one embodiment, the query (1001) may be directly entered through a service page (1000) displayed on a user terminal. In another embodiment, the query (1001) may be transmitted through an external application program interface. In yet another embodiment, the query (1001) may be transmitted from an external server, database, or other artificial intelligence service. Accordingly, the present invention can obtain input information that is the subject of response generation using an artificial intelligence model (200) without restricting the path through which the query (1001) is entered.
[0460] The control unit (195) can collect the received query (1001) through the data collection unit (130) and store it in the storage unit (120) or a previously specified storage unit (300). Here, the previously specified storage unit (300) may refer to a database included in or linked to the storage unit (120), and may refer to a storage area that stores information used to generate the query (1001) and the response corresponding to the query (1001).
[0461] In one embodiment, the control unit (195) may store the query (1001) as the query sentence itself. In another embodiment, the control unit (195) may store at least one of query input time information, service type information where the query was entered, user terminal identification information, and query input path information together with the query (1001). In yet another embodiment, the control unit (195) may convert and store the query (1001) into at least one of text data, voice-converted text data, image description text data, or structured input data so that it can be used to verify attribute information.
[0462] The control unit (195) can verify attribute information corresponding to the query (1001). Here, the attribute information may refer to information used to specify a target inference pattern (1002) suitable for the query (1001), and may include at least one of semantic information, expression structure information, response request form information, query type information, and inference request characteristic information of the query (1001).
[0463] For example, as illustrated in FIG. 16, the control unit (195) can check attribute information using a query (1001). In one embodiment, the query (1001) may be input information such as "calculate the result of adding 23 and 47 step by step," and the control unit (195) can check at least one of an arithmetic calculation request, a reasoning order-based explanation request, and a calculation result derivation request as attribute information from the query (1001).
[0464] In the present invention, attribute information may refer to information for determining whether it is appropriate for an artificial intelligence model (200) to perform inference according to a target inference pattern (1002) for generating a response to a query (1001). More specifically, attribute information may refer not only to information indicating words included in the sentence of the query (1001), but also to information indicating the response configuration form and inference requirement characteristics required by the query (1001).
[0465] In one embodiment, the control unit (195) can determine the response request type information by analyzing the expression information included in the query (1001). For example, the control unit (195) can determine which response type the query (1001) requires among an explanation based on reasoning order, an explanation based on evidence, a comparison explanation, and a safety-centered response by using expressions such as "step by step," "in order," "explain the basis," "compare," and "explain safely."
[0466] In another embodiment, the control unit (195) can determine the query type information based on the semantic information of the query (1001). For example, the control unit (195) can determine whether the query (1001) is an arithmetic calculation request, a logical reasoning request, a comparison judgment request, an information summary request, a safety judgment request, or an explanation generation request.
[0467] In another embodiment, the control unit (195) can determine inference request characteristic information by analyzing the expression structure of the query (1001). For example, the control unit (195) can determine whether the query (1001) requires a single result derivation, a comparison of multiple candidates, a basis-based explanation, or an inference order-based explanation.
[0468] Additionally, the control unit (195) may store attribute information corresponding to the query (1001) in the storage unit (120) or a previously specified storage unit (300). At this time, the attribute information may include at least one of semantic information, expression structure information, response request form information, query type information, and inference request characteristic information of the query (1001).
[0469] As described above, in the present invention, verifying attribute information corresponding to a query (1001) can be understood as structuring the response configuration form and inference request characteristics corresponding to the query (1001) and verifying information that can be used to specify a target inference pattern (1002).
[0470] The control unit (195) can specify a target inference pattern (1002) that corresponds to a query (1001) among a plurality of inference patterns and is used for inferring a response to the query (1001) based on attribute information.
[0471] For example, as illustrated in FIG. 16, the control unit (195) can specify a target inference pattern (1002) using attribute information corresponding to a query (1001). In one embodiment, the query (1001) may be input information such as "calculate the result of adding 23 and 47 step by step," and the control unit (195) can specify a "step-by-step arithmetic calculation inference pattern" as the target inference pattern (1002) using inference order-based explanation request information included in the query (1001).
[0472] In the present invention, the target inference pattern (1002) may refer to an inference method that is induced to be performed by an artificial intelligence model (200) for generating a response to a query (1001). More specifically, the target inference pattern (1002) may include inference method information used to determine at least one of a response configuration form, an inference direction, and an inference order corresponding to the query (1001).
[0473] The control unit (195) can calculate association information between the query (1001) and each of the multiple reference queries using attribute information corresponding to the query (1001). Here, the reference query may refer to a query in which a correspondence relationship is established with at least one inference pattern among the multiple queries stored in a specified storage (300). More specifically, the reference query may refer to a query connected to the inference pattern and response performance information corresponding to the query.
[0474] Meanwhile, according to one embodiment of the present specification, the step of specifying the target inference type may be performed in an environment where the plurality of inference types correspond to each of the plurality of different reference queries. In the present specification, "reference query" may refer to a query pre-registered as reference material for determining the inference type, and at least one inference type and corresponding response performance information may be mapped and stored in each reference query. More specifically, the response generation system may calculate association information based on at least one of semantic similarity, domain match, or embedding distance between the query and each of the plurality of reference queries, and may select at least one specific reference query that satisfies a preset threshold condition based on the calculated association information. In addition, the response generation system may verify the response performance information of each inference type corresponding to the specific reference query and specify the inference type with the best response performance information as the target inference type. In this specification, "response performance information" is information indicating the evaluation result of a response to which the corresponding inference type is applied, and may include at least one of accuracy information indicating the degree to which the response corresponds to the correct answer, safety information indicating the degree to which the response corresponds to a preset safety standard, and user feedback information regarding the response.
[0475] In the present invention, the association information may refer to information indicating the relationship between a query (1001) and a reference query. More specifically, the association information may be determined based on at least one of semantic similarity between queries, similarity of expression structure, similarity of response request form, similarity of query type, and similarity of inference request characteristics.
[0476] In one embodiment, the control unit (195) can convert semantic information of a query (1001) into information in the form of a vector and calculate semantic similarity between queries by comparing it with semantic information of a reference query. In another embodiment, the control unit (195) can calculate expression similarity with the reference query using expression structures and response request expressions included in the query (1001). In yet another embodiment, the control unit (195) can calculate inference request similarity between queries using query type information and inference request characteristic information.
[0477] The control unit (195) may select at least one specific reference query associated with the query (1001) among a plurality of reference queries based on the calculated association information. For example, the control unit (195) may select a reference query having the highest semantic similarity to the query (1001) as the specific reference query. In another embodiment, the control unit (195) may select a reference query with a similar response request form as the specific reference query. In yet another embodiment, the control unit (195) may select a reference query with similar query type and inference request characteristics.
[0478] Additionally, the control unit (195) can specify a target inference pattern (1002) based on at least one inference pattern corresponding to a specific standard query. For example, the control unit (195) can specify the inference pattern that has the highest correlation with the attribute information of the query (1001) among a plurality of inference patterns corresponding to a specific standard query as the target inference pattern (1002).
[0479] In one embodiment, the control unit (195) can check response performance information corresponding to each of a plurality of inference patterns corresponding to a specific reference query. Here, the response performance information may refer to information corresponding to the result of a response generation when one inference pattern is used to generate a response. More specifically, the response performance information may include at least one of whether the response is correct, whether the response is safe, and a response performance value for each inference pattern.
[0480] The control unit (195) may specify one of a plurality of inference patterns as a target inference pattern (1002) based on response performance information. For example, the control unit (195) may specify an inference pattern that corresponds to a reference query having a high correlation with the query (1001) and has high response performance information as the target inference pattern (1002). In another embodiment, the control unit (195) may specify an inference pattern with high response performance information related to whether a safe response is given for a query (1001) containing a safety-centered response request as the target inference pattern (1002).
[0481] Additionally, the control unit (195) can identify a target inference pattern (1002) using a pre-trained classification model. Here, the pre-trained classification model may refer to a model that is trained in advance to take a query as input information and to take inference pattern identification information corresponding to each of the representative classification criteria (800) as output information.
[0482] Meanwhile, according to one embodiment of the present specification, the step of specifying the target inference type may be performed using a pre-trained classification model. More specifically, the response generation system may input the context information into the classification model and receive the target inference type output from the classification model, and in the present specification, "classification model" may refer to a model that has learned the relationship between the context information of a reference query input as training data and the response performance information of an inference type corresponding to the reference query. The response generation system may determine the output of the classification model as the target inference type if the confidence score of the target inference type output from the classification model is greater than or equal to a preset threshold, and, on the other hand, if the confidence score is less than the preset threshold, the target inference type may be determined based on a preset basic inference type or reference query matching.
[0483] In one embodiment, the pre-trained classification model may be characterized by being trained using long-text inference information in which response performance information satisfies a preset criterion and a query corresponding to said long-text inference information as training data. More specifically, the pre-trained classification model may be trained to output inference pattern identification information suitable for said query when a query is input by using as training data pairs of inference pattern classification results and long-text inference information in which the response matches the correct answer information or long-text inference information in which the response satisfies a safe response criterion.
[0484] At this time, the control unit (195) inputs a new query (1001) into a pre-trained classification model and can identify an inference pattern corresponding to the inference pattern identification information output from the pre-trained classification model as a target inference pattern (1002) corresponding to the new query (1001).
[0485] Additionally, the control unit (195) may train and utilize a plurality of classification models corresponding to each of the representative classification criteria (800). In one embodiment, the control unit (195) trains a first classification model corresponding to the analysis perspective, a second classification model corresponding to the verification focus, and a third classification model corresponding to the idea development, respectively, and when a new query (1001) is input, it may combine the inference pattern identification information output from each of the first classification model, the second classification model, and the third classification model to identify a target inference pattern (1002).
[0486] Additionally, the pre-trained classification model in the present invention may be implemented in at least one form among an in-domain classification model and a cross-domain classification model, depending on the domain range of the dataset used for training. In one embodiment, the in-domain classification model may refer to a model configured to be trained using query and inference pattern classification results corresponding to a single dataset as training data, and to output a target inference pattern (1002) for a new query (1001) belonging to the same dataset. In another embodiment, the cross-domain classification model may refer to a model configured to be trained using query and inference pattern classification results corresponding to at least two different datasets as training data, and to output a target inference pattern (1002) for a new query (1001) belonging to a dataset not used during training.
[0487] Additionally, the control unit (195) may perform at least one of the following processes, either alone or in combination: a process for specifying a target inference pattern (1002) based on association information between each of a new query (1001) and a plurality of reference queries; a process for specifying a target inference pattern (1002) based on response performance information; and a process for specifying a target inference pattern (1002) using a pre-trained classification model.
[0488] In addition, in the present invention, reflecting the fact that semantic similarity may exist between inference patterns corresponding to semantically similar queries, the control unit (195) may be configured to specify a target inference pattern (1002) based on an inference pattern corresponding to a reference query semantically similar to a new query (1001), or to output inference pattern identification information learned from a learning query semantically similar to a new query (1001) from a pre-trained classification model.
[0489] Meanwhile, according to one embodiment of the present specification, when training a classification model according to the present invention, the data format of the training data (e.g., multiple-choice format and open-ended format) may affect the distribution of inference patterns determined by the trained classification model, and accordingly, the classification model training method according to the present invention can ensure consistency in determining inference patterns in an operating environment by diversifying the data format of the training data into at least one format. In addition, according to another embodiment, if there are multiple classification models or response generation models trained in different data formats, a model merging step may be additionally performed to combine the weights of the multiple models using weight merging or linear interpolation, thereby allowing the distribution of inference patterns to be flexibly adjusted without additional training.
[0490] Additionally, the control unit (195) may store the target inference pattern (1002) in a storage unit (120) or a previously specified storage unit (300). At this time, the information stored may include at least one of a query (1001), attribute information, a specific criterion query, association information, the target inference pattern (1002), and response performance information.
[0491] In addition, specifying the target inference pattern (1002) in the present invention is not limited to simply classifying the query type, but may mean selecting an inference method suitable for the response configuration form and inference requirement characteristics corresponding to the query (1001). That is, the present invention can determine an inference method suitable for the query (1001) based on the characteristics of the query and the response performance information for each inference pattern.
[0492] The control unit (195) can configure a prompt (1003) using at least one of a target inference pattern (1002) and a query (1001) so that the artificial intelligence model (200) performs inference according to the target inference pattern (1002).
[0493] For example, as illustrated in FIG. 17, the control unit (195) can configure a prompt (1003) using a query (1001) and a target inference pattern (1002). In one embodiment, the control unit (195) can configure a prompt (1003) including inference instruction information (1004) based on a target inference pattern (1002) corresponding to the query (1001).
[0494] In the present invention, the prompt (1003) may refer to input information used by the artificial intelligence model (200) to generate a response to a query (1001). More specifically, the prompt (1003) may include at least one of a query (1001), a target inference pattern (1002), and inference instruction information (1004).
[0495] Additionally, in the present invention, the inference instruction information (1004) may refer to information for inducing an artificial intelligence model (200) to perform inference according to a target inference pattern (1002). More specifically, the inference instruction information (1004) may include at least one of inference order specification information, basis explanation request information, comparison explanation request information, safety-centered response request information, and response configuration type specification information.
[0496] In one embodiment, the control unit (195) may configure a prompt (1003) by combining the query (1001) and the inference instruction information (1004) included in the target inference pattern (1002). For example, the control unit (195) may configure a prompt (1003) that includes inference instruction information (1004) that causes the artificial intelligence model (200) to perform inference according to the calculation order in response to the query (1001) "Calculate the result of adding 23 and 47 step by step."
[0497] In another embodiment, the control unit (195) may configure the prompt (1003) using only the query (1001). In yet another embodiment, the control unit (195) may configure the prompt (1003) using inference instruction information (1004) included in the target inference pattern (1002).
[0498] The control unit (195) can determine the configuration of the prompt (1003) based on the attribute information of the query (1001) and the target inference pattern (1002). For example, if the query (1001) requires an explanation based on the inference order, the control unit (195) can configure the prompt (1003) to cause the artificial intelligence model (200) to generate a response according to the inference order. In another embodiment, if the query (1001) requires a comparison explanation, the control unit (195) can configure the prompt (1003) to induce the artificial intelligence model (200) to compare multiple judgment results.
[0499] Additionally, the control unit (195) can configure different prompts (1003) according to the target inference pattern (1002) corresponding to the query (1001). For example, the control unit (195) can configure prompts (1003) containing different inference instruction information (1004) when the target inference pattern (1002) is different for the same query (1001).
[0500] Additionally, the control unit (195) can generate a prompt (1003) using a prompt configuration unit (190). Here, the prompt configuration unit (190) may refer to a component that combines a query (1001), a target inference pattern (1002), and inference instruction information (1004) to form a prompt (1003) provided to an artificial intelligence model (200).
[0501] Additionally, the control unit (195) may store the configuration result of the prompt (1003) in the storage unit (120) or a previously specified storage (300). At this time, the information stored may include at least one of the query (1001), the target inference pattern (1002), the inference instruction information (1004), and the prompt (1003).
[0502] Additionally, configuring the prompt (1003) in the present invention can be understood as setting the input structure to include inference instruction information (1004) suitable for the inference request characteristic corresponding to the query (1001) and the target inference pattern (1002).
[0503] The control unit (195) can input a prompt (1003) into the artificial intelligence model (200). Here, the prompt (1003) may mean input information including at least one of a query (1001), a target inference pattern (1002), and inference instruction information (1004).
[0504] For example, as illustrated in FIG. 18, the control unit (195) can input a prompt (1003) configured through the prompt configuration unit (190) into the artificial intelligence model (200). In one embodiment, the prompt (1003) may include inference instruction information (1004) corresponding to a query (1001) and a target inference pattern (1002), and the artificial intelligence model (200) may perform inference according to the target inference pattern (1002) based on the inference instruction information (1004) included in the prompt (1003).
[0505] In the present invention, the artificial intelligence model (200) may refer to a model that performs natural language-based reasoning to generate a response corresponding to a query (1001). More specifically, the artificial intelligence model (200) may include at least one of a large-scale language model, a generative language model, a reasoning-specialized language model, a conversational language model, a long-text reasoning model, and a multimodal artificial intelligence model.
[0506] In one embodiment, the control unit (195) may transmit a prompt (1003) to an artificial intelligence model (200) stored on an external server using the communication unit (110). In another embodiment, the control unit (195) may input a prompt (1003) to a local artificial intelligence model (200) stored in the storage unit (120). In yet another embodiment, the control unit (195) may transmit a prompt (1003) to an artificial intelligence model (200) provided through an external application program interface.
[0507] Additionally, the control unit (195) can input different prompts (1003) to the artificial intelligence model (200) according to the target inference pattern (1002) corresponding to the query (1001). For example, the control unit (195) can input a prompt (1003) corresponding to a stepwise arithmetic calculation inference pattern or a prompt (1003) corresponding to a comparison explanation-centered inference pattern for the same query (1001).
[0508] The control unit (195) can control the inference direction of the artificial intelligence model (200) based on inference instruction information (1004) included in the prompt (1003). For example, the control unit (195) can input a prompt (1003) containing inference order specification information to induce the artificial intelligence model (200) to perform a judgment according to the inference order. In another embodiment, the control unit (195) can input a prompt (1003) containing comparison explanation request information to induce the artificial intelligence model (200) to compare multiple judgment results.
[0509] Additionally, the control unit (195) can control the inference order of the artificial intelligence model (200) based on inference instruction information (1004). For example, the control unit (195) can input a prompt (1003) to cause the artificial intelligence model (200) to perform inference for a query (1001) in multiple sequences. In this case, the artificial intelligence model (200) can generate intermediate inference results according to the inference order and generate a response to the query (1001) based on the generated intermediate inference results.
[0510] Meanwhile, according to one embodiment of the present specification, the artificial intelligence model may be a chain-of-thought-based inference model. In this case, the artificial intelligence model may generate at least one intermediate inference result by determining an inference order for the query based on the inference guide and then performing step-by-step inference according to the determined inference order. In the present specification, "intermediate inference result" may be used to mean at least one of a partial inference result, partial conclusion information, verification result, and inference hypothesis information derived during the progression of the chain of thought. More specifically, a partial inference result may refer to a step-by-step output derived during the progression of the chain of thought, partial conclusion information may refer to a conclusion tentatively derived at a part of the chain of thought, and a verification result may refer to the result of self-verification or hypothesis verification performed at a part of the chain of thought. For example, when a query such as "calculate the result of adding 23 and 47 step by step" is input and a step-by-step arithmetic calculation reasoning type is specified as the target reasoning type, the artificial intelligence model can generate a series of step-by-step thought flows as intermediate reasoning results, such as "first add the ones digit," "next add the tens digit," "process any carry-overs," and "the final result is 70," and can generate a final response such as "the sum of 23 and 47 is 70" based on the intermediate reasoning results.
[0511] In the present invention, the intermediate inference result may refer to information generated during the inference process performed by the artificial intelligence model (200) to generate a final response. More specifically, the intermediate inference result may include at least one of a calculation result, a judgment result, a comparison result, and explanatory information generated during the inference process.
[0512] In one embodiment, the control unit (195) may input a prompt (1003) so that the artificial intelligence model (200) generates multiple intermediate inference results according to the calculation order when the query (1001) is a request for arithmetic calculation. In another embodiment, the control unit (195) may input a prompt (1003) so that the artificial intelligence model (200) generates an intermediate inference result that compares multiple judgment results when the query (1001) is a request for comparison explanation.
[0513] In the present invention, inputting a prompt (1003) into an artificial intelligence model (200) can be understood as providing an input structure so that the inference flow and response generation flow of the artificial intelligence model (200) are controlled according to a target inference pattern (1002) corresponding to a query (1001).
[0514] The control unit (195) may receive a response (1110) generated based on a target inference pattern (1002) from an artificial intelligence model (200). Here, the response (1110) may refer to output information containing result information generated by the artificial intelligence model (200) in response to a query (1001).
[0515] For example, as illustrated in FIG. 18, the control unit (195) inputs a prompt (1003) to the artificial intelligence model (200) and can cause the artificial intelligence model (200) to perform inference according to a target inference pattern (1002) based on inference instruction information (1004) included in the prompt (1003). At this time, the artificial intelligence model (200) can determine the inference order for a query (1001) based on the inference instruction information (1004), generate an intermediate inference result according to the inference order, and generate a response (1110) to the query (1001) based on the intermediate inference result.
[0516] Additionally, for example, as illustrated in FIG. 19, the control unit (195) may provide the generated response (1110) to the user terminal through a service page (1100). Here, the service page (1100) may refer to a user interface for displaying a response (1110) corresponding to a query (1001). The control unit (195) may use an output unit (102) to ensure that the response (1110) is provided through at least one of the screen of the user terminal, an interactive interface, a voice output interface, or an external service linkage screen.
[0517] In the present invention, the response (1110) may include result information for the query (1001), at least one of inference sequence information corresponding to the target inference pattern (1002), explanatory information based on the intermediate inference result, and judgment basis information.
[0518] In one embodiment, when the target inference pattern (1002) is a stepwise arithmetic calculation inference pattern, the artificial intelligence model (200) determines the calculation order, generates an intermediate inference result according to the determined calculation order, and can generate a response (1110) to a query (1001) based on the generated intermediate inference result. For example, if the query (1001) is "Calculate the result of adding 23 and 47 stepwise," the response (1110) may include result information such as "Adding 23 and 47 results in 70," along with explanatory information corresponding to the calculation order.
[0519] In another embodiment, when the target inference pattern (1002) is a comparison explanation-centered inference pattern, the artificial intelligence model (200) can compare a plurality of judgment results and generate a response (1110) based on the comparison results.
[0520] The control unit (195) can check whether the target inference pattern (1002) is reflected in the response (1110) by using the expression structure included in the response (1110). In one embodiment, the control unit (195) can check whether the artificial intelligence model (200) has performed inference according to the target inference pattern (1002) by using at least one of the inference order information, explanation information based on intermediate inference results, and judgment basis information included in the response (1110).
[0521] Additionally, the control unit (195) may store at least one of a response (1110), a query (1001), a target inference pattern (1002), inference instruction information (1004), and a prompt (1003) in a storage unit (120) or a previously specified storage (300).
[0522] Additionally, receiving a response (1110) in the present invention can be understood as collecting a response (1110) that reflects an inference sequence and intermediate inference results corresponding to a target inference pattern (1002). Furthermore, the control unit (195) may provide the response (1110) to a user or store it in a designated storage (300).
[0523] Meanwhile, the response generation system (100) according to the present invention described above can be applied and utilized in various artificial intelligence services or inference-based systems.
[0524] In one embodiment, the response generation system (100) according to the present invention may be applied to a question-and-answer service that generates a response to a user query using a large-scale language model (LLM).
[0525] Specifically, the control unit (195) receives a query (1001) through a user terminal, checks attribute information corresponding to the query (1001), and can identify a target inference pattern (1002) corresponding to the query (1001) among a plurality of inference patterns based on the attribute information. Additionally, the control unit (195) can configure a prompt (1003) using at least one of the target inference pattern (1002) and the query (1001) so that the artificial intelligence model (200) performs inference according to the target inference pattern (1002). Additionally, the control unit (195) inputs the prompt (1003) to the artificial intelligence model (200) and can receive a response (1110) generated based on the target inference pattern (1002) from the artificial intelligence model (200).
[0526] At this time, the response generation system according to the present invention may be configured such that different target inference patterns (1002) are specified according to the type of query (1001), the purpose of the query (1001), the inference method or response format required for the query (1001).
[0527] For example, for a mathematical problem-solving query, a stepwise computation-centered reasoning pattern can be specified as the target reasoning pattern (1002), for a document summary query, a key information extraction-centered reasoning pattern can be specified as the target reasoning pattern (1002), and for a code generation query, a procedure-based generation-centered reasoning pattern can be specified as the target reasoning pattern (1002).
[0528] In another embodiment, the response generation system (100) according to the present invention may be applied in combination with a step-by-step reasoning-based response generation service.
[0529] Specifically, the control unit (195) may specify a target inference pattern (1002) corresponding to a query (1001) and combine inference instruction information (1004) included in the target inference pattern (1002) with the query (1001) to form a prompt (1003). Accordingly, the artificial intelligence model (200) may generate a response (1110) by performing inference according to the target inference pattern (1002) based on the inference instruction information (1004) included in the prompt (1003).
[0530] Additionally, the response generation system according to the present invention may be applied to a response generation service that utilizes stepwise inference, multiple inference path-based inference, self-review-based inference, tool-use-based inference, or plan-based inference. In this case, the control unit (195) may specify one of the multiple inference patterns as a target inference pattern (1002) according to the attribute information of the query (1001), and may include inference instruction information (1004) corresponding to the specified target inference pattern (1002) in the prompt (1003).
[0531] In one embodiment, the response generation system (100) according to the present invention can be applied to a search augmentation generation-based service.
[0532] Specifically, the control unit (195) may specify a target inference pattern (1002) corresponding to a query (1001) and include documents, database information, or external service information retrieved according to the target inference pattern (1002) in the prompt (1003).
[0533] For example, for a query requiring the generation of a basis-based response, an inference pattern that sequentially reviews the basis information of the searched documents can be specified as the target inference pattern (1002). Additionally, for a comparison analysis query, an inference pattern that compares multiple search results can be specified as the target inference pattern (1002).
[0534] In another embodiment, the response generation system (100) according to the present invention may be applied to a service for enhancing safe response generation.
[0535] Specifically, when a query (1001) includes a request for a safety-related response, the control unit (195) may specify a reasoning pattern with relatively high safety response rate information among a plurality of reasoning patterns (550) corresponding to safety-related classification criteria as the target reasoning pattern (1002). In one embodiment, the control unit (195) may specify a benevolent intent reasoning pattern among a plurality of reasoning patterns (550) corresponding to content intent criteria as the target reasoning pattern (1002). In another embodiment, the control unit (195) may specify a risk prevention-centered reasoning pattern among a plurality of reasoning patterns (550) corresponding to safety priority criteria as the target reasoning pattern (1002). In yet another embodiment, the control unit (195) may specify an ethical judgment-centered reasoning pattern among a plurality of reasoning patterns (550) corresponding to ethical consideration criteria as the target reasoning pattern (1002).
[0536] Additionally, the control unit (195) may perform processing to avoid the inference pattern with relatively low safety response rate information being identified as the target inference pattern (1002). In one embodiment, the control unit (195) may perform processing to exclude a malicious intent inference pattern from the set of inference pattern candidates so that a malicious intent inference pattern among a plurality of inference patterns (550) corresponding to the content intent criteria is not identified as the target inference pattern (1002). In another embodiment, the control unit (195) may perform processing to exclude a low-risk assumption inference pattern from the set of inference pattern candidates so that a low-risk assumption inference pattern among a plurality of inference patterns (550) corresponding to the risk assessment criteria is not identified as the target inference pattern (1002).
[0537] Through this, the response generation system (100) according to the present invention can input a prompt (1003) containing inference instruction information (1004) corresponding to the target inference pattern (1002) into an artificial intelligence model (200) so that, in response to a safety-related query, a response is generated in a direction that satisfies the safety response criteria in response to an input form that induces the response to be generated in a direction that does not satisfy the safety response criteria, and so that a response is generated in a direction that satisfies the safety response criteria.
[0538] In addition, the response generation system (100) according to the present invention is not limited to a single safety determination, but can perform finely divided control at the inference pattern unit when generating safety-related responses by using together an inference pattern classification result corresponding to at least one of a map strategy criterion, an ethical consideration criterion, a safety priority criterion, a content processing criterion, a risk assessment criterion, and a content intent criterion.
[0539] In another embodiment, the response generation system (100) according to the present invention can be applied to an agent-based artificial intelligence service.
[0540] Specifically, the control unit (195) can identify a task type corresponding to a query (1001) and specify a target inference pattern (1002) corresponding to the task type. For example, a search-centered inference pattern may be applied to an information retrieval task, a sequential planning-centered inference pattern may be applied to a schedule planning task, and an execution procedure-centered inference pattern may be applied to a code execution task.
[0541] Additionally, when multiple agents perform different roles, the control unit (195) can specify a target inference pattern (1002) corresponding to the role or work type of each agent and configure the prompt (1003) input to each agent to include different inference instruction information (1004).
[0542] In another embodiment, the response generation system (100) according to the present invention can be applied to an intent classification-based conversational service.
[0543] Specifically, the control unit (195) may use the intent or intent description of the user query as attribute information of the query (1001) and specify a target inference pattern (1002) corresponding to the attribute information. For example, if the query (1001) corresponds to an intention to search for information, an inference pattern centered on evidence search may be specified; if the query (1001) corresponds to an intention to solve a problem, an inference pattern centered on step-by-step solution may be specified; and if the query (1001) corresponds to an intention to make a comparison judgment, an inference pattern centered on comparison analysis may be specified.
[0544] In one embodiment, the response generation system (100) according to the present invention can be applied to an analysis report generation service.
[0545] Specifically, the control unit (195) receives at least one of a user query, a document to be analyzed, structured data, or unstructured data, and can identify a target inference pattern (1002) corresponding to attribute information of the input information. Additionally, the control unit (195) can input a prompt (1003) containing inference instruction information (1004) corresponding to the target inference pattern (1002) into an artificial intelligence model (200) to generate a response (1110) in the form of an analysis report.
[0546] For example, a multiple data comparison-centered reasoning pattern can be applied to the generation of market analysis reports, a structure analysis-centered reasoning pattern can be applied to the analysis of technical documents, and an issue judgment-centered reasoning pattern can be applied to the analysis of legal documents.
[0547] In another embodiment, the response generation system (100) according to the present invention is not limited to a specific domain and can be applied to various industrial fields.
[0548] For example, in the medical field, medical information analysis responses can be generated using medical information review-centered reasoning patterns corresponding to medical queries. In the financial field, financial report or market outlook responses can be generated using data-based analysis-centered reasoning patterns corresponding to financial market analysis queries. In the education field, learning responses can be generated using explanatory reasoning patterns or problem-solving reasoning patterns corresponding to the learner's query type. In the legal field, legal analysis responses can be generated using legal issue analysis-centered reasoning patterns corresponding to case law analysis queries or contract analysis queries.
[0549] Additionally, the response generation system (100) according to the present invention may be applied in combination with an analysis service that uses structured data and unstructured data together. For example, the control unit (195) can identify attribute information of a query (1001) using text documents, user query logs, search data, time series data, external report data, or database information, and identify a target inference pattern corresponding to the identified attribute information.
[0550] Additionally, the response generation system (100) according to the present invention may be applied in combination with an analysis system that utilizes semantic relationship information or causal relationship information. For example, the control unit (195) may use the relationships between concepts included in the query (1001), the relationships between data to be analyzed, or the relationships between grounds required for response generation as attribute information, and may specify a target inference pattern (1002) corresponding to the attribute information.
[0551] In addition, the response generation system (100) according to the present invention may be linked with an inference data generation service or an inference pattern analysis service.
[0552] Specifically, the control unit (195) can identify a target inference pattern (1002) corresponding to a query (1001) using at least one of a classification criterion (500), an embedding vector (600), a clustering result, representative classification criteria (800), rubric data (900), or an inference pattern classification result generated in relation to text inference information.
[0553] For example, the control unit (195) can calculate association information between each of the query (1001) and a plurality of reference queries, select a specific reference query associated with the query (1001) based on the calculated association information, and specify a target inference pattern (1002) based on an inference pattern corresponding to the specific reference query.
[0554] Additionally, the control unit (195) can identify response performance information corresponding to each of at least one inference pattern corresponding to a specific standard query, and based on the response performance information, identify one of at least one inference pattern as a target inference pattern (1002).
[0555] Meanwhile, according to one embodiment of the present specification, the inference guide may be selected from among a plurality of pre-established inference guides as an inference guide corresponding to the target inference type. More specifically, the response generation system may create an inference guide corresponding to each of the plurality of inference types in a preliminary stage and store it in an inference guide repository, and when a target inference type is identified in the operation stage, the inference guide mapped to that target inference type may be retrieved from the inference guide repository and used to construct the input data. The selection of the inference guide may be performed based on a pre-established matching criterion between the plurality of inference guides and the plurality of inference types, and in the present specification, "matching criterion" may refer to a correspondence defined by at least one of the identifier (ID), name, or classification key of the inference type.
[0556] Meanwhile, according to one embodiment of the present specification, the response generation method according to the present invention may further include a feedback circuit. More specifically, the response generation system may perform the steps of calculating response performance information for the response to which the target inference type is applied, receiving user feedback information for the response, updating the response performance information based on the user feedback information, and using the updated response performance information to determine the target inference type of a subsequent query. As a result, as operational data accumulates, the response generation system of the present invention becomes progressively able to perform more consistent inference type determinations.
[0557] Meanwhile, according to one embodiment of the present specification, the classification model learning method according to the present invention is a computer-implemented method that can be performed separately from the response generation method of the present invention, and may include the steps of: extracting context information related to at least one of the meaning and domain of a reference query for each of a plurality of reference queries; generating a response by applying a plurality of inference types to each of the plurality of reference queries and calculating response performance information for the generated response; configuring a combination of the extracted context information, the applied inference type, and the calculated response performance information as training data; and training a classification model that receives context information based on the training data and outputs an inference type corresponding to the context information. More specifically, in the step of configuring the training data, the training computing system may configure the training data in the form of a triplet of (reference query context information, applied inference type, response performance score), and in the step of training, the training computing system may apply a supervised learning method in which the context information is used as input and the inference type with the highest response performance is used as a label. According to another embodiment, the classification model may include a plurality of binary classifiers corresponding to different inference method dimensions (e.g., analysis perspective dimension, search range dimension, inference type classification dimension, idea development mode dimension, verification method dimension, and clarification approach dimension), and each of the plurality of binary classifiers may be trained to output one of two inference types in the corresponding inference method dimension.
[0558] Meanwhile, according to one embodiment of the present specification, when training a classification model according to the present invention, the data format of the training data (e.g., multiple-choice format and open-ended format) may affect the distribution of inference patterns determined by the trained classification model, and accordingly, the classification model training method according to the present invention can ensure consistency in determining inference patterns in an operating environment by diversifying the data format of the training data into at least one format. In addition, according to another embodiment, if there are multiple classification models or response generation models trained in different data formats, a model merging step may be additionally performed to combine the weights of the multiple models using weight merging or linear interpolation, thereby allowing the distribution of inference patterns to be flexibly adjusted without additional training.
[0559] Meanwhile, the response generation system (100) according to the present invention described above can be implemented in the form of a program executed on a computer.
[0560] Additionally, the response generation system (100) according to the present invention may be implemented in the form of instructions or program code stored on a computer-readable recording medium.
[0561] Computer-readable recording media may include magnetic storage media, optical storage media, semiconductor storage devices, flash memory, SSD, HDD, RAM, ROM, cloud storage, or network-based storage devices.
[0562] In addition, the response generation system (100) according to the present invention may be implemented in a server computing system, a cloud computing system, or a distributed processing system environment including a plurality of processors.
[0563] In addition, the response generation system (100) according to the present invention may be implemented in a hardware environment including an FPGA, ASIC, GPU, TPU, or artificial intelligence computation acceleration device.
[0564] As described above, the response generation method and system according to the present invention can identify a target inference pattern suitable for a query among a plurality of inference patterns based on attribute information corresponding to the query. Through this, an inference pattern that matches the characteristics of the query can be selected in an environment where various inference methods can be applied even to the same query, and the problem of an artificial intelligence model generating a response based on an inference method unsuitable for the query can be structurally reduced.
[0565] In addition, the response generation method and system according to the present invention can construct a prompt by combining inference instruction information corresponding to a target inference pattern with a query. Through this, it is possible to induce the reflection of a specific inference order, verification method, idea development method, and conclusion derivation method when an artificial intelligence model generates a response, and to generate a response that reflects an inference structure suitable for the query, rather than a simple result-oriented response.
[0566] Furthermore, the response generation method and system according to the present invention can generate classification criteria for distinguishing inference patterns based on long-text inference information generated by an artificial intelligence model. Through this, classification criteria for inference patterns can be generated based on the inference methods actually appearing in the long-text inference information, without relying on fixed criteria pre-set by humans, and can flexibly reflect differences in inference methods appearing in various queries and tasks.
[0567] Furthermore, the response generation method and system according to the present invention can generate embedding vectors for each of the classification criteria and perform clustering based on the similarity between the embedding vectors. Through this, semantically similar inference pattern classification criteria can be automatically grouped, and duplicate or similar criteria can be organized to derive inference pattern classification criteria that ensure representativeness.
[0568] Furthermore, the response generation method and system according to the present invention can generate rubric data for classifying inference patterns corresponding to representative classification criteria. Through this, the definition and characteristics of each inference pattern can be provided in a structured form, and different inference patterns can be interpreted and compared according to consistent criteria.
[0569] Furthermore, the response generation method and system according to the present invention can classify inference patterns corresponding to long-text inference information using rubric data. Through this, it is possible to determine which inference pattern the long-text inference information generated by an artificial intelligence model corresponds to, and to analyze the characteristics of the inference method included in the long-text inference information.
[0570] Furthermore, the response generation method and system according to the present invention can store representative classification criteria, rubric data, classification results of inference patterns corresponding to long-text inference information, and data related to the classification results, and utilize them for processing new queries. Through this, a target inference pattern for a new query can be identified based on the previously stored information, and it becomes possible to generate a response that reflects an inference pattern suitable for the query.
[0571] Furthermore, the response generation method and system according to the present invention can generate rubric data for classifying inference patterns corresponding to representative classification criteria. Through this, the definition and characteristics of each inference pattern can be provided in a structured form, and different inference patterns can be interpreted and compared according to consistent criteria.
[0572] Furthermore, the response generation method and system according to the present invention can classify inference patterns corresponding to long-text inference information using rubric data. Through this, it is possible to determine which inference pattern the response generated by the artificial intelligence model was derived based on, and to systematically analyze the suitability of the response and the characteristics of the inference method.
[0573] Furthermore, the response generation method and system according to the present invention can store representative classification criteria, rubric data, classification results of inference patterns corresponding to long-text inference information, and data related to the classification results, and utilize them for processing new queries. Through this, accumulated data can be reused to quickly identify target inference patterns for new queries, and it becomes possible to generate responses that reflect inference patterns suitable for queries without repetitive data generation.
[0574] Meanwhile, the present invention described above may be implemented as a program that is executed by one or more processors on a computer and can be stored on a medium (or recording medium) that is readable by such a computer.
[0575] Furthermore, the response generation method and system according to the present invention can enable an artificial intelligence model to determine an inference order based on an inference guide, perform inference according to the inference order to generate an intermediate inference result, and generate a response based on the intermediate inference result. Through this, the inference type of the present invention can be reflected at the time of inference even for a chain-of-thought-based artificial intelligence model, and the step-by-step thought flow can be guided to conform to the target inference type.
[0576] Furthermore, the response generation method and system according to the present invention can identify response performance information for each inference type corresponding to a specific standard query and specify a target inference type based on the response performance information. Through this, even when multiple inference types are applicable to the same standard query, an inference type with superior response performance can be selected, and it becomes possible to determine an appropriate inference type based on operational data.
[0577] Furthermore, the data generation method according to the present invention can be configured such that the inference pattern analysis report includes at least one of the name information of the inference pattern, classification basis information, information on the section corresponding to the inference pattern within the long text inference information, and information describing the characteristics of the inference pattern. Through this, the interpretability of the inference pattern classification result can be secured, and the classification result can function as basis data when utilized in subsequent processing.
[0578] Furthermore, the data generation method according to the present invention may include the step of storing at least one of a representative classification criterion, rubric data, a classification result of an inference pattern corresponding to long text inference information, data related to the classification result, and a query in a specified storage. Through this, an inference pattern classification asset generated once can be permanently stored and called upon and reused at various application points.
[0579] Furthermore, the response generation system according to the present invention may be configured to store a plurality of inference types and inference guides in memory, and for one or more processors to determine a target inference type based on context information of a query and input it into an artificial intelligence model along with the inference guide to receive a response. Through this, the response generation flow of the present invention can be implemented in a memory-processor-based system structure, and the operation of the present invention can be performed consistently in various computing environments.
[0580] Furthermore, the classification model learning method according to the present invention can extract context information for each of a plurality of reference queries, calculate response performance information for a response to which a plurality of inference types have been applied, and train a classification model by configuring the combination of the context information, the applied inference types, and the response performance information as training data. Through this, the classification model, which is trained separately from the main flow of response generation of the present invention, can operate by receiving context information at the time of operation and outputting an appropriate inference type, and the consistency of the inference type determination of the response generation system can be gradu...
Claims
1. In a method for generating a response performed by a computer, A step of obtaining at least one query; A step of verifying context information related to at least one of the meaning and domain of the above query; A step of specifying a target inference type to be used for inference regarding the above query based on the above context information; A step of generating input data including a query and an inference guide corresponding to the target inference type so that the artificial intelligence model performs inference according to the target inference type; A step of inputting the above input data into the above artificial intelligence model; A step of receiving a response to the above query generated from the above artificial intelligence model based on an inference result according to the above target inference type, and A method for generating a response characterized by including a step of providing a response to the above query.
2. In Paragraph 1, The step of specifying the above target inference type is, A response generation method characterized by including the step of selecting, among a plurality of inference types corresponding to different inference methods, at least one inference type corresponding to the context information as the target inference type.
3. In Paragraph 2, The above reasoning guide is, A response generation method characterized by including at least one of name information of the target inference type, definition information of an inference method according to the target inference type, and guidance information guiding an inference procedure according to the inference method.
4. In Paragraph 1, The above artificial intelligence model is, Based on the inference guide included in the input data, the inference according to the target inference type is performed, and A response generation method characterized by generating the above response to the above query based on the above inference result.
5. In Paragraph 4, The above artificial intelligence model is, Determine the inference order for the above query based on the above inference guide, and Inference is performed according to the above inference order to generate intermediate inference results based on the above inference, and A response generation method characterized by generating the response to the above query based on the above intermediate inference result.
6. In Paragraph 5, The above intermediate inference result is, A response generation method characterized by including at least one of a partial inference result, partial conclusion information, and a verification result included in a chain of thoughts.
7. In Paragraph 2, The above multiple types of inference are, Corresponding to each of multiple different standard queries, The step of selecting the above target inference type is, A step of calculating association information between the above query and each of the above plurality of standard queries, Based on the association information calculated above, the step of selecting at least one specific reference query associated with the query among the plurality of reference queries, and A response generation method characterized by including the step of specifying the target inference type based on at least one inference type corresponding to the specific criterion query among the plurality of inference types above.
8. In Paragraph 7, The step of specifying the above target inference type is, A step of verifying response performance information corresponding to each of at least one inference type corresponding to the above specific standard query and A response generation method characterized by including the step of specifying any one of the at least one inference type as the target inference type based on the above response performance information.
9. In Paragraph 8, The above response performance information is, A method for generating a response characterized by including at least one of accuracy information indicating the degree to which a response to which at least one inference type is applied corresponds to the correct answer, safety information indicating the degree to which the response corresponds to a preset safety standard, and user feedback information regarding the response.
10. In Paragraph 1, The step of verifying the above context information is, A step of extracting at least one of semantic information representing the meaning of the query and domain information representing the domain of the query from the above query; and A response generation method characterized by including the step of generating context information based on at least one of the extracted semantic information and domain information.
11. In Paragraph 1, A response generation method characterized in that the above inference guide is selected as an inference guide corresponding to the target inference type among a plurality of previously established inference guides.
12. In Paragraph 11, A response generation method characterized by selecting the inference guide above, wherein the inference guide corresponding to the target inference type is selected based on a pre-established matching criterion between the plurality of inference guides and the plurality of inference types.
13. In Paragraph 1, The above query is entered by the user, and The above inference guide is determined based on the above context information, and A response generation method characterized by the above input data being generated by combining the above query and the above determined inference guide.
14. In Paragraph 1, The step of specifying the above target inference type is, The method includes the step of inputting the context information into a pre-trained classification model and receiving the target inference type from the classification model. A response generation method characterized in that the above classification model is a model that learns the relationship between context information of a reference query input as training data and response performance information of an inference type corresponding to the above reference query.
15. In Paragraph 14, The step of specifying the above target inference type is, A step of determining the output of the classification model as the target inference type when the confidence score of the target inference type output from the classification model is greater than or equal to a preset threshold, and A response generation method characterized by further including the step of determining the target inference type based on a preset basic inference type or reference query matching when the above reliability score is below the above threshold.
16. In Paragraph 1, A step of calculating response performance information for the above response to which the above target inference type is applied, Step of receiving user feedback information regarding the above response, A step of updating the response performance information based on the above user feedback information and A response generation method characterized by further including the step of using the above-mentioned updated response performance information to determine the target inference type of a subsequent query.
17. One or more processors; and A response generation system comprising a memory storing instructions executed by one or more processors, The above one or more processors, Based on the context information of the query, determine the inference type corresponding to the above query among a plurality of inference types, and Input an inference guide corresponding to the above-determined inference type into the artificial intelligence model along with the above query, and A response generation system characterized by being configured to provide a response output from the above artificial intelligence model.
18. A program that can be stored on a computer-readable recording medium, The above program is executed by one or more processors, A step of obtaining at least one query; A step of verifying attribute information corresponding to the above query; A step of specifying, among a plurality of inference patterns based on the above attribute information, a target inference pattern corresponding to the above query and to be used for inferring the response to the above query; A step of configuring a prompt using at least one of the target inference pattern and the query so that the artificial intelligence model performs inference according to the target inference pattern; The step of inputting the above prompt into the artificial intelligence model; and A program characterized by including a command that performs the step of receiving a response generated based on the target inference pattern from the artificial intelligence model.
19. In a method for training a classification model performed by a computer, For each of a plurality of reference queries, a step of extracting context information related to at least one of the meaning and domain of the reference query; A step of generating a response by applying a plurality of inference types to each of the plurality of reference queries, and calculating response performance information for the generated response; A step of configuring the combination of the extracted context information, the applied inference type, and the calculated response performance information as training data; and A method for training a classification model characterized by including the step of training a classification model that receives context information based on the above-mentioned training data and outputs an inference type corresponding to the context information.
20. In Paragraph 19, The step of calculating the response performance information includes the step of calculating the response performance information for each domain according to the domain information of each of the plurality of reference queries, and A classification model learning method characterized by the above classification model being learned based on response performance information calculated for each domain.
21. In Paragraph 1, The above artificial intelligence model includes a pre-trained large language model, and The step of generating the above input data includes the step of generating a prompt that combines an inference guide corresponding to the above target inference type and the above query. A response generation method characterized in that the artificial intelligence model is configured to generate intermediate inference results step by step according to an inference procedure defined by the inference guide included in the prompt, and to generate the response based on the intermediate inference results, thereby generating a response corresponding to the target inference type without involving parameter updates of the artificial intelligence model.
22. In Paragraph 1, The above context information is transmitted to a classification model that determines the target inference type through a first path, and an inference guide corresponding to the target inference type output from the classification model is combined with the query through a second path to form the input data, and the formed input data is configured to be transmitted to the artificial intelligence model through a third path. A response generation method characterized by being configured such that, in the case where the query is transmitted to the artificial intelligence model through a single input path by means of data flow through the first path, the second path, and the third path, an inference procedure corresponding to the target inference type is reflected in the response generation of the artificial intelligence model.
23. As a non-transient computer-readable storage medium, When executed by one or more processors, the one or more processors are allowed to, An operation to obtain at least one query; An operation to verify context information related to at least one of the meaning and domain of the above query; An action of specifying a target inference type to be used for inference regarding the above query based on the above context information; An operation to generate input data including an inference guide corresponding to the query and the target inference type, so that a pre-trained artificial intelligence model performs inference according to the target inference type; The operation of inputting the above input data into the above artificial intelligence model; The operation of receiving a response to the above query generated from the above artificial intelligence model based on the inference result according to the target inference type; and A non-transient computer-readable storage medium characterized by storing instructions that perform an action to provide a response to the above query.
