Method and system for providing follow-up question in conversational artificial intelligence service

Abstract

The present disclosure relates to a method and a system for providing a follow-up question in real time through distributed computation between a server and a client in a conversational artificial intelligence service. The server derives a conversation purpose and related knowledge through chain-of-thought (CoT) inference based on a conversation history, so as to generate and rank follow-up question candidates, stores metadata including a KV cache pointer, and then transmits a candidate pack to the client. The client constructs a prefix trie index by using the received data, searches the index locally without server communication when a user inputs a keystroke, and provides a matching candidate in real time. Through this, computational load on the server is reduced and response latency is minimized.

Description

Method and system for providing follow-up questions in conversational AI services

[0001] The present disclosure relates to a conversational artificial intelligence service, and more specifically, to a server-client distributed computation method and system for efficiently generating follow-up questions based on a conversation history between a user and a system in a conversational artificial intelligence service, ranking the generated follow-up questions, real-time matching on the client side, and generating responses to the follow-up questions.

[0002] With the widespread adoption of conversational AI services based on Large Language Models (LLMs), interest is growing in technologies that automatically generate and provide follow-up questions appropriate to the conversational context to support users' information search.

[0003] Conventional technical approaches to providing follow-up questions can be broadly classified into retrieval-based and generation-based methods. The retrieval-based method involves searching for questions highly relevant to the conversation context from a pre-built question bank and presenting them to the user.

[0004] This search-based method has a limitation in that it is difficult to provide appropriate follow-up questions when the user's intention to search for information goes beyond the scope of the question bank, as the follow-up questions that can be provided are limited by the scope of the pre-established question bank.

[0005] The generative approach involves using a large-scale language model to directly generate follow-up questions from the conversational context. While the generative approach can overcome the limitations of the search-based approach, it has been pointed out that it has limitations in that it is difficult to generate goal-oriented follow-up questions that reflect the user's ultimate purpose of information search, as it relies primarily on the contextual continuation of the user's immediate utterance.

[0006] In this regard, research is being conducted on a technology that applies a Chain-of-Thought (CoT)-based inference process to sequentially infer user intent and conversational purpose from conversation history, and optimizes the diversity and complementarity of follow-up question candidates through list-wise ranking.

[0007] One embodiment of the present disclosure aims to provide a method and system that improve the efficiency of server-side computing resource usage, reduce response delay to user input, and reduce redundant computations when generating answers when providing follow-up questions in a conversational artificial intelligence service.

[0008] Specifically, one embodiment of the present disclosure aims to solve the following technical problems.

[0009] First, instead of sending inference requests to a large-scale server-side language model whenever a user keystroke occurs, candidate matching is performed using an index structure built on the client side, thereby reducing the number of network round trips and lowering the server-side GPU computational load.

[0010] Second, when ranking follow-up question candidates, the estimated serving cost of generating an answer for each candidate question is integrated with the usability score to optimize the composition of follow-up questions provided to the user in conjunction with the utilization efficiency of server-side computing resources.

[0011] Third, the intermediate inference results generated during the generation process of follow-up question candidates are bound to each candidate as metadata, and when a user selects a follow-up question, the previously generated cache state is directly restored to the inference pipeline using the Key-Value cache (KV) pointer included in the metadata, thereby initiating answer generation without re-encoding the entire conversation history, shortening TTFT and reducing GPU prefill computation.

[0012] Fourth, when a user inputs a recommended follow-up question with partial modifications, the token divergence point with the question before modification is detected, and the intermediate operation result prior to the divergence point is selectively recycled to perform additional inference operations only on the modified part, thereby generating an answer without re-executing the entire inference pipeline.

[0013] A method for providing follow-up questions in a conversational artificial intelligence service according to an embodiment of the present disclosure for solving the above problem is a method performed by a computer device, wherein at least one processor of the computer device generates a conversation purpose and related knowledge by performing a Chain-of-Thought (CoT) based inference process using a machine learning model based on a conversation history received from a user computing device via a network; wherein the at least one processor stores a session context including the conversation history, the conversation purpose, and the related knowledge in memory, and includes a pointer to a Key-Value (KV) cache state generated in the CoT-based inference process in the session context; wherein the at least one processor generates a plurality of follow-up question candidates based on the conversation purpose and the related knowledge, performs ranking on the plurality of follow-up question candidates to determine ranked candidates, and for each of the ranked candidates, generates metadata including a pointer to a KV cache snapshot calculated during the generation process of the candidate and stores it in memory. The method may include: a step in which at least one processor comprises a candidate pack including text, token sequences, and ranking scores of the ranked candidates; a step in which the at least one processor constructs a prefix tri-index in memory based on the token sequences included in the candidate pack, and stores an identifier and a ranking score of a candidate matching a corresponding prefix in each node of the tri-index; and a step in which a processor of the user computing device, in response to a user input event, searches the tri-index with a token sequence corresponding to the text entered so far to determine a matching candidate, and implements the determined candidate in the user interface of the user computing device.

[0014] According to one embodiment of the present disclosure, a prefix trie index structure is constructed on a user computing device based on a candidate pack generated by a server computing system, and candidate matching is performed by searching the index structure for a user's keystroke input, thereby eliminating server-side inference requests whenever a keystroke event occurs. Accordingly, response delay for user input is reduced.

[0015] In addition, according to one embodiment of the present disclosure, when ranking follow-up question candidates, the ranking is determined based on an integrated score that combines a usability score and an estimated serving cost, thereby allowing the configuration of follow-up questions provided to the user to be optimized in conjunction with the utilization efficiency of server-side computing resources. Accordingly, among candidates with equal usability, candidates with a high cache hit probability, short prefill operations, or those that do not require external search are recommended preferentially, thereby reducing the average response latency and GPU computation costs of the entire service.

[0016] In addition, according to one embodiment of the present disclosure, a pre-calculated KV cache snapshot at the time of generating a follow-up question candidate is bound to the metadata of each candidate, and when a user selects a follow-up question, the bound KV cache is directly restored to the inference pipeline, thereby eliminating the re-encoding of the entire conversation history. Accordingly, when the conversation history consists of N tokens, the number of prefill operation tokens is reduced compared to the prior art, and the TTFT can be shortened by about 97% to 99%.

[0017] In addition, according to one embodiment of the present disclosure, when a user inputs a recommended follow-up question with partial modification, the amount of computation can be reduced compared to a conventional method that re-executes the entire inference pipeline by selectively reusing the intermediate computation results of the part before divergence based on the token divergence point and performing incremental inference by a lightweight model only on the part after divergence.

[0018] In addition, according to one embodiment of the present disclosure, server-side regeneration / re-ranking is conditionally performed only when the entropy of the candidate matching result on the client side exceeds a predetermined threshold by an uncertainty-based trigger mechanism, so the number of server-side GPU inference calls during the entire session can be reduced.

[0019] Due to the above technical effects, according to one embodiment of the present disclosure, in providing follow-up questions and generating answers based thereon in a conversational artificial intelligence service, efficient use of network communication and computational resources between the server and the client is achieved, while real-time responsiveness to user input can be maintained.

[0020] FIG. 1 illustrates an example of a block diagram of a computing system in which the present disclosure can be implemented.

[0021] FIG. 2 briefly illustrates a neuromorphic circuit structure that may be included in a processor of a follow-up question system in an artificial intelligence service according to one embodiment.

[0022] FIG. 3a illustrates a block diagram example of a computing device that may be included in a user computing device, a server computing system, or a training computing system as an embodiment of a computing system in which the present disclosure may be implemented.

[0023] FIG. 3b is a block diagram illustrating the internal architecture and data processing pipeline of an artificial intelligence model according to one embodiment of the present disclosure.

[0024] FIG. 4a illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system that performs follow-up questions in an artificial intelligence service according to an embodiment of the present disclosure.

[0025] FIG. 4b is a block diagram illustrating the data flow and system interaction of a follow-up question application process in an artificial intelligence service according to one embodiment of the present disclosure.

[0026] FIG. 5 illustrates a multi-agent system architecture according to one embodiment of the present disclosure.

[0027] FIG. 6 is a diagram showing the entire pipeline of a method for providing follow-up questions according to one embodiment of the present disclosure.

[0028] FIG. 7 is a diagram showing the processing flow of the first step according to one embodiment of the present disclosure.

[0029] FIG. 8 is a diagram showing the processing flow of the second step according to one embodiment of the present disclosure.

[0030] FIG. 9 is a diagram showing the relationship between a candidate-specific metadata structure and a candidate pack according to one embodiment of the present disclosure.

[0031] FIG. 10 is a flowchart illustrating the decision flow of uncertainty-based conditional re-ranking / regeneration according to one embodiment of the present disclosure.

[0032] FIG. 11 is a diagram illustrating the process of generating an answer by restoring the KV cache (prompt stitching) in the fourth step according to one embodiment of the present disclosure.

[0033] As the present disclosure is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present disclosure is not necessarily limited to what is depicted.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0035]

[0036] First, we will explain the configuration of the system that provides follow-up questions.

[0037] FIG. 1 illustrates an example of a block diagram of a computing system in which the present disclosure can be implemented.

[0038] Referring to FIG. 1, a computing system (1000) for performing a follow-up question generation method in an artificial intelligence service according to one embodiment of the present disclosure includes a user computing device (110), a training computing system (150), and a server computing system (130), and each device and system is connected to communicate through a network (170).

[0039] According to an embodiment of the present disclosure, 1) a user computing device (110) can perform a method for generating follow-up questions in an artificial intelligence service by using a local or / and external machine learning model (120) or a machine learning model (140) provided by a server.

[0040] Additionally, according to another embodiment of the present disclosure, 2) a server computing system (130) communicating with a user computing device (110) may provide a follow-up question generation service in an artificial intelligence service to the user computing device (110) on an application or / and the web in response to a request from a user through the user computing device (110).

[0041] In addition, according to another embodiment of the present disclosure, 3) a user computing device (110) and a server computing system (130) may perform at least a part of the follow-up question generation method in an artificial intelligence service in conjunction with each other to provide a follow-up question generation service to the user in an artificial intelligence service.

[0042] Additionally, according to various embodiments of the present disclosure, a user computing device (110) and / or a server computing system (130) can learn a machine learning model (120 / 140) that provides follow-up questions in a conversational artificial intelligence service through interaction with a training computing system (150) that is communicatedly connected via a network (170).

[0043] Additionally, in embodiments of the present disclosure, machine learning models (120, 140) may include various forms of AI agents or agentic architectures beyond simple prediction models.

[0044] In one embodiment, the machine learning model may be an AI agent having a structure that receives system prompts and user prompts, autonomously plans and executes tasks through core components such as planning, memory, reasoning, and tools, and improves itself through feedback.

[0045] In another embodiment, the machine learning model may include a Search Augmented Generative (RAG) architecture that retrieves relevant information from an external database and generates a response based thereon to provide an accurate answer based on the latest information or expertise.

[0046] In another embodiment, the machine learning model may be a multi-agent system in which multiple AI agents cooperate to achieve a specific goal. The multi-agent system may have a supervisory pattern in which a central supervisor agent distributes tasks to subordinate expert agents and aggregates the results. Alternatively, it may have a hierarchical pattern in which a meta-agent acts as an intermediary manager to control and coordinate subordinate agents. Furthermore, it is possible to include a multi-agent debate pattern in which multiple agents present different opinions and derive an optimal conclusion through discussion and evaluation.

[0047] The server computing system (130) can host AI agents such as those mentioned above, particularly multi-agent systems requiring complex computations or large-scale long-term memory. Additionally, the server computing system (130) includes a Multi-Channel Processing (MCP) server to manage integration with various external tools, and can relay communication with cloud APIs, payment services, search engines, etc.

[0048] The training computing system (150) can generate a ToolFormer model that learns how to use a specific tool, or perform iterative learning that gradually improves the performance of the agent through a self-reflection mechanism in which another LLM evaluates and modifies the results generated by the agent.

[0049] At this time, the training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0050] And at this time, the artificial intelligence model can be 1) trained directly locally by a user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training and learning techniques. It may also be implemented by transmitting the artificial intelligence model trained by the training computing system (150) to the user computing device (110) and / or the server computing system (130) through the network (170) to provide / update it.

[0051] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110). The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0052] The user computing device (110) includes at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a Tensor Processing Unit (TPU), a graphics processing unit (GPU), ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0053] In particular, according to the embodiment, this processor (111) may be configured based on a Field Programmable Gate Array (FPGA) implementation and / or an Application Specific Integrated Circuit (ASIC), which is a hardware technology for implementing a certain digital circuit.

[0054] Here, a field programmable gate array (FPGA) can refer to a flexible digital circuit that is programmable according to user needs.

[0055] As an example, a field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results or state information of operations to support synchronized operation of the FPGA, programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits configurable according to user needs, and an input interface that receives signals from external devices or sensors and transmits them to internal circuits as a channel for receiving data from outside the FPGA.

[0056] Through the combination of the above components, a field-programmable gate array implementation can provide flexible and various types of digital circuits.

[0057] As an example, the application-dedicated integrated circuit may include a register, which is a small memory device for temporarily storing and managing data and supports the rapid processing of ASIC operations by storing intermediate calculation results or state information; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and coordinates the operation of the entire system by performing various operations or generating control signals when necessary; and an input block, which is an interface for receiving data from the outside, which receives data to be processed by the ASIC and transmits it internally, and receives various input data through connections with sensors or external devices.

[0058] Through the combination of the components mentioned above, an application-specific integrated circuit can perform specific purpose tasks in an optimized manner.

[0059] For example, ASICs can have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits.

[0060] FIG. 2 briefly illustrates the structure of a neuromorphic circuit (300) that may be included in a processor (111, 131, 151) according to one embodiment.

[0061] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of postsynaptic neuron circuits (320), a plurality of postsynaptic lines (321) extending longitudinally from the plurality of postsynaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at the intersection of the plurality of presynaptic lines (311) and the plurality of postsynaptic lines (321).

[0062] A plurality of free synaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of free synaptic lines (311).

[0063] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0064] Furthermore, multiple post-synaptic neuron circuits (320) may transmit electrical signals to multiple synaptic circuits (330) through multiple post-synaptic lines (321).

[0065] A plurality of synapse circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0066] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the resistance value of the plurality of synaptic circuits (330) changes by a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and can store weight data according to this resistance change.

[0067] The neuromorphic circuit (300) is formed by mimicking the structure of neurons and synapses, which are essential elements of the human brain. When a deep neural network (DNN) is realized using the neuromorphic circuit (300), the data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0068] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs the storage function of memory on the internet. This memory (112) may store data and instructions necessary for the at least one processor (111) to perform the operation of an application for providing follow-up questions in a conversational artificial intelligence service.

[0069] In one embodiment, the user computing device (110) may store at least one machine learning model (120). For example, the user computing device (110) may be various machine learning models, such as multiple neural networks (e.g., deep neural networks) that provide follow-up questions in a conversational artificial intelligence service based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0070] For example, the prediction model may include linear regression, decision trees, random forests, gradient boosting, or / and follow-up question generation models in deep learning-based artificial intelligence services. The neural network model may include feedforward neural networks, recurrent neural networks (e.g., LSTM), convolutional neural networks, Transformers, or / and combinations thereof.

[0071] In one embodiment, the user computing device (110) can provide follow-up questions in real time in a conversational AI service by utilizing a follow-up question generation model in an AI service stored locally.

[0072] In another embodiment, a user computing device (110) transmits a request to generate a follow-up question in an artificial intelligence service to a server computing system (130), and the server may provide a result of generating a follow-up question in an artificial intelligence service to the user device based on at least one prediction model.

[0073] The user computing device (110) may include one or more input components (121) that detect user input. For example, the input components (121) may include a touch sensor, an image sensor, a microphone, a button, a mouse, a keyboard, etc., and may include an interface capable of receiving input from an external controller (e.g., a mouse, a keyboard, etc.).

[0074] A server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of one or more electrical computing devices for performing implicit policy alignment-based language model optimization operations, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), or an FPGA, a microprocessor, etc.

[0075] For example, ASICs may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits (see Fig. 7).

[0076] The memory (132) may include non-transient or transient storage media such as RAM, ROM, flash memory, magnetic disk, etc., and may store instructions or data for the execution and inference of a follow-up question generation model in an artificial intelligence service executed by the processor (131).

[0077] The prediction model (140) may include at least one of a feedforward neural network, a recurrent neural network (LSTM), a convolutional neural network (CNN), or a Transformer-based model.

[0078] In one embodiment, the server computing system (130) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0079] The server computing system (130) may further include a data store. This data store may consist of a relational database, a NoSQL database, a data warehouse, or a local file system.

[0080] The training computing system (150) may include at least one processor (151) and memory (152). The memory (152) may store a dataset for model training, a loss function, an optimization algorithm, parameters, etc.

[0081] For example, the training computing system (150) can update the parameters of a model that provides follow-up questions in a conversational AI service by executing a learning algorithm including backpropagation. This process may include iterative learning to improve the performance of a model that provides follow-up questions in a conversational AI service, and generalization techniques such as dropout and weight reduction may be applied.

[0082] The model trainer (160) is a logical component that executes this learning and can be implemented in the form of hardware, firmware, or software. For example, the model trainer can be executed by a processor after loading a learning program file stored in a storage device into memory.

[0083] The system can be connected via a wired / wireless network (170) for communication between a user computing device (110) and a server computing system (130). This network may include various forms such as the Internet, LAN, wireless LAN, WAN, and mobile communication network, and data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, and SSL.

[0084] This server computing system (130) may further include a plurality of specialized engines and repositories that are logically and physically separated to provide follow-up questions in conversational artificial intelligence services.

[0085] In one embodiment, the server computing system (130) may include at least one engine among a reasoning engine that processes user natural language input or system events to establish an action plan, a membership management engine, and a supervision signal generation engine.

[0086] Here, the term engine may include not only a set of instructions executed by a processor (131) to perform specific logic, but also dedicated hardware circuits to accelerate said logic.

[0087] Such engines may run on hardware accelerators optimized to handle the computational load of large-scale generative artificial intelligence models. The hardware accelerators are processors specialized for matrix operations and vector processing and may include at least one of the aforementioned Tensor Processing Unit (TPU), Graphics Processing Unit (GPU), Field-Programmable Gate Array (FPGA), or Application-Specific Integrated Circuit (ASIC). These hardware accelerators can provide technical improvements that distribute the computational load of large-scale language models (LLM) and / or diffusion models and enable real-time inference.

[0088] Additionally, the data (133) may not be a simple set of data, but may include a structured embedding repository to support search augmentation generation (RAG) of an AI model. The embedding repository stores high-dimensional vector representations of text, code, or audio data, thereby enabling an engine providing follow-up questions in a conversational AI service to perform high-speed search based on semantic similarity. This can serve as a technical means to suppress hallucinations in the model and increase the accuracy of the output.

[0089] Specifically, the memory (132) of the server computing system (130) may include a structured Knowledge Base Layer to physically support Search Augmentation Generation (RAG). The Knowledge Base Layer may include an Embedding Repository that stores high-dimensional vector representations of unstructured text, code, or multimodal data, and a Policy Document Repository that stores behavioral constraints and business rules of agents.

[0090] At this time, the follow-up question generation engine in the artificial intelligence service can be configured to vectorize the query received from the user computing device (110) and query the embedding repository to retrieve context information with high semantic similarity in real time. This structure can technically suppress the hallucination phenomenon of the artificial intelligence model by allowing the agent to refer to external verified knowledge rather than relying only on intrinsic parameters.

[0091] Additionally, the user computing device (110) may include a trigger event detector that provides an interface for interaction with an agent and initiates the operation of the agent. The trigger event detector can detect not only user input but also the arrival of a specific time, a change in the state of an external system, etc., and transmit a processing request to a server computing system (130).

[0092] Additionally, the model trainer (160) of the training computing system (150) may include a Supervision Signal Engine. The Supervision Signal Engine may compare the output generated by the agent (e.g., Raw Output) with a verified result (e.g., Grounded Output) obtained through an external tool (e.g., search engine, API) to calculate a difference value, and execute a reinforcement learning process to update the Reward Model or fine-tune the agent model based on this.

[0093] As such, in one embodiment, the system (1000) of the present disclosure may be implemented not as a simple set of software algorithms, but as a technical system in which specialized hardware accelerators, vectorized data storage, and physical engines controlling them are organically combined.

[0094] FIG. 3a illustrates a block diagram example of a computing device (100) that may be included in a user computing device (110), a server computing system (130), and a training computing system (150), as an embodiment of a computing system (1000) in which the present disclosure can be implemented.

[0095] The computing device (100) may include a plurality of applications (e.g., Application 1 to Application N), and each application may include a machine learning library and a model execution environment for performing follow-up question functions in a machine learning-based artificial intelligence service.

[0096] For example, applications may include image processing (e.g., detection, classification, and / or segmentation, etc.) applications, text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications and / or chat-bot applications, etc.

[0097] Additionally, for example, the computing device (100) may include an application that provides a relevant service to a user, including a model that provides follow-up questions in a conversational artificial intelligence service, that is, a specialized application for generating follow-up questions in an artificial intelligence service.

[0098] In one embodiment, the computing device (100) may include a model trainer (160) for pre-training or retraining a model that provides follow-up questions in a conversational artificial intelligence service.

[0099] The model trainer (160) learns or updates parameters based on input text data and saves the learned model so that it can be used in the follow-up question generation task in the artificial intelligence service.

[0100] Each application included in the computing device (100) can communicate with sensors, device status managers, or external input devices within the device through an API (Application Programming Interface).

[0101] In one embodiment, each application can perform the role of an interface with device components, such as receiving sensor data or status data through a public or private API, or transmitting prediction results to an output device.

[0102] FIG. 3b is a block diagram illustrating the internal architecture and data processing pipeline of an artificial intelligence model according to one embodiment of the present disclosure.

[0103] Referring to FIG. 3b, the computing device may have a pipeline structure that receives input data (202) to provide follow-up questions in a conversational artificial intelligence service and generates output data (212) through a series of transformation processes. This process is performed through a preprocessing module (204), an encoder / embedding model (206), a neural network layer (208), and a decoder / generation head (210).

[0104] First, the preprocessing module (204) receives input data (202) (e.g., text prompt, image, or multimodal signal) from a user or system. The preprocessing module (204) performs tokenization and normalization on the input data to generate a sequence of tokens, which are the smallest units that the model can process.

[0105] Next, the encoder / embedding model (206) receives the generated token as input and converts it into a vector / embedding mapped to a number in a high-dimensional vector space. At this stage, the discrete information of the input data is converted into a continuous numeric matrix, making it a form that can be computed by the machine learning model.

[0106] Next, the neural network layer (208) receives the vector / embedding and performs deep computation. The neural network layer (208) may have a structure in which a plurality of sub-layers (e.g., Layer 1 to Layer N) are stacked. Each layer abstracts and refines input features through an attention mechanism or convolution operation, etc.

[0107] In particular, the final output of the neural network layer (208) is defined as a latent representation. This latent representation has a structure different from the original input data (202) and may correspond to an intermediate representation in which the semantic features of the data are highly compressed and abstracted. This implies that it is not a simple transmission of data, but a technical data structure that is valid only within the system.

[0108] Finally, the decoder / generation head (210) receives the potential representation as a conditioning input. The decoder / generation head (210) interprets the compressed potential representation and reconstructs or generates output data (212) in a form recognizable by the user (e.g., natural language text, image pixels, control codes, etc.).

[0109] Such a stepwise data transformation structure (token vector latent representation output) can clearly demonstrate operation as a concrete device that technically processes input data to generate useful information, rather than a simple sequence of operations.

[0110]

[0111] FIG. 4a illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system (1000) that performs a follow-up question generation method in an artificial intelligence service according to an embodiment of the present disclosure.

[0112] The computing device (200) may include a plurality of applications (e.g., Application 1 to Application N), and each application may communicate with a central intelligence layer.

[0113] For example, applications may include image processing applications, text messaging applications, email applications, dictation applications, virtual keyboard applications and / or browser applications, etc. In one embodiment, each application may communicate with a central intelligence layer (and models stored therein) using an API (e.g., an API common across all applications).

[0114] In addition, in one embodiment of the present disclosure, the application may include a follow-up question generation application, an energy management application, a logging and analysis application, etc., in an artificial intelligence service.

[0115] Each application can interface with shared models within the central intelligence layer through APIs (e.g., common APIs).

[0116] The central intelligence layer includes a machine learning model that performs follow-up question generation in one or more artificial intelligence services, and can share this among multiple applications or provide it independently to each.

[0117] In one embodiment, the central intelligence layer may be integrated as part of the operating system or implemented as a separate logical layer, and may perform the role of passing input text data to the corresponding model to return a prediction result.

[0118] In addition, the central intelligence layer can communicate with the central device data layer. The central device data layer integrates and stores sensor data, device status information, and external environment information stored within computing devices, and can provide this as input data necessary for generating follow-up questions in artificial intelligence services. Each device component (e.g., sensors, state managers, etc.) can communicate with the corresponding data layer through private APIs, etc.

[0119] The technology described in this specification may be composed of a single or multiple computing devices, and a model for providing follow-up questions in a conversational artificial intelligence service may be executed sequentially or in parallel on one component or multiple distributed components.

[0120] Data repositories, predictive models, and applications can operate in a distributed configuration locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0121] FIG. 4b is a block diagram illustrating the data flow and system interaction of a follow-up question generation service application process in an artificial intelligence service according to one embodiment of the present disclosure.

[0122] Referring to FIG. 4b, the computing system may be configured as an organic data pipeline between a trigger event detector (310), an inference engine (330), and a mobile device screen (350).

[0123] First, the trigger event detector (310) is configured to monitor and detect a signal initiating the operation of the system. The trigger event detector (310) receives at least one of (i) a time event indicating the arrival of a specific point in time, (ii) a user action resulting from a user's physical input, or (iii) a system state indicating a change in internal system data, and transmits an activation signal to the inference engine (330). This means that the service can be actively initiated depending on the situation without an explicit request from the user.

[0124] The reasoning engine (330) performs a multi-stage operation process that transforms raw data into a final result in response to the activation signal. This process is implemented as a series of logically connected prompt chains.

[0125] For example, the first prompt, contextualization (332), allows the inference engine (330) to receive unstructured raw data (e.g., user logs, channel metadata), analyze and refine it, and generate compressed summary information.

[0126] And the second prompt, Content Generation (334), can generate multiple candidate results that match the user's intent or situation using a generative AI model based on the summary information generated above.

[0127] In addition, the third prompt, Verification (336), performs filtering and verification by comparing the generated candidate results with a predefined policy (e.g., safety guidelines, format rules) to derive a reliable final result.

[0128] Finally, the final result generated by the inference engine (330) can be transmitted to and implemented on the mobile device screen (350) through an auto pre-filling (354) operation. Specifically, the system changes the state of the interface by directly writing the final result to a memory address of a specific target field (352) (e.g., text input field, setting value) within the mobile device screen (350). Here, the mobile device screen (350) may be an example of a user computing device (110).

[0129] Such a configuration can go beyond the simple display of information and provide specific technical means for data generated by an external trigger to physically control and complete the input interface of the user terminal.

[0130] FIG. 5 illustrates the architecture of a multi-agent system (Universal Dynamic Multi-Agent System, 500) according to one embodiment of the present disclosure.

[0131] Referring to FIG. 5, the system (500) may be configured around an orchestration engine (510) that determines and executes an optimal agent collaboration structure in real time according to the nature of the user's request or task. The system (500) may operate in an organically combined manner, including an input analyzer (505), an agent pool (520), a shared memory fabric (530), and a tool execution interface (540).

[0132] The input analyzer (505) evaluates the complexity of the user query and the required domain expertise. Based on this evaluation result, the orchestration engine (510) dynamically configures the optimal agent collaboration topology for task resolution.

[0133] For example, in the case of a simple question, the engine (510) activates a single agent mode.

[0134] If a complex plan is required, the engine (510) configures a Supervisor pattern and instantiates one supervisor agent to control subordinate agents.

[0135] When high accuracy is required, the engine (510) configures a debate pattern and sets a control path so that multiple agents perform mutual criticism.

[0136] The agent pool (520) is a repository of template agents that have prompts and tool sets specialized for specific functions (e.g., web search, code generation, data analysis). The orchestration engine (510) can select and activate the necessary agents at runtime according to the determined topology.

[0137] The shared memory fabric (530) is a data pipeline that synchronizes the state and context between multiple collaborating agents. By mediating the short-term memory of individual agents and the knowledge base (RAG) of the entire system, it can ensure that the output of agent A is transmitted to agent B as input without loss.

[0138] Each agent communicates with an external API (search engine, calculator, AWS, etc.) through a tool execution interface (540). The execution result generated at this time is fed back to the orchestration engine (510), and the engine can determine the consistency of the result and perform self-correction logic to instruct the agent to rework or proceed to the next step.

[0139] This structure enables the implementation of an adaptive artificial intelligence system that flexibly changes the system's processing structure according to the nature of the input problem, rather than a fixed (static) algorithm.

[0140] Meanwhile, the orchestration engine (510) analyzes the characteristics of the task received from the input analyzer (505) (e.g., creativity, logic, whether coding is required) and selects specialized agents waiting in the agent pool (520) to form a dynamic collaboration topology. The topology can be reconfigured into various operation modes as follows depending on the type of task.

[0141] For tasks such as a single-flow creation or report writing, the orchestration engine (510) connects multiple agents in series. For example, it forms a pipeline in the order of user agent, writer agent, and style agent to control the output of the previous stage to be passed as input to the next stage.

[0142] In cases where complex sub-tasks are mixed, the engine (510) forms a centralized star topology in which one agent is designated as a supervisor and the remaining agents (e.g., research agent, math agent) are assigned as workers. The supervisor agent distributes tasks to sub-agents and aggregates the results.

[0143] In cases of high complexity, such as large-scale project management, the engine (510) establishes a command system by forming a tree structure topology in which a meta-agent is placed as the upper manager and multiple specialized agent groups are placed below it.

[0144] In cases where the correct answer is unclear or high reliability is required (e.g., social issue analysis), the engine (510) constructs a competitive topology in which multiple agents present different perspectives on the same topic and perform mutual criticism and voting to derive the optimal conclusion.

[0145] When parallel processing is required, the engine (510) forms a parallel processing structure that places multiple agents in multiple layers to perform tasks simultaneously and finally integrates the results through an aggregator agent.

[0146] Additionally, the agent pool (520) includes various template agents that can be deployed into the topology. Each agent is instantiated to perform the following core methodologies under the control of the orchestration engine (510).

[0147] For example, at least one agent can execute a loop that repeats reasoning and action to perform reasoning and action (ReAct Paradigm) that refines answers based on results from external tools (e.g., Google Search).

[0148] In addition, at least one agent can perform code-based behavior (CodeAct Paradigm) that involves complex calculations or data analysis by generating and executing executable code (e.g., Python) instead of natural language.

[0149] In addition, at least one agent can perform self-reflection to improve the quality of the output by carrying out a metacognitive process of self-evaluating (Critique) and modifying the generated output.

[0150] Furthermore, the system provides a shared memory fabric (530) to enable individual agents to cooperate organically without disconnection. This prevents context loss by managing the conversation history (Short-Term Memory) between agents and the external knowledge base (RAG) in an integrated manner. Additionally, the tool execution interface (540) securely connects the agents with external APIs, cloud services, and databases through protocols such as Multi-Channel Processing (MCP).

[0151] In this way, the system (500) of the present disclosure implements an adaptive artificial intelligence platform that is not a fixed single model, but rather dynamically changes the processing structure and behavior of the system according to the nature of the input problem.

[0152] Specifically, one embodiment of the present disclosure may be implemented as an Agentic RAG (Retrieval-Augmented Generation) system that supports advanced question-answering by including agent functions. This system is a search-based generation system that retrieves information from external data sources and generates answers based thereon.

[0153] The system's data processing pipeline can consist of a data extraction phase and an Agentic RAG pipeline phase. In the data extraction phase, content in various formats, such as text and images, is collected from designated websites. Subsequently, text and metadata are extracted from the collected content, the text is chunked into small units, and each chunk is vectorized using an embedding model and stored in a vector database.

[0154] The Agentic RAG pipeline can be divided into search and generation phases. In the search phase, when a user query is input, query rewriting and embedding are performed, and highly relevant documents are identified through similarity searches in a vector database. Subsequently, the retrieved results can be ranked based on relevance to construct context. In the generation phase, the user query and the retrieved context are combined to construct a prompt, which is then input into a large language model to generate the final response. During this process, the agent can respond to complex queries by utilizing agentic elements such as memory storage, calling external tools, and planning.

[0155] An AI agent according to one embodiment of the present disclosure can support complex decision-making and task execution through a hierarchical memory structure similar to human memory. The agent's memory can be broadly divided into short-term memory and long-term memory.

[0156] Short-term memory is a temporary memory space focused on the currently ongoing workflow and may include working memory, which manages workflow-specific reasoning and task flows, and cache memory, which provides quick access to frequently used data or results.

[0157] Long-term memory is a memory space based on knowledge and experience that is continuously preserved, and may include episodic memory, which records events or incidents manually saved in a specific workflow; semantic memory, which stores conceptual or factual knowledge; and procedural memory, which stores knowledge of how to perform specific tasks or procedural knowledge.

[0158] This memory structure can be integrated with the language model framework through a central memory controller. Additionally, it can leverage external knowledge or support real-time integration by connecting with external vector databases, semantic databases, or third-party APIs via the MCP server. Through this, the agent can generate user-customized responses that comprehensively reflect past experiences and current context.

[0159] One embodiment of the present disclosure may include various agentic workflows to solve complex problems. These workflows may be designed to suit specific business purposes.

[0160] For example, the system may include a Plan and Execute workflow. In this workflow, a Planner breaks down a single top-level task into multiple sub-tasks, specialized agents process each sub-task, and then integrates the results. This can be utilized for business process automation or data pipeline orchestration.

[0161] As another example, the system may include an Orchestrator-Worker workflow. In this structure, a central orchestrator language model breaks down tasks, distributes them to multiple worker language models for processing, and then integrates the results. This can be used in the implementation of Agentic RAGs or coding agents.

[0162] As another example, the system may include a routing workflow. This workflow is structured to analyze input tasks, classify them into the most suitable ones among several predefined subtasks, and forward them to a specialized language model or path capable of handling the task. This can be applied to customer support agents or multi-agent discussion systems.

[0163] One embodiment of the present disclosure may include a protocol for efficient and secure communication between a plurality of agents or between an agent and an external tool.

[0164] In one embodiment, an Agent2Agent (A2A) protocol may be used for communication between agents. The A2A protocol can enhance security by enabling each agent to communicate without directly sharing their internal data. Through this protocol, multiple agents can share tasks and negotiate, and each agent can operate independently using its own language model, framework, and database.

[0165] In another embodiment, the Multi-Channel Processing (MCP) protocol may be used for communication between an agent and an external function server. MCP has a structure that separates each external function, such as file access, search, and cloud API calls, into a separate server for communication. For example, one agent may communicate with a local file system or a search engine via the MCP protocol, while another agent may communicate with a cloud provider such as AWS or a communication tool such as Slack via the same protocol.

[0166]

[0167] Hereinafter, with reference to FIGS. 6 to 11, a method and system for providing follow-up questions in an interactive artificial intelligence service according to one embodiment of the present disclosure will be described in detail.

[0168] FIG. 6 is a diagram showing the entire pipeline of a follow-up question providing method according to one embodiment of the present disclosure. Referring to FIG. 6, the follow-up question providing method of the present disclosure sequentially includes a first step (S100: receiving conversation history and CoT-based context inference), a second step (S200: generating follow-up question candidates, ranking, configuring candidate packs, and binding metadata), a third step (S300: building a client-side prefix trie index, real-time keystroke matching, and uncertainty-based conditional regeneration), and a fourth step (S400: generating answers based on KV cache restoration and dynamic knowledge assembly).

[0169] In the entire pipeline described above, the first step (S100) and the second step (S200) are performed by the processor (131) of the server computing system (130), the third step (S300) is performed by the processor (111) of the user computing device (110), and the fourth step (S400) is performed by the processor (131) of the server computing system (130). In part of the third step (uncertainty-based server regeneration trigger), the interaction between the user computing device (110) and the server computing system (130) is performed via the network (170). Additionally, the machine learning models (FQ-Generator, FQ-Ranker) used in the first and second steps are pre-trained by the model trainer (160) of the training computing system (150).

[0170] Below, steps 1 through 4 will be explained in detail in order.

[0171] Hereinafter, with reference to FIGS. 6 and 7, the first step of the method for providing a follow-up question according to one embodiment of the present disclosure, namely the reception of conversation history and the CoT (Chain-of-Thought)-based context inference step, will be described in detail.

[0172] The first step (S100) is a step in which a processor (131) of a server computing system (130) systematically infers contextual information necessary for generating follow-up questions based on a conversation history received from a user computing device (110) via a network (170). Specifically, the processor (131) sequentially generates user intent, conversation goal, and related knowledge by performing a Chain-of-Thought (CoT) based inference process using the conversation history as input.

[0173] The contextual information (user intent, conversation purpose, relevant knowledge) generated in the first stage is not only provided as input for the subsequent second stage (candidate generation and ranking), but is also utilized as basic data for metadata binding for KV cache restoration in the fourth stage (answer generation). Accordingly, the inference result of the first stage is maintained in an intermediate state in the memory (132) of the server computing system (130).

[0174] Referring to FIG. 7, the processor (131) first receives a conversation history from a user computing device (110) via a network (170) (S110).

[0175] In this disclosure, conversation history refers to a chronological set of question-answer pairs exchanged between a user and a system in the current conversation session. Specifically, a conversation sample at the i-th turn It can be defined as follows.

[0176] [Mathematical Formula 1]

[0177]

[0178] Here, is the user query at the i-th turn, and Is It is the system's answer to, is a conversation history from the 1st turn to the (i-1)th turn, and is defined as follows.

[0179] [Mathematical Formula 2]

[0180]

[0181] In one embodiment, the processor (131) can convert the received conversation history into a structured input format. For example, the processor (131) can normalize the conversation history into the following structured prompt format.

[0182] [DIALOGUE HISTORY]Turn 1 - User: { }Turn 1 - System: { }Turn 2 - User: { }Turn 2 - System: { }...Turn (i-1) - User: { }Turn (i-1) - System: { }[CURRENT TURN]User: { }System: { }

[0183] The above structured prompt is intended to increase the accuracy of turn-by-turn intent extraction in the subsequent CoT inference process by explicitly distinguishing the speaker (user / system) and the order of utterance for each turn. In one embodiment, if the length of the conversation history exceeds a predetermined threshold (e.g., maximum number of tokens), the processor (131) may truncate the history in a sliding window manner that includes only the most recent N turns. In this case, the value of N may be determined based on the computing resources of the system and the context window size of the model.

[0184] In another embodiment, when the conversation history is single-turn,

[0185] in other words This empty set ( Even in the case of ), the processor (131) queries the current turn and response Subsequent CoT inference can be performed using only this. This means that the method of the present disclosure is applicable not only to multi-turn conversations but also to single-turn conversations.

[0186] The processor (131) is a structured conversation history Using as input, CoT-based context inference is performed through a pre-trained follow-up question generation model (hereinafter, FQ-Generator) (S120). Here, the FQ-Generator is an example of a machine learning model (140) stored in the memory (132) of a server computing system (130). Referring to FIG. 7, the CoT-based context inference process sequentially includes the following three sub-steps.

[0187] As a first sub-step of CoT inference, the processor (131) [requires] the conversation history Based on this, the user's conversational intent I is extracted for each turn (S121).

[0188] [Mathematical Formula 3]

[0189]

[0190] Here, M represents the FQ-Generator model, and I is the set of extracted user intentions.

[0191] User Intention I represents the direction of information search implied by the user query in each turn, and may include, for example, "intention to verify the definition of a specific concept," "intention to explore details of a previous answer," "intention to expand to related topics," etc.

[0192] To explain with a specific example, if the conversation history is as follows:

[0193] Turn 1 - User: "Did Woodrow Wilson go to college?"

[0194] Turn 1 - System: "Wilson worked as a lecturer at Cornell University in 1886–87."

[0195] Turn 2 - User: "What did Woodrow Wilson teach at Cornell University?"

[0196] Turn 2 - System: "Taught Ancient Greek and Roman History."

[0197] Turn 3 - User: "What did Woodrow Wilson teach at Brin?"

[0198] Turn 3 - System: "This is the history of ancient Greece and Rome."

[0199] Turn 4 - User: "Where did Woodrow Wilson teach after Brin?"

[0200] Turn 4 - System: "This is Wesleyan University."

[0201] In this case, the turn-by-turn user intent I extracted by the processor (131) can capture the aspect in which the user's direction of information search gradually becomes more specific as the conversation progresses, such as, for example, "Turn 1: Intention to verify Wilson's educational background", "Turn 2-3: Intention to specify the field of education", "Turn 4: Intention to track teaching career history".

[0202] As a second sub-step of CoT inference, the processor (131) [retrieves] the user intent I extracted in the first sub-step and the conversation history By combining them, the comprehensive conversational goal G of the current conversation is derived (S122).

[0203] [Mathematical Formula 4]

[0204]

[0205] Here, conversational objective G represents a higher-level information search goal derived by synthesizing the intentions of individual turns. That is, conversational objective G is not a simple set of intentions I for each turn, but rather the result of inferring the information search goal that the user ultimately intends to achieve by analyzing the relationships between the intentions of the turns.

[0206] In the previous example, conversational objective G can be derived as a higher-level goal that encompasses the intentions of individual turns, such as, for example, "a systematic understanding of Woodrow Wilson's entire teaching career."

[0207] The derivation of conversational purpose G performs the function of ensuring that subsequent questions generated in the subsequent second stage do not merely remain a contextual continuation of the previous conversation, but are guided in a direction to achieve the user's ultimate purpose of information search. This is the core technical concept of the present disclosure, in contrast to conventional technology which has focused primarily on contextual relevancy.

[0208] As a third sub-step of CoT inference, the processor (131) [reviews] the conversation history By combining user intent I and conversational purpose G, relevant knowledge K that can contribute to the achievement of the conversational purpose is generated (S123).

[0209] [Mathematical Formula 5]

[0210]

[0211] Here, relevant knowledge K is generated based on the parametric knowledge of the FQ-Generator and includes background information, relevant facts, or relevant concepts that the user must be aware of or address in follow-up questions to achieve conversational purpose G.

[0212] Specifically, relevant knowledge K may consist of a set of multiple knowledge items, each knowledge item being a unit of information that complements different aspects of conversational purpose G.

[0213] In the preceding example, the relevant knowledge K may include, for example: "Wilson's tenure at Wesleyan University: 1888–1890", "Post-Wesleyan teaching career: Professorship at Princeton University", "The influence of Wilson's teaching experience on his later political career", or "Wilson's academic writings: Congressional Government", and each knowledge item corresponds to a different sub-aspect of the conversational objective ("systematic understanding of Wilson's entire teaching career").

[0214] The generation of such relevant knowledge K provides a foundation that enables the creation of follow-up question candidates in the subsequent second stage, allowing for the generation of follow-up questions that encompass areas of relevant information the user is unaware of, rather than simply questions that fit the conversational flow.

[0215] In one embodiment, when generating relevant knowledge K, the processor (131) can enhance the knowledge by linking an Embedding Repository or an external knowledge source provided in the memory (132) of the server computing system (130) in addition to the parametric knowledge of the FQ-Generator.

[0216] The processor (131) stores the user intent I, conversation purpose G, and related knowledge K generated through the CoT inference process (S120) as a session context in the memory (132) of the server computing system (130) (S130).

[0217] Specifically, the processor (131) creates a context data structure associated with a session identifier (session ID) corresponding to the current conversation session in memory (132) and loads the following information into the structure.

[0218] Field Description Data Type session_id Unique identifier of the current conversation session String dialogue_history Structured conversation history Sequence<(q, a)>user_intents Extracted user intents per turn ISequenceconversational_goal Derived conversational goal GTextknowledge_items Generated knowledge items KSequencekv_cache_stateCoT Pointer to the KV cache state generated during inference Pointertimestamp Timestamp at the time of creation DateTime

[0219] In particular, the kv_cache_state field stores a pointer that references the state of the Key-Value cache generated within the FQ-Generator during the process in which the processor (131) performs CoT inference. This is intended to shorten the Time-to-First-Token (TTFT) of answer generation by directly restoring the stored KV cache without re-encoding the entire conversation history when the user selects a follow-up question in the subsequent fourth step. A detailed explanation of the KV cache restoration is provided later in the fourth step. As such, the intermediate state generated in the first step is maintained in memory (132), thereby eliminating the need to repeatedly perform the same inference in subsequent steps, and consequently improving the computational efficiency of the entire pipeline. In one embodiment, the CoT inference process of the first step may be optionally simplified according to the requirements of the service operating environment.

[0220] Specifically, the model trainer (160) of the training computing system (150) trains the FQ-Generator through knowledge distillation from a teacher model. The teacher model is a model trained to explicitly generate user intent I, conversation purpose G, and related knowledge K from conversation history. The processor (151) of the training computing system (150) uses the output of the teacher model as a supervised signal to perform training so that the student model, the FQ-Generator, internalizes the reasoning ability of the teacher model.

[0221] Accordingly, the FQ-Generator distributed to the server computing system (130) after the completion of learning omits the explicit extraction step (S121) of user intent I during inference time, and the conversation history You can directly generate conversational objectives G and related knowledge K from it.

[0222] [Mathematical Formula 6]

[0223]

[0224] Here, represents the student model (FQ-Generator), and G' and K' represent conversational purpose and related knowledge generated directly by omitting the intent extraction step.

[0225] By omitting this optional step, the number of generated tokens during inference is reduced, thereby shortening the processing time of the first step and reducing the computational load and memory (132) usage of the processor (131) of the server computing system (130).

[0226] In another embodiment, the processor (131) may additionally reflect user personalization information during the CoT inference of the first stage.

[0227] Specifically, the processor (131) may include the user's past conversation history, preference information (e.g., areas of interest, preferred difficulty, purpose of information search), or user profile data (e.g., field of expertise, usage patterns) in the input of CoT inference along with the conversation history.

[0228] In this case, the input for CoT inference is Combining user profile P with ( It takes the form of , P), and the conversational purpose G and related knowledge K generated based on this reflect information customized to the characteristics of the user.

[0229] Referring to FIG. 6, the output of the first step (S100) is connected to subsequent steps as follows.

[0230] First, the conversation purpose G and related knowledge K are provided as direct inputs to the FQ-Generator for generating follow-up question candidates in the second step (S200).

[0231] Second, the intermediate state (I, G, K) and KV cache pointer stored in the session context data structure (Table 2) are used as the basis data for each candidate's metadata binding in the second stage.

[0232] Third, the KV cache state (kv_cache_state) is utilized for KV cache restoration (Prompt Stitching) during answer generation in the fourth step, enabling rapid answer generation without re-encoding the entire conversation history.

[0233] Hereinafter, with reference to FIGS. 8 and 9, the steps of generating follow-up question candidates, ranking, configuring a candidate pack, and binding metadata, which are the second steps of the follow-up question provision method according to one embodiment of the present disclosure, will be described in detail.

[0234] The second step (S200) is a step in which the processor (131) of the server computing system (130) generates multiple follow-up question candidates based on the context information (conversation purpose G, related knowledge K) inferred in the first step (S100), ranks the generated candidates, and composes the ranking results into a structured data packet (hereinafter referred to as a Candidate Pack) for transmission to the user computing device (110). In addition, in the second step, for the KV cache restoration in the fourth step (S400), a metadata structure binding the intermediate state of CoT inference for each candidate is generated and loaded into the memory (132) of the server computing system (130).

[0235] Referring to FIG. 8, the second step (S200) includes the following sub-steps: (i) generation of follow-up question candidates (S210), (ii) listwise ranking (S220), (iii) metadata binding and KV cache pointer linking (S230), and (iv) configuration and transmission of candidate packs (S240).

[0236] The second stage corresponds to the technical core stage of the present disclosure in that it simultaneously considers the quality of follow-up questions and the serving efficiency of the system. Specifically, in the second stage, four technical functions are organically combined: (a) goal-oriented candidate generation utilizing CoT inference results, (b) ranking that integrally considers utility and estimated serving costs, (c) memory management that binds candidate-specific metadata together with KV cache pointers, and (d) generation of structured data packets that support the construction of client-side indexes.

[0237] Specifically, the processor (131) first, a pre-trained follow-up question generator model (FQ-Generator, Multiple follow-up question candidates are generated using ) (S210).

[0238] FQ-Generator is the conversation history stored in the session context data structure (see Table 2) of memory (132) in the first stage Receives as input, generates conversation objectives G and related knowledge K through the CoT process, and then, based on these, a candidate set containing m follow-up question candidates Prints.

[0239] [Mathematical Formula 7]

[0240]

[0241] Here, m is the number of candidates generated (e.g., 5 to 10), and each (j = 1, 2, 쪋, m) is the text of individual follow-up question candidates.

[0242] Continuing with the Woodrow Wilson conversation example from the first stage described above, the candidate set that the FQ-Generator can generate based on the conversation objective G ("systematic understanding of Woodrow Wilson's entire teaching career") and related knowledge K is, for example, as follows.

[0243] Candidate ID Follow-up Question Candidate Text ( ) How long did Woodrow Wilson work at Wesleyan University? Where did Woodrow Wilson continue his teaching career after Wesleyan University? What were Wilson's major academic achievements during his tenure at Wesleyan University? How did Wilson's teaching experience influence his later political career? How did your experience at Cornell University influence your teaching at Brin and Wesleyan? Did Woodrow Wilson write any major works while he was working at the university? What subjects did Wilson teach at Wesleyan? What were the characteristics of Wilson's teaching career compared to other contemporary scholars? What were Wilson's research interests during his time at Wesleyan University? "What role did Wilson's professorship play in his inauguration as President of Princeton University?"

[0244] As can be seen in the example above, the generated candidates are not merely a contextual continuation of the query from the previous turn ("Where did you teach after Brin?"), but include questions encompassing various aspects (tenure, subsequent career, academic achievements, political influence, etc.) that correspond to the conversational objective G ("systematic understanding of the entire teaching career"). It is noteworthy in this disclosure that the FQ-Generator M_G does not output only the candidate set F_c during inference, but also outputs the conversational objective G and related knowledge K, which are intermediate results of the CoT process. These intermediate results are critically utilized in the metadata binding (S230) described below and the KV cache restoration of the fourth step. Specifically, the FQ-Generator A set of candidates During the process of generating, the conversation history inside the FQ-Generator , conversation purpose G, relevant knowledge K, and candidate questions being generated A KV cache corresponding to the tokens is cumulatively formed. The processor (131) can link the state of this KV cache at a specific point in time as a pointer to a metadata structure described later. Details regarding this are explained in Section 5 below.

[0245] And the processor (131) is the candidate set output from the candidate generation step (S210). ...is input into a pre-trained ranking model (FQ-Ranker, M_R) to perform list-wise ranking to determine the rank for each candidate (S220). The FQ-Ranker M_R is conversation history and candidate set Given a set of candidates Rankings are determined by considering all candidates included in the list simultaneously. This contrasts with the point-wise method, which evaluates individual candidates independently, or the pair-wise method, which compares pairs of candidates. Through the list-wise method, the diversity and complementarity of the Top-k candidate combinations ultimately provided to the user are optimized.

[0246] In one embodiment, FQ-Ranker Silver, each candidate Utility score U( It can calculate ). The usefulness score U is a numerical representation of the extent to which the candidate question contributes to the user's information search objective, and is determined by the implicit evaluation criteria learned by the FQ-Ranker through the ranking information of the teacher model during the learning process.

[0247] Specifically, the usefulness score U is a score that comprehensively reflects (i) relevance: contextual consistency with the conversation history, (ii) informativeness: the extent to which it provides the user with new information or perspectives, (iii) truthfulness: whether it is based on factually valid premises, (iv) clarity: the clarity and naturalness of the questions, and (v) coherence: logical connectivity with the conversation flow. These evaluation criteria are based on Grice's Maxims of Conversation, where relevance corresponds to the Maxim of Relation, informativeness to the Maxim of Quantity, truthfulness to the Maxim of Quality, and clarity and coherence to the Maxim of Manner.

[0248] In another embodiment, the processor (131) may calculate an expected serving cost C for each candidate question in addition to the utility score U, and perform ranking based on an integrated score S that combines the utility score and the expected serving cost (hereinafter referred to as cost-aware ranking).

[0249] [Mathematical Formula 8]

[0250]

[0251] Here, is a hyperparameter that adjusts the relative weight between utility and serving cost (0 < < 1)

[0252] Estimated serving cost C( ) is, the user's corresponding candidate question It is a value pre-estimated as the size of computing resources required to generate a response when selected. Specifically, the estimated serving cost C can be calculated as a weighted sum of the following components.

[0253] [Mathematical Formula 9]

[0254]

[0255] Here, each component is defined as follows.

[0256] Component Symbol Description Cache Miss Probability The probability that an existing answer to a similar query in the semantic cache will not be hit when generating an answer for the corresponding candidate. Cost increases because a cache miss requires full inference by the LLM. Expected prefill length The number of tokens for prefill operations that the LLM must perform when generating an answer for the corresponding candidate. As the prefill length increases, GPU computation and memory usage increase. This can be estimated based on the token length of the corresponding candidate question, the size of the associated knowledge K, and the length of the conversation history. RAG Search Cost The estimated cost of searching for external documents in the RAG (Retrieval-Augmented Generation) pipeline when generating an answer for the candidate. Weighted proportional to the extent that the candidate's topic requires external search. Weights (Σw_i = 1) representing the relative importance of each cost component

[0257] Through cost-aware ranking, among two candidates of equal utility, the candidate with the lower estimated serving cost—that is, the candidate with a higher cache hit probability upon answer generation, a shorter prefill length, or one that does not require external search—is ranked higher. The technical effect of this cost-aware ranking is to reduce the average response latency and server-side GPU computational load of the entire conversational service, given that the follow-up questions recommended to the user are not merely "good questions" but also "questions that the system can answer efficiently." This performs a technical function that optimizes the computing resource management of the conversational AI system, going beyond the information provision function of recommending follow-up questions. In one embodiment, hyperparameters It can be dynamically adjusted based on the system's current load status (e.g., GPU utilization, number of concurrent users, cache hit rate). For example, during periods of high server load Increase the value to give greater weight to serving costs, and during times when server load is low By reducing values, usefulness-based ranking can be performed. Through this dynamic adjustment, it is possible to maintain the user experience while adaptively responding to the system's load state.

[0258] An example of the ranking results for the candidates in Table 3 is as follows.

[0259] Ranking Candidate ID Usability U Serving Cost C Integrated Score S ( =0.3)1 4.70.34.612 4.80.84.563 4.50.24.444 4.60.74.395 4.50.64.32...............

[0260] In the above example, ("The Impact of Wilson's Teaching Experience on His Later Political Career") has the highest utility score (4.8), but ranks second in the combined score because the external search cost for generating answers to it is relatively high (C = 0.8). On the other hand, ("Where did you continue your teaching career after Wesleyan University?") has a slightly lower utility score (4.7), but is ranked 1st in the combined score because it has a high cash hit probability and a short prefill length, resulting in a low serving cost (0.3). In an embodiment where cost-aware ranking is not applied, ranking is performed based solely on the utility score U, in this case It is ranked as 1st. Application of cost-aware ranking and hyperparameters The value of can be set according to the system operation policy. In one embodiment, the processor (131) performs each candidate as part of the ranking step (S220) or prior to ranking. The answerability for can be determined in advance.

[0261] Specifically, the processor (131) can search for supporting evidence from a RAG index or an external knowledge source for each candidate question and, based on the existence and sufficiency of the supporting evidence, assign an answerability flag to the candidate.

[0262] Response potential flags can be divided into three levels, for example, "HIGH" (sufficient supporting evidence), "MEDIUM" (partial supporting evidence), and "LOW" (lack of supporting evidence, production is possible but there is a risk of hallucination).

[0263] The above response probability information can be reflected in the calculation of the aforementioned estimated serving cost C. Specifically, for candidates with a response probability of "LOW," it can be estimated that the serving cost increases because additional external search or verification steps are required during response generation. Accordingly, the pre-determination of response probability is not merely intended to display reliability information to the user, but performs a technical function to improve the accuracy of the estimated serving cost calculation and, furthermore, to optimize the management of the system's computing resources.

[0264] As one of the second stage alcohol configurations, the processor (131) comprises each ranked candidate Regarding this, a metadata structure (hereinafter referred to as "Candidate Metadata") that binds the intermediate state of the CoT inference derived during the generation process of the candidate is created and loaded into the memory (132) of the server computing system (130) (S230).

[0265] Referring to Fig. 9, each candidate The candidate-specific metadata structure corresponding to is defined as follows.

[0266] Field Description Data Type question_id Unique identifier of the candidate question String question_text Text of the candidate question ( )Texttoken_sequence Sequence of tokens obtained by splitting candidate questions by the tokenizer Sequenceconversational_goal Conversational goal derived in Step 1 GTextknowledge_items Related knowledge generated in Step 1 KSequenceuser_intents User intents extracted in Step 1 (Optional) Sequencerank_score Ranking score (U or S) Floatcategory_id Category of candidate question (e.g., Extended, Deepened, Confirmed) Enumanswerability_flag Answerability flag (Optional) Enum{HIGH, MEDIUM, LOW} kv_cache_pointer Memory pointer to the KV cache state at the time the candidate was created Pointerembedding_vector Embedding vector of the candidate question Float[d]ttl Validity period of the metadata (Time-to-Live) Integer (seconds)

[0267] In Table 6, the field of particular note is kv_cache_pointer. Below, with reference to Figures 9 and 11, the technical significance and connection method of the KV cache pointer are explained in detail. FQ-Generator Conversation history Perform the CoT process from the conversation objective G, relevant knowledge K, and candidate questions In the process of sequentially generating, a Key-Value cache corresponding to each token location is cumulatively formed in the neural network layer (see FIG. 8b) inside the FQ-Generator. In the present disclosure, the processor (131) has the FQ-Generator as a candidate The point immediately before creating, i.e., conversation history The KV cache state at the time when tokens up to the conversation purpose G and related knowledge K have been processed is preserved in memory (132) as a snapshot. kv_cache_pointer is a pointer that references the memory address of this snapshot.

[0268] [Mathematical Formula 10]

[0269]

[0270] Here, represents the concatenation of a token sequence, and The candidate It indicates the point in time immediately before the creation of begins, is a snapshot of the KV cache at that point in time.

[0271] The preservation of these KV cache snapshots is, in step 4 (S400), the user [candidates] When this is selected, the purpose is to drastically reduce the Time to First First Time (TTFT) of answer generation by directly restoring the preserved KV cache snapshot to the main inference pipeline without re-encoding the entire conversation history and CoT inference results. The specific process of restoring the KV cache is described in detail in Step 4 (Fig. 11).

[0272] In one embodiment, since preserving individual KV cache snapshots for all candidates can increase memory usage, the processor (131) has a conversation history A method can be applied in which only one common KV cache prefix corresponding to , conversation purpose G, and related knowledge K is preserved, and for individual candidates, only the delta derived from the said common prefix is ​​stored. Accordingly, the memory capacity required for KV cache storage for m candidates is reduced from O(m < Х < N) to O( It can be reduced to ) (where N is the number of tokens of the common prefix, is the average size of delta per individual candidate).

[0273] In another embodiment, the processor (131) may preserve KV cache snapshots only for candidates corresponding to Top-k rankings and release the KV caches of the remaining candidates. By doing so, memory usage may be limited to a level proportional to the number of Top-k candidates.

[0274] And the processor (131) constructs a candidate pack for transmission to a user computing device (110) based on candidates for which ranking and metadata binding have been completed (S240).

[0275] A candidate pack is a structured data packet that includes not only the text of follow-up question candidates but also additional information necessary for real-time matching and display on the client side. In conventional follow-up question recommendation systems, it was common for the server to transmit only a list of follow-up question texts to the client; however, in this disclosure, a candidate pack containing a subset of candidate-specific metadata is transmitted to support real-time index construction (Step 3) and keystroke-based matching on the client side.

[0276] Referring to Fig. 9, the structure of the candidate pack is defined as follows.

[0277] Field Description Remarks session_id Unique identifier for the conversation session; session synchronization between server and client timestamp Timestamp at the time of creation pack_version Version number of the candidate pack; used for version increment upon regeneration and cache invalidation candidates[] Array of data by candidate (see Table 8 below); can include Top-k or higher candidates global_context Context information common to all candidates; summary of conversation purpose G, category distribution, etc.

[0278] Field Description Data Type question_id Unique identifier for the candidate question String question_text Text of the candidate question Text token_sequence Token sequence Sequence category_id Question category Enum rank_score Ranking score Float mbedding_vector Embedding vector (lightweight version) Float[d'] answerability_flag Answerability flag Enum tl Validity period Integer (seconds)

[0279] Here, the information included in the candidate pack may be a subset of the candidate-specific metadata (Table 6). Specifically, kv_cache_pointer, conversational_goal (detailed version), knowledge_items (detailed version), and user_intents are maintained only on the server side and are not included in the candidate pack. This is because (i) the KV cache pointer refers to an internal address in server memory, so there is no point in sending it to the client; (ii) sending detailed CoT inference results to the client results incurs unnecessary network bandwidth consumption; and (iii) it is undesirable to expose the system's internal inference process to the user. On the other hand, the embedding_vector is included in the candidate pack as a lightweight version (d' dimension, e.g., d' = 64 or 128) with reduced dimensions of the original embedding vector to support lightweight reranking on the client side. Details regarding this will be described later in Step 3. Next, the processor (131) transmits the configured candidate pack to the user computing device (110) via the network (170). In one embodiment, the candidate pack may be encoded and transmitted in a serialization format such as JSON or Protocol Buffer.

[0280] In one embodiment, the transmission of the candidate pack may be performed in parallel with the generation of the response (answer a_i) to the user's current query. That is, the processor (131) may process the generation of the response and the generation / ranking of subsequent question candidates asynchronously so that the candidate pack reaches the client before the streaming of the response text is completed. By doing so, while the user is reading the response, the client side can already start building an index based on the candidate pack (step 3), so that real-time matching is immediately possible when the user starts the next input.

[0281] The above candidate pack is not static data, but can be updated during regeneration / re-ranking by an uncertainty-based trigger described later in the third step. At this time, the processor (131) increments the pack_version of the updated candidate pack and retransmits it to the client, and the client reconstructs the existing index structure into a new candidate pack only if the received pack_version is newer than the version currently held. Through this version management, the client's index consistency is maintained even if the old version pack arrives late due to network latency.

[0282] In one embodiment, the processor (131) may classify the generated candidates into a predetermined category. The category of the candidate question indicates the direction in which the question guides the user's information search, and may be classified, for example, as follows.

[0283] Category Description Examples (Wilson Conversations) Expansion: Expanding the exploration from the current topic into new related subject areas "The impact of Wilson's teaching experience on his political career" Deepening: Exploring specific aspects of the current topic in greater depth "Major academic achievements during his tenure at Wesleyan" Verification: Confirming or quantifying specific facts from a previous answer "Period of tenure at Wesleyan" Comparison: Expanding the context through comparison with other cases or subjects "Comparison with contemporary scholars"

[0284] The above category information (category_id) is included in the candidate pack (Table 8) and transmitted to the client, and can be used to display grouping by category or distribution information by category when displaying follow-up questions to the user in the third step. In another embodiment, the processor (131) can perform personalized ranking based on the user profile P in the ranking step (S220). Specifically, the processor (131) can generate a personalization vector by vectorizing personalization features (e.g., morpheme usage frequency, average query length, preferred category distribution, modification frequency) extracted from the user's past conversation history, and add this to the input of the ranking model.

[0285] In this case, the integrated score S of mathematical formula 8 can be extended as follows.

[0286] [Mathematical Formula 11]

[0287]

[0288] Here, Sim( , P) is a candidate question It represents the similarity (e.g., cosine similarity) between and the user personalization vector P, and μ is a hyperparameter representing the personalization weight.

[0289] Through this personalized ranking, even within the same conversation context, verification questions are ranked higher for beginner users, while deepening or expansion questions are ranked higher for expert users, making it possible to recommend follow-up questions tailored to user characteristics.

[0290] Referring to FIG. 6, the output of the second step (S200) is connected to subsequent steps as follows.

[0291] First, the candidate pack (Tables 7 and 8) is transmitted to the user computing device (110) and used as basic data for constructing a prefix trie index and real-time keystroke matching in the third step (S300). The token_sequence included in the candidate pack is used for constructing the trie, the rank_score is used for sorting the scores of the trie nodes, and the embedding_vector is used for client-side lightweight re-ranking. Details regarding this will be described later in the third step.

[0292] Second, the candidate-specific metadata (Table 6) loaded in the server-side memory (132) is used to quickly restore the KV cache snapshot through the kv_cache_pointer bound to the candidate when the user selects a specific candidate in the fourth step (S400). Details regarding this will be described later in the fourth step.

[0293] Third, if regeneration / re-ranking is requested by an uncertainty-based trigger in the third stage, the processor (131) partially or wholly re-executes the sub-steps (S210 to S240) of the second stage and re-transmits the updated candidate pack (pack_version increment) to the client. Details regarding this will be described later in the third stage.

[0294] Thus, the second stage achieves both qualitative standards and system efficiency in follow-up question recommendations for conversational AI systems by integrating four technical functions: (a) high-quality candidate generation based on CoT inference results, (b) resource-efficient ranking that combines utility and serving costs, (c) metadata binding that enables KV cache reuse in subsequent stages, and (d) structured data packet transmission that supports client-side real-time matching.

[0295] Hereinafter, the third step of the follow-up question provision method according to one embodiment of the present disclosure, which is the client-side prefix tri-index construction, real-time keystroke matching, and uncertainty-based conditional regeneration step, will be described in detail.

[0296] The third step (S300) is a step in which the processor (111) of the user computing device (110) provides follow-up questions to the user in real time based on the candidate pack (see Tables 7 and 8) received from the server computing system (130) in the second step (S200).

[0297] The technical challenge of the third stage is to achieve updating and displaying follow-up question candidates corresponding to the input content without delay at every keystroke when the user inputs text, while not sending an inference request to the server at every keystroke to achieve this. If the server's LLM inference is called at every keystroke, problems arise such as (i) real-time response becoming impossible due to network round-trip latency, (ii) the server-side GPU computational load increasing rapidly in proportion to the keystroke frequency (typically 3 to 8 times per second), and (iii) the server's throughput rapidly degrading when there are many concurrent users.

[0298] To address these technical challenges, in the third stage, a conditional computation control mechanism is applied, which (a) constructs a prefix trie index based on a candidate pack on the client side to perform keystroke-based candidate matching locally on the client without round trips to the server, (b) automatically detects situations where client-local matching alone does not sufficiently reflect user intent using an entropy metric, and (c) triggers lightweight re-ranking and server-side regeneration in stages only when the entropy exceeds a predetermined threshold.

[0299] Referring to FIG. 10, the third step (S300) includes the following sub-steps: (i) constructing a prefix tri-index (S310), (ii) keystroke-based real-time matching (S320), (iii) calculating uncertainty (S330), and (iv) a conditional re-ranking or regeneration trigger (S340).

[0300] Referring to FIG. 10, the third step (S300) is performed by the division of roles between the user computing device (110) and the server computing system (130).

[0301] The processor (131) of the server computing system (130) performs CoT-based context inference in the first stage (S100), candidate generation, ranking, and metadata binding in the second stage (S200), and KV cache restoration-based answer generation in the fourth stage (S400).

[0302] The processor (111) of the user computing device (110) performs the prefix trie construction, real-time matching, uncertainty calculation, and lightweight re-ranking of the third stage (S300).

[0303] By this division of roles, compute-intensive inference is performed in the server computing system (130), and latency-sensitive keystroke matching is performed in the user computing device (110), thereby resolving the trade-off between response delay to user input and server load.

[0304] Referring to FIG. 10, the processor (111) of the user computing device (110) constructs a prefix trie index structure in memory (112) based on the candidate token sequence (token_sequence) included in the candidate pack (Tables 7, 8) received from the server computing system (130) (S310).

[0305] A prefix trie (hereinafter referred to as a trie) is a tree-shaped index data structure capable of performing prefix-based search on a set of strings with O(L) time complexity (L: length of the search string). In the present disclosure, each path of the trie (a path from the root to a specific node) corresponds to a prefix of a token sequence of candidate questions, and each node stores an identifier of the candidate question and related information that matches the corresponding prefix.

[0306] The processor (111) constructs a trie for each candidate j (j = 1, 2, ..., k') included in the candidates[] array of the candidate pack in the following process, where k' is the number of candidates included in the candidate pack, and in addition to the Top-k candidates of the second step, lower-ranked candidates for reference may be additionally included.

[0307] Step (i): A sequence of tokens obtained by splitting candidate j's question_text into subword tokens. Acquires. Here, is the length of the token sequence of candidate j. Since the token sequence is pre-calculated and included in the token_sequence field of the candidate pack, separate tokenization on the client side is unnecessary.

[0308] Step (ii): From the root node of the trie, the token sequence Each token of Follow the steps in order, and if there is no corresponding child node, create a new node.

[0309] Step (iii): Store information about candidates matching the corresponding prefix in each intermediate node and leaf node of the token sequence.

[0310] The following information is stored in each trie node.

[0311] Field Description Data Type children Map of child nodes corresponding to the next token<Token_ID, Node> matched_candidates List of candidates matching the prefix (top n, sorted in descending order of score) List<(question_id, adjusted_score)> category_distribution Distribution of candidate categories for the prefix Map<Category_ID, Float> prefix_depth The depth of the current node within the trie (number of prefix tokens) Integer is_terminal Whether the entire token sequence of one or more candidates is completed at this node Boolean

[0312] In Table 10, the matched_candidates field stores the top n (e.g., n = 5) candidate identifiers and adjusted scores among all candidates sharing the prefix reaching the corresponding node. The adjusted score is calculated by reflecting the prefix match ratio in the candidate pack's rank_score and is described in detail in Section 4.2 below. The category_distribution field represents the proportion of candidates matching the corresponding prefix by category (see Table 9) and can be used as an auxiliary tool for the uncertainty calculation (S330) described below. In the aforementioned Woodrow Wilson conversation example, splitting the text of the Top-5 candidates (Table 5) included in the candidate pack into subword tokens constructs the following trie.

[0313] Node (Token)matched_candidatesRoot[All Candidates]Wesleyan university ]at how much since Woodrow Employment time Wilson's union experience since Experience Princeton

[0314] In the example above, when the user inputs up to "Wesleyan", it reaches the "Wesleyan" node of the trie and matched_candidates = [ ] is returned immediately. If the user inputs up to "at Wesleyan University", it reaches a deeper node and matched_candidates = [ The range is narrowed to )]. Candidate matching by such trie search is performed with O(L) time complexity for the number of user input tokens L, and no network request is made to the server computing system (130).

[0315] In one embodiment, the processor (111) may build an index using a Radix Trie (or Patricia Trie) instead of a conventional Prefix Trie. A Radix Trie is a data structure that compresses a path with only one child node into a single edge, reducing memory usage by decreasing the number of nodes compared to a conventional Prefix Trie, while maintaining an O(L) search time complexity.

[0316] In the scenario of the present disclosure, where the number of candidates k' is relatively small (e.g., 10 to 20) and the token sequence length of each candidate is different, the compression rate by the radix trie can be significantly high. For example, in the example of Table 11, the path “Wesleyan” “University” “After” “ is compressed into a single edge “Wesleyan University After” in the radix trie, reducing the number of nodes from 4 to 1.

[0317] And the processor (111) receives a user's keystroke event in real time through an input component (121) (S320). When the user inputs text, the processor (111) divides the string entered so far into subword token units, and the token sequence of the current input prefix It maintains it in real-time.

[0318] In one embodiment, token splitting can be performed by loading the same tokenizer used to generate the token_sequence of the candidate pack onto the user computing device (110). Since the subword tokenizer (e.g., BPE, SentencePiece) typically requires only a vocabulary of a few kilobytes (KB) in size, the memory burden from loading on the client side is negligible.

[0319] In another embodiment, instead of token splitting on the client side, it is also possible to perform character-level matching. In this case, even when constructing the trie, the path is generated at the character level rather than the token level; while precision may be slightly reduced, this has the advantage of eliminating the need for a tokenizer.

[0320] Additionally, the processor (111) traverses the constructed prefix trie sequentially from the root node for the current input prefix token sequence T_input to reach the node corresponding to T_input.

[0321] If the search is successful, the processor (111) obtains a list of matched candidates from the matched_candidates field of the corresponding node. At this time, the adjusted score for each candidate is calculated by multiplying the ranking score of the candidate included in the candidate pack by the prefix matching ratio. The higher the prefix matching ratio, the higher the adjusted score of the candidate.

[0322] If the search fails, the processor (111) returns a matching candidate for the node corresponding to the longest matching prefix, and may apply a damping factor proportional to the number of mismatch tokens to the adjustment score.

[0323] Next, the processor (111) sorts the matched candidates based on the calculated adjustment scores and displays the top candidates to the user through the display of the user computing device (110). The display method may include, for example, (i) displaying as an auto-complete dropdown at the bottom of the input field, (ii) displaying as a list of recommended questions next to the input field, or (iii) displaying the most likely candidate as ghost text within the input field.

[0324] In one embodiment, the processor (111) may display to the user the category information (category_id) and answerability flag (answerability_flag) of each candidate along with the matching result. For example, visual distinctions may be applied, such as displaying extended questions in a first color (e.g., blue) and deepening questions in a second color (e.g., green), and adding a check icon to questions with an answerability of "HIGH".

[0325] The entire process of the above-mentioned try search and matching result display is performed only within the local memory (112) of the user computing device (110) for each keystroke event and does not involve a network request to the server computing system (130). As a result, the delay between the keystroke event and the matching result display consists only of the time required for the O(L) operation of the try search and the rendering of the display unit, thereby completely eliminating the usual network round-trip time (tens to hundreds of milliseconds).

[0326] Here, when the user selects one of the displayed candidates (e.g., Tab key or touch) or accepts the displayed autocomplete text (e.g., Enter key), the processor (111) transmits the question_id of the selected candidate to the server computing system (130). The server computing system (130) retrieves candidate-specific metadata (Table 6) corresponding to the received question_id from memory (132) and proceeds to the KV cache restoration and answer generation of the fourth step (S400). Details regarding this will be described later in the fourth step.

[0327] If the user confirms the input by directly completing the text without selecting a candidate, the processor (111) transmits the entire text entered by the user to the server computing system (130), and in this case, dynamic knowledge assembly may be applied in the fourth step.

[0328] As a core technical configuration of the third stage, the processor (111) calculates the uncertainty regarding the result of the keystroke matching (S320) in real time (S330). The uncertainty is a metric that quantitatively indicates how well the currently matched candidate list reflects the user's input intent.

[0329] In cases of low uncertainty (i.e., when a specific candidate shows an overwhelmingly high match with the current input), the user's intent can be sufficiently reflected using only try-matching based on the existing candidate pack, so additional server calls are unnecessary. On the other hand, in cases of high uncertainty (i.e., when matching scores are evenly distributed among multiple candidates or the overall matching score is low), server-side re-ranking or regeneration is required because the user's input is likely outside the range of the existing candidate list.

[0330] The calculation of these uncertainty metrics and conditional server calls based thereon functions not merely as a user interface improvement, but as a resource control mechanism that allocates the server's GPU computational resources only when necessary.

[0331] And the processor (111) calculates information entropy as an uncertainty metric based on the distribution of adjustment scores of the currently matched candidates.

[0332] [Mathematical Formula 12]

[0333]

[0334] Here, n is the number of currently matched candidates, and is the normalized adjusted score (converted into a probability distribution) of the matched candidate j.

[0335] [Mathematical Formula 13]

[0336]

[0337] Entropy calculated from mathematical equation 12 It is interpreted as follows.

[0338] 1) When H is low (e.g.: ): A single candidate accounts for the majority of the adjustment score, and the match between the user's input intent and existing candidates is clear. No additional computation is required.

[0339] 2) When H is high (e.g.: ): Adjustment scores are evenly distributed among multiple candidates, making it difficult to identify the user's input intent based solely on existing candidates. Additional computation may be required.

[0340] The processor (111) triggers conditional re-ranking or regeneration based on the calculated uncertainty metric (S340). In this disclosure, a two-stage escalation architecture is adopted to adaptively respond to changes in user intent while minimizing unnecessary server calls.

[0341] The first stage escalation (lightweight reordering, S341) is a lightweight operation performed within the user computing device (110) and does not involve a server call. The second stage escalation (server regeneration, S342) is triggered by sending a regeneration request to the server computing system (130) only if the uncertainty is not resolved even by the first stage escalation.

[0342] The trigger conditions for the first stage escalation (S341) and the second stage escalation (S342) are defined as follows.

[0343] Trigger Condition Execution Location 1st Stage Escalation (S341) [Condition A] Debounce Timer Expiration (No additional input for a specified time after user input) AND [Condition B] Entropy User computing device (110) 2nd stage escalation (S342) [Condition C] Entropy even after performing S341 AND [condition D] elapsed time since the last server regeneration (rate-limit) server computing system (130)

[0344] This is intended to prevent escalation from being triggered with every keystroke while the user is typing. By determining uncertainty only at the point where the user pauses, the frequency of unnecessary calculations is reduced. Condition D (rate-limit) is intended to prevent the server regeneration request from occurring repeatedly within a short period of time, thereby preventing the GPU computation load of the server computing system (130) from increasing excessively.

[0345] When conditions A and B are satisfied, the processor (111) performs lightweight re-ranking using the embedding_vector of the candidate pack stored in the client memory (112) without calling the server (S341).

[0346] Specifically, the processor (111) performs the following process.

[0347] Step (i) ― Vectorization of current input: The processor (111) processes the user input text up to now ...the current state vector through a lightweight encoder mounted on the user computing device (110) Convert to.

[0348] Here, the lightweight encoder is a small model (e.g., a shallow transformer of 1 to 2 layers, or a simple projection network) separate from the main LLM on the server side, having a level of computation that can be executed in milliseconds on the processor (111) of the user computing device (110). The output dimension d' of the lightweight encoder is the same as the dimension of the embedding_vector included in the candidate pack (e.g., d' = 64 or 128).

[0349] Step (ii) — Vector similarity calculation: The processor (111) calculates the current state vector and embedding_vector of each candidate in the candidate pack Calculate the cosine similarity between them.

[0350] Step (iii) ― Re-ranking: The processor (111) recalculates the score of each candidate by combining the original ranking score and vector similarity based on a predetermined weight.

[0351] Step (iv) — Trie Update: The processor (111) updates the matched_candidates below the current node of the prefix trie based on the re-ranked result.

[0352] Step (v) ― Recalculate uncertainty after update: The processor (111) determines the entropy for the updated matching result Recalculate H' If it is less than or equal to the first stage escalation, it is determined that the uncertainty has been resolved solely through the first stage escalation, and no further escalation is performed.

[0353] The entire process of the first stage escalation described above is completed within the user computing device (110) and does not involve a network request to the server computing system (130). The inference and vector similarity calculation of the lightweight encoder consists of performing k' times (number of candidates) of inner product operations on a d'-dimensional vector, and can be completed within milliseconds.

[0354] Even after performing the first stage escalation (S341), the entropy H' Exceeding, and from the previous server regeneration When the above period has elapsed (conditions C and D are satisfied), the processor (111) sends a regeneration request to the server computing system (130) (S342).

[0355] The regeneration request includes (i) user input text up to now , (ii) pack_version of the current candidate pack, and (iii) current uncertainty metric (H', Margin) are included.

[0356] When the processor (131) of the server computing system (130) receives a regeneration request, it re-executes the necessary range of the sub-steps of the second step (S200). Specifically, the range of regeneration can be determined as one of the following two modes.

[0357] Mode Condition Re-execution Scope Cost Re-ranking Mode (Rerank-only) If this partially matches one or more of the existing candidates, only S220 (Ranking) and S240 (Pack Reorganization) are re-performed. Recalculate the rankings of existing candidates using additional context Low Full-regeneration mode If this does not match any existing candidate, re-perform the entire S210 (candidate generation) to S240 (pack reconfiguration). Generate new candidates by adding to conversation history

[0358] When the re-execution is completed, the server computing system (130) transmits the updated candidate pack (pack_version increment) to the user computing device (110). The processor (111) of the user computing device (110) confirms that the received pack_version is newer than the version currently held, and then reconstructs the prefix trie based on the updated candidate pack (re-execution of S310). In this way, by constructing the prefix trie index on the client side and performing candidate matching in keystrokes without round trip to the server, the candidate update delay for user input is limited to only O(L) trie search time. Accordingly, compared to the method of sending an inference request to the server for every keystroke in the prior art, the number of network round trips is reduced to 0 (when input is within the candidate list range) or at most 1 (when regeneration is triggered) in the number of keystrokes. In addition, since server-side LLM inference is performed only conditionally by the uncertainty-based trigger mechanism, the number of server GPU inference calls during the entire conversation session is significantly reduced. Considering that dozens of keystrokes occur for a user to input a single follow-up question in a typical conversation session, the effect of reducing server calls according to the present disclosure is significant.

[0359] In addition, through the two-stage escalation architecture, server calls do not occur in most cases where uncertainty is resolved by client-side lightweight reordering (first stage escalation), and server regeneration (second stage escalation) is triggered only when the user intent cannot be reflected by the client side alone, thereby achieving a balance between user experience quality and server resource efficiency.

[0360] In addition, due to debounce and rate-limit conditions, escalation is suppressed while the user is typing quickly, preventing unnecessary operations in intermediate states, and uncertainty judgment is performed only when the user pauses briefly, thereby mitigating the visual instability problem where the candidate list fluctuates excessively.

[0361] Referring to FIG. 11, the output of the third step (S300) is connected to the subsequent step as follows.

[0362] When the user selects one of the displayed candidates (tap, touch, or enter), the processor (111) transmits the question_id of the selected candidate to the server computing system (130), and the server computing system (130) performs the fourth step (S400). In the fourth step, a rapid answer generation by KV cache restoration (Prompt Stitching) is performed by referring to the kv_cache_pointer of the candidate-specific metadata (Table 6) corresponding to the selected question_id.

[0363] If the user confirms the input by directly completing the text without selecting a candidate, the server computing system (130) performs dynamic knowledge assembly (Delta Reasoning) in the fourth step and partially recycles the metadata of the most similar candidate.

[0364] Hereinafter, with reference to FIG. 11, the fourth step of the method for providing a follow-up question according to one embodiment of the present disclosure, which is the KV cache restoration-based answer generation and dynamic knowledge assembly step, will be described in detail.

[0365] Step 4 (S400) is a step in which the processor (131) of the server computing system (130) generates an answer to a subsequent question selected by the user or confirmed as input in Step 3 (S300).

[0366] Step 4 performs (a) KV cache restoration (hereinafter Prompt Stitching), which avoids re-encoding by directly restoring the KV cache snapshot bound to candidate-specific metadata (Table 6) in Step 2 to the main inference pipeline, and (b) Dynamic Knowledge Assembly (hereinafter Delta Reasoning), which incrementally augments knowledge only for the modified parts when the user partially modifies the recommendation question.

[0367] Referring to FIG. 11, the fourth step (S400) branches into two paths depending on the type of user input confirmation: (i) when the user selects the recommended candidate as is (S410: prompt stitching path), and (ii) when the user partially modifies the recommended candidate or directly enters new text (S420: dynamic knowledge assembly path).

[0368] When the user selects one of the candidates displayed in the third step (e.g., Tab key, touch, or accept autocomplete), the user computing device (110) transmits the question_id of the selected candidate to the server computing system (130). The processor (131) of the server computing system (130) retrieves candidate-specific metadata (Table 6) corresponding to the received question_id from memory (132) and generates an answer by prompt stitching (S410).

[0369] Prompt stitching is a technique that, instead of inputting the entire conversation history and CoT inference results back into the LLM in text form, directly loads a KV cache snapshot (see Equation 10) that is pre-calculated in the second stage and stored in memory (132) into the inference pipeline of the LLM, thereby omitting most of the prefill operations. The term "stitching" implies that the KV cache blocks of the stored KV cache and the KV cache of the newly added question token are stitched together seamlessly to form a single continuous context.

[0370] Answer generation by prompt stitching (S410) includes the following sub-steps in sequence.

[0371] Substep (S411) ― Metadata retrieval and KV cache pointer dereferencing:

[0372] The processor (131) searches for candidate-specific metadata (Table 6) corresponding to the received question_id in memory (132). From the retrieved metadata, the kv_cache_pointer is referenced to obtain a KV cache snapshot at the time of generating the corresponding candidate. Acquires the KV cache snapshot It consists of a set of key tensors and value tensors for each transformer layer of the LLM.

[0373] Sub-step (S413) ― Initiate additional prefill and decoding:

[0374] The processor (131), with the KV cache loaded, selects candidate questions Only the token sequence of is treated as an incremental prefill.

[0375] [Mathematical Formula 14]

[0376]

[0377] Here, is, candidate question It is the KV cache increment generated by performing a prefill operation only on the tokens. Number of tokens in the candidate question | Since | is typically at the level of 10 to 30 tokens, the amount of computation for this additional prefill is very small.

[0378] Once the additional prefill is complete, the LLM refers to KV_full and immediately initiates the sequential decoding (auto-regressive decoding) of answer tokens. As a result, the time-to-First-Token (TTFT) from when the user selects a candidate until the first answer token is generated is the prefill target From tokens | As it decreases into tokens, it is drastically shortened.

[0379] Sub-step (S414) ― Streaming response transmission:

[0380] The processor (131) streams the generated answer tokens in real time to the user computing device (110) via the network (170). The display of the user computing device (110) renders the received tokens sequentially and displays them to the user.

[0381] Restoring a KV cache preserved by prompt stitching is the context at the time the corresponding KV cache was created (i.e., It is valid when the point in time when tokens up to , G, and K are processed exactly matches the context required for generating the answer.

[0382] In the present disclosure, KV cache snapshot is, a candidate in the second stage Conversation history to generate , is the state at the time when conversation purpose G, and related knowledge K are entered into the LLM. Therefore, the user has the same candidate If selected as is, the context required for answer generation ( , G, K, )silver, to Connected additional prefills It matches exactly.

[0383] Furthermore, since the intermediate results (G, K) of CoT inference are embedded within the KV cache during answer generation, the LLM generates answers while retaining the background context of "why this question was recommended." This provides the effect of improved contextual coherence of answers and reduced risk of hallucinations, compared to conventional methods that treat only follow-up questions as new input.

[0384] As described above in Step 2 (S230), the following method may be applied to manage the memory usage of the KV cache snapshot.

[0385] Method (i) ― Sharing a common prefix: Conversation history Only one common KV cache prefix corresponding to , conversation objective G, and related knowledge K is preserved, and for individual candidates, only the delta derived from the common prefix is ​​stored. During prompt stitching, the common prefix and the delta of the selected candidate are combined. Reconfigures it. The resulting memory saving effect is as described above in the second step.

[0386] Method (ii) ― TTL-based automatic release: A time-to-live (TTL) is set for each KV cache snapshot (see the ttl field in Table 6), and snapshots whose TTL has expired are automatically released from memory. This prevents server memory from being overoccupied even when the user is inactive for a long time.

[0387] Method (iii) ― Top-k limited preservation: Only the KV caches of candidates corresponding to the Top-k rank are preserved, and the KV caches of the remaining candidates are released immediately after the ranking is completed.

[0388] Method (iv) ― LRU (Least Recently Used) Eviction: A capacity is set for the KV cache-only area of ​​server memory, and an LRU policy is applied to eviction the KV cache of the session that has not been referenced for the longest time when the capacity is exceeded. In one embodiment, a learning-based eviction policy may be applied to predict the conversational persistence of a session (e.g., frequency of recent input, number of conversation turns) and to prioritize eviction of the KV cache of a session with low persistence.

[0389] If the user does not select the recommended candidates as is, but instead (i) finalizes the input by adding, deleting, or modifying some words based on the recommended candidates, or (ii) enters completely new text unrelated to the recommended candidates, it is not appropriate to apply prompt stitching as is. The user's final input is the candidate Because it is different from, This is because a discrepancy occurs between the context inherent in it and the actual input.

[0390] However, when there is significant similarity between the user's final input and a specific candidate (e.g., 70–90% of the candidate is retained and only a part is modified), it is computationally wasteful to completely discard the Knowledge K and KV caches bound to the metadata of the candidate and infer from scratch. In such situations, the present disclosure applies Dynamic Knowledge Assembly to reuse reusable parts and perform incremental inference only on the changed parts.

[0391] As a first sub-step of the dynamic knowledge assembly (S420), the processor (131) [complies] with the user's final input text Tokenize into a token sequence { Generate} and this as the most similar candidate The token sequence of { Compared to}, the token divergence point Detects (S421).

[0392] Here, Choi Yusa candidate is the candidate that had the highest adjustment score with the user's final input in the third stage, or the original candidate that the user started modifying based on autocomplete.

[0393] [Mathematical Formula 15]

[0394]

[0395] In Equation 15, the token divergence point Dp represents the location where the token sequences of the user input and the most similar candidate first diverge. The tokens prior to Dp ( ) is a common prefix where user input and candidates match, and the tokens after D_p are divergent suffixes modified by the user.

[0396] In one embodiment, the detection of token divergence points is performed by a simple linear scan that sequentially compares two token sequences from the beginning, in O( It is performed with time complexity.

[0397] The processor (131) partially reuses the KV cache snapshot of the most similar candidate based on the detected token divergence point (S422).

[0398] Specifically, the processor (131) is, The KV cache corresponding to the common prefix is ​​determined to be valid and loaded into the inference pipeline. The KV cache after the divergence point is invalidated.

[0399] [Mathematical Formula 16]

[0400]

[0401] Here, [:N_prefix + ] is from the beginning in the KV cache snapshot ( It is a slicing of only the KV pairs up to the )th token position. is the number of tokens in the common context.

[0402] Next, the processor (131) is a diverging portion of the user input ( Expand the KV cache by performing additional prefills only for ).

[0403] [Mathematical Formula 17]

[0404]

[0405] As a result, the prefill operation for common prefixes is omitted, and since prefill is performed only on tokens of the divergence part, the total amount of prefill operations is ( Decreases by the amount of ) tokens.

[0406] The processor (131) selectively maintains or updates each knowledge item of related knowledge K bound to the metadata (Table 6) of the most similar candidate based on the divergence point (S423). This process is the core of dynamic knowledge assembly.

[0407] The processor (131) performs the following process.

[0408] Step (i) ― Determine relevance by knowledge item:

[0409] The processor (131) has relevant knowledge K = {of the most similar candidate Each knowledge item of} For this, determine whether the knowledge item is primarily referenced by the common prefix (token before the divergence point) or by the divergence part (token after the divergence point).

[0410] This judgment is performed using the attention score calculated during the candidate generation process in the second stage. Specifically, the processor (131) calculates a retention ratio for each knowledge item by comparing the attention weights from tokens of the common prefix part with the attention weights from tokens of the divergence part. Knowledge items with a retention ratio greater than or equal to a predetermined threshold are maintained as valid knowledge that is mainly referenced by the common prefix, and knowledge items with a retention ratio less than the threshold are classified as targets for update.

[0411] Step (ii) ― Delta Knowledge Inference:

[0412] The processor (131) infers new knowledge (hereinafter delta knowledge K') to replace knowledge items classified as update targets using a lightweight inference model. The lightweight inference model generates supplementary knowledge related to the modified part by considering the user's modified input, the existing conversational purpose G, and the maintained knowledge.

[0413] The processor (131) performs decoding of the answer by utilizing the relevant knowledge merged with the KV cache reconstructed based on the divergence point (S424). Specifically, the merged knowledge is tokenized and reflected as additional context by performing an additional prefill at the end of the KV cache.

[0414] The processor (131) is, Decoding of the answer tokens is initiated by referring to the, and the generated tokens are streamed to the user computing device (110).

[0415] User's final input If it does not show significant similarity to any candidate in the candidate pack (e.g., token divergence point) When is 1, i.e., when different from the first token), the utility of dynamic knowledge assembly is minimal. In this case, the processor (131) uses, as a fallback path, the common KV cache prefix stored in the kv_cache_state of the session context (Table 2) preserved in the first step ( Restore only the corresponding text, and generate an answer by performing additional prefills on the user's entire input text T_user.

[0416] In this case as well, conversation history Since the prefill for conversation purpose G and related knowledge K is replaced by restoring the common KV cache, the amount of prefill operations is reduced compared to the conventional method of re-encoding the entire history from the beginning.

[0417] After the answer generation is complete, the processor (131) of the server computing system (130) receives the newly generated answer Updates the session context by adding it to the conversation history. Updated conversation history Based on this, the entire pipeline of the first step (S100) to the fourth step (S400) can be executed again. By doing so, an iterative conversation flow is implemented in which the recommendation of follow-up questions and the generation of answers are repeatedly performed at each turn of the conversation.

[0418] At this time, in the first stage of the new turn, it is possible to reuse a portion of the KV cache preserved in the previous turn to limit the re-encoding range of the conversation history to the newly added portion. The specific implementation of this is achieved by the aforementioned KV cache management method (common prefix sharing, TTL-based automatic release, etc.).

[0419] In one embodiment, the answer generation of the fourth step is not limited to a text-based answer but can be extended to a multimodal form. Specifically, if the answer to the user's follow-up question includes image, video, or voice data, the processor (131) can quickly construct a text context by restoring the KV cache and then generate an answer with additional modalities through a multimodal generation module (e.g., a text-to-image generation model, a text-to-speech conversion model).

[0420] Even in this case, since the text context prefill time is shortened by restoring the KV cache, the technical effect of reducing the total generation time of multimodal answers is maintained.

[0421] In another embodiment, the KV cache snapshot and candidate-specific metadata stored in the memory (132) of the server computing system (130) may contain personal information (e.g., username, location information, personal identification information) included in the conversation history. Accordingly, the processor (131) may apply an anonymization technique to mask or noise-inject the KV cache value of the token location corresponding to the sensitive information when creating or storing the KV cache snapshot.

[0422] In addition, access to KV cache snapshots is subject to access control based on session identifiers (session_id) and user authentication tokens, blocking unauthorized access from other sessions or users.

[0423]

[0424] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0425] The specific embodiments described in this disclosure are examples and do not limit the scope of this disclosure in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of this disclosure.

[0426] Furthermore, although the detailed description of the present disclosure has been explained with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that the present disclosure can be modified and changed in various ways without departing from the spirit and technical scope of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0427]

[0428] The method and system for providing follow-up questions in a conversational artificial intelligence service according to the present disclosure can be widely applied to various industrial fields and commercial service platforms utilizing large-scale language models (LLM). In particular, since real-time responsiveness can be secured and server infrastructure maintenance costs can be drastically reduced through computational distribution between the server and the client and KV cache recycling technology, it has high applicability in the following industrial fields.

Claims

1. A method performed by a computer device, A step in which at least one processor of the computer device generates conversational purpose and related knowledge by performing a Chain-of-Thought (CoT) based inference process using a machine learning model based on a conversation history received from a user computing device via a network; The above at least one processor stores a session context in memory that includes the conversation history, the conversation purpose, and the related knowledge, and includes a pointer to a Key-Value (KV) cache state generated in the CoT-based inference process in the session context; The above at least one processor generates a plurality of follow-up question candidates based on the conversation purpose and the relevant knowledge, performs ranking on the plurality of follow-up question candidates to determine the ranked candidates, and for each of the ranked candidates, generates metadata including a pointer to a KV cache snapshot generated during the generation process of the candidate and stores it in the memory; The above at least one processor comprises the step of configuring a candidate pack including the text, token sequence, and ranking score of the ranked candidates; The above at least one processor constructs a prefix tri-index in memory based on the token sequence included in the candidate pack, and stores an identifier and a ranking score of a candidate matching the corresponding prefix in each node of the tri-index; and The processor of the user computing device, in response to a user input event, searches the trie index with a token sequence corresponding to the text entered so far to determine a matching candidate, and implements the determined candidate in the user interface of the user computing device; comprising the step of Method for providing follow-up questions in a conversational AI service.

2. In Paragraph 1, When one of the above-mentioned implemented candidates is selected by a user, the user computing device transmits the identifier of the selected candidate to a server computing system; and A processor of the server computing system obtains a KV cache snapshot by referencing a KV cache pointer corresponding to the identifier of the selected candidate in the metadata stored in the memory, and loads the KV cache snapshot into the inference pipeline of the machine learning model to generate an answer for the selected candidate of the conversation history; A method for providing follow-up questions in a conversational AI service that further includes 3. In Paragraph 2, The step of generating the above answer is, A step of loading the above KV cache snapshot into the KV cache memory area of ​​the inference pipeline so that the machine learning model has an internal state identical to that of having already processed tokens corresponding to the conversation history, the conversation purpose, and the related knowledge; and A step comprising: performing additional prefill operations only on the tokens of the selected candidates above, and then initiating the generation of answer tokens; Method for providing follow-up questions in a conversational AI service.

4. In Paragraph 2, A step in which a processor of the above-mentioned user computing device modifies text based on any one of the above-mentioned implemented candidates and transmits the input to a server computing system; A processor of the above-described server computing system compares the token sequence of the user's final input text with the token sequence of any one of the above-described candidates to detect a token divergence point, which is the location where it first diverges; and The method further comprises the step of reusing a KV cache corresponding to a common prefix prior to the token divergence point from a snapshot of the KV cache of any one of the candidates, and generating an answer by performing additional inference only on the changed parts after the token divergence point. Method for providing follow-up questions in a conversational AI service.

5. In Paragraph 4, The step of performing the above additional inference is, Among the relevant knowledge included in the metadata of any one of the above candidates, a knowledge item in which the attention weight for tokens prior to the token divergence point is greater than or equal to a predetermined threshold is determined as a retention target, additional knowledge is generated based on the changed tokens after the token divergence point, and an answer is generated by merging the retention target knowledge and the additional knowledge. Method for providing follow-up questions in a conversational AI service.

6. In Paragraph 1, The step of performing the above ranking is, For each of the plurality of follow-up question candidates, an integrated score based on a utility score and an estimated serving cost is calculated, wherein the integrated score is calculated by subtracting the value obtained by multiplying the estimated serving cost by a weight from the utility score, and the ranked candidates are determined based on the integrated score. Method for providing follow-up questions in a conversational AI service.

7. In Paragraph 6, The above estimated serving cost is calculated based on at least one of the probability that a cache hit will not occur when generating an answer for the relevant candidate, the number of tokens required for the prefill operation, and the cost incurred for external knowledge retrieval, Method for providing follow-up questions in a conversational AI service.

8. In Paragraph 6, The above weight is dynamically adjusted based on the server's current load status, and when the server load is high, the weight increases to give greater weight to the above estimated serving cost. Method for providing follow-up questions in a conversational AI service.

9. In Paragraph 1, A processor of the user computing device calculates uncertainty regarding the ranking score distribution of matched candidates based on the search result of the trie index; and If the above uncertainty exceeds a first threshold, the user computing device performs re-ranking based on the embedding vector included in the candidate pack; including, Method for providing follow-up questions in a conversational AI service.

10. In Paragraph 9, If, even after the above re-ranking, the uncertainty exceeds a second threshold greater than the first threshold, the user computing device transmits a regeneration request to a server computing system, the server computing system performs the regeneration or re-ranking of the plurality of subsequent question candidates and transmits an updated candidate pack to the user computing device. Method for providing follow-up questions in a conversational AI service.

11. In Paragraph 1, The above ranking is a listwise ranking that determines the ranking by simultaneously considering all of the above multiple follow-up question candidates, Method for providing follow-up questions in a conversational AI service.

12. In Paragraph 1, The above CoT-based inference process is, A sub-step for extracting turn-by-turn user intent from the above conversation history; A sub-step for deriving the conversation purpose based on the extracted user intent; and A sub-step for generating the relevant knowledge based on the purpose of the above conversation; comprising Method for providing follow-up questions in a conversational AI service.

13. In Paragraph 1, The above KV cache snapshot is a common KV cache corresponding to the conversation history, the conversation purpose, and the related knowledge, and is separated into individual candidate differential data and stored in the memory, Method for providing follow-up questions in a conversational AI service.

14. In Paragraph 1, The step of constituting the above candidate pack is, Including the text, token sequence, ranking score, and dimensionality-reduced embedding vector of the above-mentioned ranked candidates, Method for providing follow-up questions in a conversational AI service.

15. In Paragraph 1, The step of performing the above ranking is, Calculating the similarity between each candidate and a personalized vector extracted from the user's past conversation history, and further including reflecting the similarity in the ranking. Method for providing follow-up questions in a conversational AI service.

16. In Paragraph 1, The configuration and transmission of the above candidate pack are performed asynchronously with the generation of a response to the user's current query, so that the candidate pack is transmitted to the user computing device before the streaming of the response is completed. Method for providing follow-up questions in a conversational AI service.

17. As a system that provides follow-up questions in a conversational artificial intelligence service, It includes a server computing system and a user computing device connected to communicate via a network, The above server computing system includes a processor and memory, and the processor, Based on the conversation history received from the user computing device, a CoT-based inference process using a machine learning model is performed to generate a conversation purpose and related knowledge, and a session context including the conversation history, the conversation purpose, the related knowledge, and a pointer to the KV cache state generated in the CoT-based inference process is stored in the memory; Based on the above conversation purpose and the above related knowledge, a plurality of follow-up question candidates are generated and ranking is performed; for each of the ranked candidates, metadata including a pointer to a KV cache snapshot is generated and stored in the memory; and a candidate pack including the text, token sequence, and ranking score of the ranked candidates is constructed and transmitted to the user computing device; The above user computing device includes a processor and memory, and the processor, Based on the token sequence included in the candidate pack received from the server computing system, a prefix trie index is constructed in the memory; A method configured to respond to a user input event, search the trie index with a token sequence corresponding to the text entered so far to determine a matching candidate, and implement the determined candidate in a user interface. A system that provides follow-up questions in a conversational AI service.

Images