Method and electronic apparatus for training model

Abstract

Disclosed is a method for training a model, the method being performed by at least one processor and comprising the steps of: acquiring input data; on the basis of a machine learning model, generating response data about the input data; acquiring feedback on the response data; generating a training dataset on the basis of the feedback; and training the machine learning model by using the training dataset, wherein the training dataset comprises true response data, false response data, and location information of a portion that becomes a target for training in the response data.

Description

Model learning method and electronic device

[0001] The present disclosure relates to a method for learning a model and an electronic device.

[0002] Recently, in the field of natural language processing technology, techniques are being developed to optimize model performance based on large language models (LMs) to meet user requirements. In addition, as LLMs have become capable of processing multi-modality data, there is a growing demand for training tasks that extract specific information from data of various formats.

[0003] Meanwhile, in the case of models such as LLM, the model can be modified by reflecting user feedback on responses generated based on the model. However, conventionally, user feedback on responses only included positive or negative evaluations, such as "correct" or "incorrect" or "prefer" or "do not prefer," making it necessary to learn the entire text provided in the response. In this case, massive training data and resources are required, and the model may overfit or errors may occur during the training process as it learns unnecessary parts; furthermore, it may be difficult to train the model to suit the functions desired by users. Accordingly, there is a need for the development of technology that generates a training dataset suitable for target learning based on feedback on responses and trains the model using the generated training dataset.

[0004] The present disclosure provides a model learning method and an electronic device for solving the above-mentioned problems.

[0005] The present disclosure may be implemented in various ways, including a computer-readable non-transient recording medium that records instructions for execution in a method, device (system), and / or computer.

[0006] According to one embodiment of the present disclosure, a method for learning a model, performed by at least one processor, comprises the steps of acquiring input data, generating response data for the input data based on a machine learning model, acquiring feedback for the response data, generating a training data set based on the feedback, and training a machine learning model using the training data set, wherein the training data set may include true response data, false response data, and location information of a part of the response data that is the target of learning.

[0007] According to one embodiment, the step of generating a training data set may include: setting the response data as false response data if the feedback includes data indicating negation; generating true response data by modifying the incorrect part of the response data based on a language model; and obtaining location information based on the location of the incorrect part.

[0008] According to one embodiment, the step of generating a training data set may include the step of setting response data as true response data when the feedback includes data indicating a positive response, the step of generating false response data by changing a part of the response data, and the step of obtaining location information based on the location of the changed part.

[0009] According to one embodiment, the step of generating a training data set may include, if the feedback includes data in which the incorrect part of the response data has been corrected, setting the corrected data as true response data, setting the response data as false response data, and obtaining location information based on the location of the incorrect part.

[0010] According to one embodiment, the location information may include a predetermined token inserted to indicate a part that is the target of learning within the true response data and the false response data.

[0011] According to one embodiment, the location information may include a masking vector having the same dimensions as the response data, wherein the element value corresponding to the part targeted for learning is set differently from the element value of another part.

[0012] According to one embodiment, the step of training a machine learning model may include a step of restricting parameters of the machine learning model from being updated, excluding the part that is the target of training.

[0013] A computer-readable, non-transient recording medium may be provided that records instructions for executing a learning method of a model according to one embodiment of the present disclosure on a computer.

[0014] According to one embodiment of the present disclosure, an electronic device comprises a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program comprises instructions for acquiring input data, generating response data for the input data based on a machine learning model, acquiring feedback on the response data, generating a training data set based on the feedback, and training a machine learning model using the training data set, and the training data set may include true response data, false response data, and location information of a part of the response data that is a target for learning.

[0015] According to one embodiment, at least one program may include instructions for setting response data as false response data when feedback includes data indicating negation, generating true response data by modifying the incorrect part of the response data based on a language model, and obtaining location information based on the location of the incorrect part.

[0016] According to one embodiment, at least one program may include instructions for setting response data as true response data when feedback includes data indicating a positive response, changing a part of the response data to generate false response data, and obtaining location information based on the location of the changed part.

[0017] According to one embodiment, at least one program may include instructions for obtaining location information based on the location of the incorrect part, and setting the modified data as true response data and setting the response data as false response data when the feedback includes data in which the incorrect part of the response data has been modified.

[0018] According to one embodiment, at least one program may include instructions for restricting parameters of a machine learning model from being updated, excluding the part that is the target of learning.

[0019] According to some embodiments of the present disclosure, a training dataset suitable for target learning is generated based on feedback on the response, and by training a model using the generated training dataset, overfitting and errors in the model can be prevented, and the model can be trained to be optimized for the functions desired by the user.

[0020] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims.

[0021] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0022] FIG. 1 is a diagram illustrating an exemplary electronic device for learning a model according to one embodiment of the present disclosure.

[0023] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals in relation to the learning of a model according to one embodiment of the present disclosure.

[0024] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0025] FIG. 4 is a drawing for explaining the configuration of an electronic device for learning a model according to one embodiment of the present disclosure.

[0026] FIG. 5 is a drawing for explaining a target learning method according to one embodiment of the present disclosure.

[0027] FIG. 6 is a drawing for explaining a method of obtaining user feedback according to one embodiment of the present disclosure.

[0028] FIG. 7 is a drawing for illustrating another method of obtaining user feedback according to one embodiment of the present disclosure.

[0029] FIG. 8 is a drawing for explaining a method of learning a model according to one embodiment of the present disclosure.

[0030] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0031] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0032] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0033] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0034] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0035] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0036] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Optical Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0037] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are used merely to distinguish one component from another, and the essence, order, or sequence of the said component is not limited by such terms.

[0038] Additionally, in the following embodiments, where it is stated that one component is 'connected', 'coupled', or 'joined' to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be 'connected', 'coupled', or 'joined' between each component.

[0039] Additionally, as used in the following embodiments, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0040] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0041] FIG. 1 is a diagram illustrating an exemplary electronic device (100) for learning a model according to one embodiment of the present disclosure. Referring to FIG. 1, the electronic device (100) for learning a model can learn a machine learning model (110) based on feedback (106) for response data (104). Here, the machine learning model (110) may include a language model (e.g., LLM (large language model)), a multi-modal model, etc. A multi-modal model may represent an artificial neural network model capable of identifying interactions or contexts between data and generating outputs by simultaneously processing and learning various types of data (e.g., image data, text data, audio data, video data, etc.).

[0042] Such an electronic device (100) may include a display, memory, and at least one processor. However, the configuration of the electronic device (100) is not limited thereto. According to various embodiments, the electronic device (100) may include at least one additional component in addition to the components described above. For example, the electronic device (100) may further include a communication circuit (or communication module) for communication with an external electronic device.

[0043] The display can provide information visually. For example, the display can display various content (e.g., text, images, audio, video, icons and / or symbols, etc.) to the user. As an example, the display can display input data (102) and response data (104) for the input data (102). As another example, the display can provide a user interface for receiving feedback (106) for the response data (104).

[0044] According to one embodiment, the display may include a touch screen and may receive touch, gesture, proximity, or hovering inputs, for example, using an electronic pen or a part of the user's body. Additionally, the display may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of a force generated by a touch, and may include a sensor circuit or a control circuit for controlling the sensor.

[0045] The memory may store various data used by at least one component (e.g., a processor) of the electronic device (100). The data may include, for example, input data or output data for software (or programs) and related instructions. The memory may include volatile memory or non-volatile memory.

[0046] The processor may be connected to a display and memory and configured to execute at least one computer-readable program contained in the memory. For example, the processor may execute software (or a program) to control at least one other component (e.g., a hardware or software component) of an electronic device (100) connected to the processor and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor may load instructions or data received from another component (e.g., a communication circuit) into volatile memory, process the instructions or data stored in volatile memory, and store the resulting data in non-volatile memory.

[0047] At least one program executed by the processor may include instructions associated with the training of the model. In the following description, the processor is described as performing a certain function, but this is for the convenience of explanation, and the function performed by the processor can be understood as the processor executing instructions included in at least one program stored in memory.

[0048] The processor can obtain input data (102). Then, the processor can generate response data (104) for the input data (102) based on a machine learning model (110). For example, the processor can obtain a natural language query from a user and generate a response to the natural language query based on LLM.

[0049] Then, the processor may obtain feedback (106) regarding the response data (104). Here, the feedback (106) may include data indicating a positive or negative evaluation of the response data (104), such as "correct" / "incorrect" or "preferred" / "disliked." Alternatively, the feedback (106) may include data in which the incorrect part (or disliked part) of the response data (104) has been corrected. For example, the user may provide feedback (106) regarding the response data (104) by directly correcting the incorrect part (or disliked part) of the response data (104). In this way, to obtain feedback (106) from the user, the processor may provide a user interface. For example, the processor may display an object corresponding to the response data (104) through a touchscreen display and receive a signal from the user corresponding to the feedback (106) regarding the response data (104).

[0050] Then, the processor can generate a training data set based on the feedback (106). For example, the processor can generate a training data set useful for target learning of the machine learning model (110) based on the feedback (106). Here, target learning refers to training the model primarily on local data required for the generation of output, clearly indicating the data that is the target of learning and ensuring that learning does not occur for other data. For example, target learning in this disclosure may refer to training the machine learning model (110) such that only the parameters involved in the incorrect part (or unfavorable part) of the response data (104) among the parameters of the machine learning model (110) are updated. Additionally, the training data set may include true response data, false response data, and location information of the part that is the target of learning in the response data (104). True response data, that is, true data (ground truth data), may represent output data intended to induce generation for the input data when training the machine learning model (110). In addition, false response data, i.e., false data (false data or hallucination data), may represent output data for suppressing generation for input data when training a machine learning model (110).

[0051] According to one embodiment, if the feedback (106) contains data indicating negation, the processor can set the response data (104) as false response data. Additionally, the processor can generate true response data by revising the incorrect part of the response data (104) based on a language model (e.g., LLM). Additionally, the processor can obtain location information of the part to be targeted for learning based on the location of the incorrect part of the response data (104). For example, even if the feedback (106) contains only data indicating a negative evaluation, the processor can generate a training data set suitable for target learning using a language model.

[0052] According to one embodiment, the processor may set the response data (104) as true response data if the feedback (106) includes data indicating a positive. Additionally, the processor may generate false response data by hallucinating a part of the response data (104). Additionally, the processor may obtain location information of the part that is the target of learning based on the location of the hallucinated part of the response data (104). For example, even if the feedback (106) includes only data indicating a positive evaluation, the processor may generate a training data set suitable for target learning using a language model. According to one embodiment, the process of generating false response data by hallucinating a part of the response data (104) may be performed by a user or as a result of computation to replace specific text tokens by a separately trained natural language processing model.

[0053] According to one embodiment, if the feedback (106) includes data in which the incorrect part of the response data (104) has been corrected, the processor can set the corrected data as true response data. Additionally, the processor can set the response data (104) as false response data. Additionally, the processor can obtain location information of the part that is the target of learning based on the location of the incorrect part of the response data (104). For example, if the user directly corrects the incorrect part of the response data (104) and provides feedback (106) on the response data (104), the processor can directly configure the feedback (106) as a learning data set.

[0054] According to one embodiment, location information of a part targeted for learning may include a predetermined token (e.g., "[CLS]") inserted within true response data and false response data to indicate the part targeted for learning. For example, when response data is "ABCDE" and the part targeted for learning is "D", the response data containing location information of the part targeted for learning may be "ABC[CLS]D[CLS]E".

[0055] According to one embodiment, the location information of the part targeted for learning may have the same dimensions as the response data (104) and may include a masking vector in which the element value corresponding to the part targeted for learning is set differently from the element value of another part. For example, when the size of the embedding vector is L as a result of embedding the response data (104), the masking vector may be a binary vector of size L. Specifically, for example, when the response data is "ABCDE" and the part targeted for learning is "D", the sequence length may be 5, and the masking vector may be expressed as "[0, 0, 0, 1, 0]" in which the fourth element value (e.g., "1") is set differently from the element value of another part (e.g., "0"). That is, the masking vector may be a masking vector that activates only the fourth token.

[0056] Then, the processor can train the machine learning model (110) using the generated training data set. For example, the processor can train the machine learning model (110) so that only the parameters involved in the part that is the target of training are updated using the generated training data set. At this time, the processor can use the generated training data set to restrict the parameters of the machine learning model (110) so that the parameters involved in other parts, excluding the part that is the target of training, are not updated.

[0057] According to one embodiment, the processor can train a machine learning model (110) using a loss function to suppress the generation of false response data for input data (102) and increase the generation of true response data. At this time, the processor can include only the preference for the part that is the target of learning in the loss function, thereby eliminating (or limiting) learning for the remaining parts excluding the part that is the target of learning. According to one embodiment, the processor can train the machine learning model (110) using KL divergence (Kullback-Leibler divergence) so that data that is not required to be learned is not trained.

[0058] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) is connected to communicate with a plurality of user terminals (210_1, 210_2, 210_3) in relation to the learning of a model according to one embodiment of the present disclosure. The information processing system (230) may include system(s) capable of providing services associated with the learning of the model. In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to the learning of the model, or one or more distributed computing devices and / or distributed databases based on cloud computing services. For example, the information processing system (230) may include separate systems (e.g., servers) for the learning of the model.

[0059] Services related to the learning of a model provided by the information processing system (230) may be provided to the user through applications installed on each of the plurality of user terminals (210_1, 210_2, 210_3), web browser applications, etc. The function of the electronic device for learning the model described with reference to FIG. 1 (e.g., the electronic device (100) of FIG. 1) may be performed by the operation of the user terminal (210_1, 210_2, 210_3), by the operation of the information processing system (230), or by the combined operation of the user terminal (210_1, 210_2, 210_3) and the information processing system (230).

[0060] Multiple user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) through a network (220). The network (220) can be configured to enable communication between the multiple user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) may be configured as a wired network such as Ethernet, Power Line Communication, telephone line communication devices and RS-serial communication, a mobile communication network, a Wireless LAN (WLAN), Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited and may include not only communication methods utilizing communication networks that the network (220) may include (e.g., mobile communication network, wired internet, wireless internet, broadcasting network, satellite network, etc.) but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0061] For example, multiple user terminals (210_1, 210_2, 210_3) can transmit a data processing request and a command associated with a user request for data processing to an information processing system (230) through a network (220), and the information processing system (230) can receive this.

[0062] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto, and the user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and running applications, etc. For example, user terminals may include smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. Additionally, FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing system (230) through a network (220), but is not limited thereto, and may be configured so that a different number of user terminals communicate with an information processing system (230) through a network (220).

[0063] FIG. 3 is a block diagram showing the internal configuration of a user terminal (210) and an information processing system (230) according to an embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing applications, etc. and capable of wired / wireless communication, and may include, for example, the mobile phone terminal (210_1), tablet terminal (210_2), PC terminal (210_3) of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data through the network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) through the input / output interface (318).

[0064] The memory (312, 332) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as ROM (read-only memory), a disk drive, an SSD (solid-state drive), or a flash memory. As another example, a permanent mass storage device such as ROM, an SSD, a flash memory, or a disk drive may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. Additionally, the memory (312, 332) may store an operating system and at least one program code (e.g., code for an application associated with learning a model).

[0065] These software components may be loaded from a computer-readable recording medium separate from memory (312, 332). This separate computer-readable recording medium may include a recording medium that can be directly connected to the user terminal (210) and the information processing system (230), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (312, 332) via a communication module (316, 336) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (312, 332) based on a computer program (e.g., an application related to model training) that is installed by files provided through a network (220) by developers or a file distribution system that distributes installation files for the application.

[0066] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a recording device such as memory (312, 332).

[0067] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system). For example, a request or data (e.g., a data processing request or data, etc.) generated by the processor (314) of the user terminal (210) according to program code stored in a recording device such as memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) can be received by the user terminal (210) through the communication module (336) and the network (220) via the communication module (316) of the user terminal (210).

[0068] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, or a mouse, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface (318) may be a means for interfacing with a device in which the configuration or function for performing input and output is integrated into one, such as a touchscreen. Although the input / output device (320) is depicted in FIG. 3 as not being included in the user terminal (210), it is not limited thereto and may be configured as a single device with the user terminal (210). Additionally, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interface (318, 338) is shown as an element configured separately from the processor (314, 334), but is not limited thereto, and the input / output interface (318, 338) may be configured to be included in the processor (314, 334).

[0069] The user terminal (210) and the information processing system (230) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the conventional technical components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. Additionally, the user terminal (210) may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that are generally included in a smartphone, and may be implemented to include various components such as an accelerometer, a gyroscope, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0070] According to one embodiment, the processor (314) of the user terminal (210) may be configured to operate an application or web browser application that provides a service associated with the training of a model. At this time, program code associated with the application may be loaded into the memory (312) of the user terminal (210). While the application is running, the processor (314) of the user terminal (210) may receive information and / or data provided from an input / output device (320) through an input / output interface (318) or receive information and / or data from an information processing system (230) through a communication module (316), and may process the received information and / or data and store it in the memory (312). Additionally, such information and / or data may be provided to the information processing system (230) through the communication module (316).

[0071] While the application is running, the processor (314) may receive voice data, text, images, videos, etc., that are input or selected through an input device such as a touch screen, keyboard, audio sensor and / or image sensor, camera, microphone, etc., connected to an input / output interface (318), and may store the received voice data, text, images and / or videos, etc. in memory (312) or provide them to an information processing system (230) through a communication module (316) and a network (220). In one embodiment, the processor (314) may receive user input input through an input device and provide data / requests corresponding to the received user input to an information processing system (230) through a network (220) and a communication module (316).

[0072] The processor (314) of the user terminal (210) can transmit information and / or data to an input / output device (320) through an input / output interface (318) and output it. For example, the processor (314) of the user terminal (210) can output the processed information and / or data through an output device (320), such as a display output device (e.g., touch screen, display, etc.) or a voice output device (e.g., speaker).

[0073] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals (210) and / or a plurality of external systems. The information and / or data processed by the processor (334) may be provided to the user terminals (210) through a communication module (336) and a network (220).

[0074] FIG. 4 is a diagram illustrating the configuration of an electronic device (100) for learning a model according to one embodiment of the present disclosure. Referring to FIG. 4, the electronic device (100) for learning a model may include a chatbot module (410), a feedback acquisition module (420), a learning data set generation module (430), and a model learning module (440). However, the types of components included in the electronic device (100) (e.g., chatbot module (410), feedback acquisition module (420), learning data set generation module (430), and model learning module (440)) are classified according to functions related to learning the model, and the types and number thereof are not limited thereto. Additionally, the components included in the electronic device (100) may be implemented by at least one of software elements or hardware elements. For example, at least one of the components included in the electronic device (100) may be implemented in the form of an instruction stored in memory, and the instruction may be executed by a processor.

[0075] The chatbot module (410) can process user input (e.g., natural language query) and generate an appropriate response accordingly. For example, the chatbot module (410) can acquire input data and generate and output response data for the input data. To do this, the chatbot module (410) can provide a user interface. Additionally, the chatbot module (410) can perform natural language understanding (NLU) functions to analyze sentences entered by the user and understand their meaning, conversation management functions to manage the flow of conversation and track the state of interaction with the user, and response generation functions to generate an appropriate response to the user's questions or requests.

[0076] The feedback acquisition module (420) can acquire user feedback on response data. To do this, the feedback acquisition module (420) can provide a user interface. The user interfaces provided by the chatbot module (410) and the feedback acquisition module (420) may be the same or different. For example, the chatbot module (410) and the feedback acquisition module (420) may provide a user interface by linking or interlocking. Through the user interface, the feedback acquisition module (420) can acquire data indicating a positive or negative evaluation, or acquire data in which the incorrect parts (or unfavorable parts) of the response data have been corrected.

[0077] The training data set generation module (430) can generate a training data set suitable for target learning based on feedback obtained through the feedback acquisition module (420). Here, the training data set may include true response data, false response data, and location information of the part of the response data that is the target of learning.

[0078] According to one embodiment, the training data set generation module (430) can set the response data as false response data if the feedback obtained through the feedback acquisition module (420) includes data indicating negation. Additionally, the training data set generation module (430) can generate true response data by correcting the incorrect part of the response data based on a language model (e.g., LLM). Additionally, the training data set generation module (430) can obtain location information of the part to be targeted for learning based on the location of the incorrect part of the response data.

[0079] According to one embodiment, the learning data set generation module (430) can set the response data as true response data if the feedback obtained through the feedback acquisition module (420) includes data indicating a positive response. Additionally, the learning data set generation module (430) can generate false response data by changing a part of the response data. Additionally, the learning data set generation module (430) can obtain location information of the part that becomes the target of learning based on the location of the changed part of the response data.

[0080] According to one embodiment, the learning data set generation module (430) can set the modified data as true response data if the feedback obtained through the feedback acquisition module (420) includes data in which the incorrect part of the response data has been modified. Additionally, the learning data set generation module (430) can set the response data as false response data. Additionally, the learning data set generation module (430) can obtain location information of the part that is the target of learning based on the location of the incorrect part of the response data.

[0081] The model learning module (440) can train a machine learning model (e.g., the machine learning model (110) of FIG. 1) using the generated training data set. Here, the machine learning model may include a language model (e.g., LLM), a multimodal model, etc. The model learning module (440) can train the machine learning model so that only the parameters involved in the part that is the target of training are updated using the generated training data set. At this time, the model learning module (440) can restrict the machine learning model so that parameters involved in parts other than the part that is the target of training are not updated using the generated training data set.

[0082] FIG. 5 is a diagram illustrating a target learning method according to an embodiment of the present disclosure. Referring to FIG. 5, a processor of an electronic device for learning a model (e.g., the electronic device (100) of FIG. 1 and FIG. 4) may generate a learning data set for target learning. The learning data set may include true response data (520), false response data (510), and location information of a part that is the target of learning. For example, when the true response data (520) is "King Sejong the Great is the 4th king of Joseon" and the false response data (510) is "King Sejong the Great is the 8th king of Goryeo", the location of the token (522) corresponding to "Joseon" in the true response data (520) is the same as the location of the token (512) corresponding to "Goryeo" in the false response data (510), and information regarding that location may be set as location information of the part that is the target of learning. Likewise, the position of the token (524) corresponding to "4" in the true response data (520) is the same as the position of the token (514) corresponding to "8" in the false response data (510), and information about that position can also be set as position information of the part that is the target of learning.

[0083] According to one embodiment, the processor can selectively train only specific parts of the true response data (520) and false response data (510) rather than the whole, based on selective language modeling (SLM) (530). For example, through selective language modeling (530), the processor can exclude the remaining parts of the true response data (520) and false response data (510) other than the parts that are the target of training (e.g., the token corresponding to "Joseon" (522), the token corresponding to "Goryeo" (512), the token corresponding to "4" (524), and the token corresponding to "8" (514)). Additionally, the processor may exclude the remaining parts from the true response data (520) other than the parts targeted for learning (e.g., the token (522) corresponding to "Joseon" and the token (524) corresponding to "4"), and exclude the remaining parts from the false response data (510) other than the parts targeted for learning (e.g., the token (512) corresponding to "Goryeo" and the token (514) corresponding to "8"), and then compare them to train a machine learning model (500). According to one embodiment, the processor may train the machine learning model (500) using KL divergence, thereby ensuring that data for which learning is not required is not trained.

[0084] FIG. 6 is a diagram illustrating a method for obtaining user feedback according to an embodiment of the present disclosure. Referring to FIG. 6, a processor of an electronic device for learning a model (e.g., the electronic device (100) of FIG. 1 and FIG. 4) may obtain input data through a chatbot system (e.g., the chatbot module (410) of FIG. 4) and generate and output response data for the obtained input data. For example, through a user interface (600) provided by the chatbot system, the processor may receive user input (610) and provide a response (620) for the received input (610).

[0085] Then, the processor can obtain user feedback on the response data. To do this, the processor can output an object (630) requesting user feedback on the response (620) provided by the chatbot system through the user interface (600). For example, the processor can display a visual object (632) representing a positive evaluation and a visual object (634) representing a negative evaluation on the user interface (600). At this time, the processor can place the visual object (632) representing the positive evaluation and the visual object (634) representing the negative evaluation adjacent to the response (620) provided by the chatbot system.

[0086] Then, when a user selects either a visual object (632) representing a positive evaluation or a visual object (634) representing a negative evaluation, the processor can generate feedback data corresponding to the selected visual object. For example, the processor can include positive data in the feedback data in response to receiving user input selecting a visual object (632) representing a positive evaluation. Additionally, the processor can include negative data in the feedback data in response to receiving user input selecting a visual object (634) representing a negative evaluation.

[0087] Then, the processor can generate a training dataset based on the feedback data. For example, if the feedback data contains data indicating negativity, the processor can set the response data as false response data, modify the incorrect part of the response data based on a language model (e.g., LLM) to generate true response data, and generate location information of the target part for learning based on the location of the incorrect part of the response data. As another example, if the feedback data contains data indicating positivity, the processor can set the response data as true response data, change a part of the response data to generate false response data, and generate location information of the target part for learning based on the location of the changed part of the response data.

[0088] FIG. 7 is a diagram illustrating another method for obtaining user feedback according to one embodiment of the present disclosure. Referring to FIG. 7, a processor of an electronic device for learning a model (e.g., the electronic device (100) of FIG. 1 and FIG. 4) may obtain input data through a chatbot system (e.g., the chatbot module (410) of FIG. 4) and generate and output response data for the obtained input data. For example, through a user interface (700) provided by the chatbot system, the processor may receive user input (710) and provide a response (720) for the received input (710).

[0089] Then, the processor can obtain user feedback on the response data. To do this, the processor can output an object (730) requesting user feedback on the response (720) provided by the chatbot system through the user interface (700). For example, the processor can display an object (730) (e.g., a button object) for modifying the response (720) provided by the chatbot system on the user interface (700). At this time, the processor can place the object (730) for modifying the response (720) adjacent to the response (720) provided by the chatbot system.

[0090] Then, when the user selects an object (730) to modify the response (720), the processor may provide an object (740) (e.g., a text field) that can modify the response (720). Then, when the user modifies the response (720) through the object (740) that can modify the response (720), feedback data can be generated with the modified data. For example, the user can provide feedback on the response (720) by directly modifying the incorrect parts (or parts they do not prefer) of the response (720).

[0091] Then, the processor can generate a training data set based on the feedback data. For example, if the feedback data contains corrected data for the incorrect part of the response data, the processor can set the corrected data as the true response data and the response data as the false response data, and generate location information of the target part for learning based on the location of the incorrect part of the response data.

[0092] FIG. 8 is a diagram illustrating a method for training a model according to one embodiment of the present disclosure. Referring to FIG. 8, a processor of an electronic device for training a model (e.g., the electronic device (100) of FIG. 1 and FIG. 4) may acquire input data in step 810 (S810). For example, the processor may receive user input (e.g., natural language query) through a chatbot system (e.g., the chatbot module (410) of FIG. 4).

[0093] In step 820 (S820), the processor may generate response data for input data based on a machine learning model (e.g., the machine learning model (110) of FIG. 1 or the machine learning model (500) of FIG. 5). For example, the processor may receive a natural language query from a user through a chatbot system, generate a response to the natural language query based on LLM, and output the response generated through the chatbot system.

[0094] In step 830 (S830), the processor may obtain feedback on the response data. For example, the processor may receive user feedback on the response data through a user interface (e.g., a chatbot system). Here, the feedback may include data indicating a positive or negative evaluation. Additionally, the feedback may include data in which the incorrect part (or the unfavorable part) of the response data has been corrected.

[0095] In step 840 (S840), the processor may generate a training dataset based on feedback. For example, the processor may generate a training dataset useful for target learning of a machine learning model based on feedback. Here, the training dataset may include true response data, false response data, and location information of the part of the response data that is the target of learning. For example, if the feedback contains data indicating negativity, the processor may set the response data as false response data, modify the incorrect part of the response data based on a language model (e.g., LLM) to generate true response data, and obtain location information of the part that is the target of learning based on the location of the incorrect part of the response data. For another example, if the feedback contains data indicating positivity, the processor may set the response data as true response data, change a part of the response data to generate false response data, and obtain location information of the part that is the target of learning based on the location of the changed part of the response data. As another example, if the feedback contains corrected data for the incorrect part of the response data, the processor sets the corrected data as the true response data and the response data as the false response data, and can obtain location information of the target part for learning based on the location of the incorrect part of the response data.

[0096] According to one embodiment, the location information of the part targeted for learning may include a predetermined token (e.g., "[CLS]") inserted within true response data and false response data to indicate the part targeted for learning. According to one embodiment, the location information of the part targeted for learning may include a masking vector having the same dimensions as the response data, wherein the element value corresponding to the part targeted for learning is set differently from the element value of other parts.

[0097] In step 850 (S850), the processor can train a machine learning model using a training dataset. For example, the processor can train the machine learning model using the generated training dataset so that only the parameters involved in the target part of the training are updated. At this time, the processor can constrain the machine learning model so that parameters involved in parts other than the target part of the training are not updated using the generated training dataset.

[0098] According to one embodiment, a processor can train a machine learning model using a loss function to suppress the generation of false response data for input data and increase the generation of true response data. In this case, the processor can include only the preference for the part targeted for training in the loss function, thereby eliminating (or limiting) training for the remaining parts excluding the part targeted for training. According to one embodiment, the processor can train a machine learning model using KL divergence, thereby inducing that data for which training is not required is not trained.

[0099] The above-described flowchart and description are merely examples and may be implemented differently in some embodiments. For instance, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0100] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0101] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0102] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0103] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0104] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0105] When implemented in software, the techniques described above may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available media accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0106] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disks and discs include CDs, laser discs, optical discs, DVDs (digital versatile discs), floppy disks, and Blu-ray discs, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0107] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0108] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0109] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

1. A method for training a model, performed by at least one processor, Step of acquiring input data; A step of generating response data for the input data based on a machine learning model; A step of obtaining feedback on the above response data; Based on the above feedback, a step of generating a training data set; and A step of training the machine learning model using the above training data set Includes, The above training dataset is, A method for training a model, comprising true response data, false response data, and location information of a part of the response data that is the target of learning.

2. In Paragraph 1, The step of generating the above training data set is, If the above feedback includes data indicating negativity, the step of setting the response data as the false response data; Based on a language model, a step of generating true response data by correcting the incorrect parts of the response data; and A step of obtaining the location information based on the location of the above incorrect answer part A model training method including 3. In Paragraph 1, The step of generating the above training data set is, If the above feedback includes data indicating a positive response, the step of setting the response data as the true response data; A step of generating the false response data by changing a part of the above response data; and A step of obtaining the location information based on the location of the above-mentioned changed part A model training method including 4. In Paragraph 1, The step of generating the above training data set is, If the above feedback includes data in which the incorrect part of the above response data has been corrected, the step of setting the corrected data as the above true response data; Step of setting the above response data to the above false response data; and A step of obtaining the location information based on the location of the above incorrect answer part A model training method including 5. In Paragraph 1, The above location information is, A method for training a model comprising a predetermined token inserted to indicate a target portion of the learning within the true response data and the false response data.

6. In Paragraph 1, The above location information is, A method for training a model, comprising a masking vector having the same dimensions as the above response data, wherein the element value corresponding to the part targeted for learning is set differently from the element value of another part.

7. In Paragraph 1, The step of training the above machine learning model is, A step of constraining parameters of the machine learning model so that parameters involved in parts other than the target part of the learning are not updated, excluding the part that is the target of the learning. A model training method including 8. A computer-readable, non-transient recording medium recording instructions for executing the method according to paragraph 1 on a computer.

9. In electronic devices, Memory; and It includes at least one processor connected to the memory and configured to execute at least one computer-readable program contained in the memory, and The above at least one program is, Acquire input data, and Based on a machine learning model, response data for the above input data is generated, and Obtain feedback on the above response data, and Based on the above feedback, generate a training dataset, and Includes instructions for training the machine learning model using the above training data set, and The above training dataset is, An electronic device comprising true response data, false response data, and location information of a part of the response data that is the target of learning.

10. In Paragraph 9, The above at least one program is, If the above feedback includes data indicating negation, the above response data is set as the above false response data, and Based on a language model, the incorrect parts of the above response data are corrected to generate the above true response data, and An electronic device comprising instructions for obtaining location information based on the location of the above incorrect answer part.

11. In Paragraph 9, The above at least one program is, If the above feedback includes data indicating a positive response, the above response data is set as the above true response data, and Generates the false response data by modifying a part of the above response data, and An electronic device comprising instructions for obtaining position information based on the position of the above-mentioned changed portion.

12. In Paragraph 9, The above at least one program is, If the above feedback includes data in which the incorrect part of the above response data has been corrected, the above corrected data is set as the above true response data, and Set the above response data as the above false response data, and An electronic device comprising instructions for obtaining location information based on the location of the above incorrect answer part.

13. In Paragraph 9, The above location information is, An electronic device comprising a predetermined token inserted to indicate a target portion of the learning within the true response data and the false response data.

14. In Paragraph 9, The above location information is, An electronic device comprising a masking vector having the same dimensions as the above response data, wherein the element value corresponding to the part targeted for learning is set differently from the element value of another part.

15. In Paragraph 9, The above at least one program is, An electronic device comprising instructions for restricting parameters of the machine learning model such that parameters involved in parts other than the target part of the learning are not updated.

Images