Electronic device for providing user-optimized coding assistant, and operation method thereof

Abstract

The present disclosure provides an electronic device. The electronic device may: receive a first user prompt via a coding assistant that understands a user prompt by using a large language model (LLM) and generates a response to the user prompt on the basis of a code generation model; select a code generation model adapter via task clustering for the first user prompt; generate a response to the first user prompt on the basis of the code generation model to which the code generation model adapter is applied; acquire a user evaluation of the response; acquire a quality evaluation according to a result of analyzing the response on the basis of the LLM; store the first user prompt, the response, the user evaluation, and the quality evaluation in a history; and perform training on at least one code generation model adapter related to the code generation model on the basis of data stored in the history.

Description

Electronic device providing a user-optimized coding assistant and method of operation thereof

[0001] The present disclosure relates to an electronic device that provides a coding assistant optimized for a user and a method of operating the same.

[0002] Coding assistants are based on technology that supports developers' coding tasks by utilizing artificial intelligence (AI) and machine learning (ML). Based on natural language processing (NLP) and code analysis algorithms, they can provide features such as code auto-completion, error detection, debugging, and code optimization. Representative examples include GitHub Copilot and ChatGPT. By learning from vast amounts of open-source code and programming documentation, these tools can understand the context of the code entered by developers and provide appropriate recommendations. Coding assistants can continuously improve their performance by utilizing large language models (LLM) and reinforcement learning. They may also include features such as analyzing code patterns to identify security vulnerabilities or suggesting refactoring. Furthermore, coding assistants can be specialized for specific programming languages ​​or development environments, offering high utility by increasing development productivity and reducing repetitive tasks.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] An electronic device according to one embodiment of the present disclosure comprises at least one processor including a display; a memory; and a processing circuit; wherein the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: receive a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; select a code generation model adapter through task clustering for the first user prompt; generate an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; obtain a user evaluation of the answer; obtain a quality evaluation based on the result of analyzing the answer based on the LLM; store the first user prompt, the answer, the user evaluation, and the quality evaluation in a history; and perform learning on at least one code generation model adapter related to the code generation model based on the data stored in the history.

[0005] A method of an electronic device according to one embodiment of the present disclosure may include: receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; selecting a code generation model adapter through task clustering for the first user prompt; generating an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; obtaining a user evaluation of the answer; obtaining a quality evaluation based on the result of analyzing the answer based on the LLM; storing the first user prompt, the answer, the user evaluation, and the quality evaluation in a history; and performing learning for at least one code generation model adapter related to the code generation model based on the data stored in the history.

[0006] A non-transitory storage medium storing at least one computer-readable instruction according to one embodiment of the present disclosure may include, when the instructions are executed by at least one processor including a processing circuit of an electronic device, the electronic device: receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; selecting a code generation model adapter through task clustering for the first user prompt; generating an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; obtaining a user evaluation of the answer; obtaining a quality evaluation based on the result of analyzing the answer based on the LLM; storing the first user prompt, the answer, the user evaluation, and the quality evaluation in a history; and performing learning on at least one code generation model adapter related to the code generation model based on the data stored in the history.

[0007] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be determined in various ways without departing from the scope and subject matter of this disclosure.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2 is an example of the use of a coding assistant of an electronic device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a schematic diagram illustrating the operation of a coding assistant of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to automatically generate code according to a user prompt.

[0012] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure acquiring feedback information regarding a response from a coding assistant.

[0013] FIG. 6 is an example of an electronic device according to one embodiment of the present disclosure evaluating the response of a coding assistant using an LLM.

[0014] FIG. 7 is a schematic diagram illustrating the learning operation of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 8 is a schematic diagram illustrating the inference operation of an electronic device according to one embodiment of the present disclosure.

[0016] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0017] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the technical features of this disclosure. For example, a component expressed in the singular form should be understood as a concept including singular or plural components unless the context clearly indicates only the singular form.

[0018] In the present disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed with the corresponding phrase, or all possible combinations thereof. The term “and / or” as used in the present disclosure should be understood to encompass any possible combination by one or more of the plurality of items listed with the corresponding term. Terms such as “first,” “second,” “first,” or “second” as used in the present disclosure may be used merely to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order).

[0019] Where it is stated that any (e.g., 1st) component is “coupled,” “connected,” “linked,” “coupled,” “supported,” “connected,” or “contacted” with or without the terms “functionally” or “communicationly,” it includes not only cases where the component is directly coupled, connected, linked, coupled, supported, or contacted with the other component, but also cases where it is indirectly coupled, connected, linked, coupled, supported, or contacted through a third component.

[0020] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. When a component is described as being located "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where another component exists between the two components.

[0021] The expression “configured to” as used in this disclosure may be appropriately substituted, depending on the context, for example, “suitable for,” “capable of,” “designed to,” “modified to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean only that which is “specially designed” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “device configured (or set) to perform A, B, and C” may mean a device dedicated to performing the said operation, or a general-purpose device capable of performing various operations including said operation.

[0022] Terms used in this disclosure, such as "upper side," "lower side," and "front-rear direction," are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0023] Although the description in this disclosure is centered on specific embodiments, this disclosure is not limited to such specific embodiments and should be understood to encompass all various modifications, equivalents, and / or substitutions of the various embodiments described in this disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0024] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0025] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0026] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0027] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0028] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0029] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0032] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0033] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0034] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0038] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0040] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or largescale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0043] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0045] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0048] FIG. 2 is an example of the use of a coding assistant of an electronic device according to one embodiment of the present disclosure.

[0049] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) may provide a coding assistant that assists in programming tasks (e.g., writing code, debugging) in response to a user's request. Referring to FIG. 2, the electronic device (101) may automatically generate and output Android-based button creation code (202) in response to a user prompt (201) of "Write code to create a button in the Android app" in a conversation window (210) with the coding assistant. For example, the button creation code may be generated as "class MainActivity : AppCompatActivity(), View.OnClickListener { override fun onCreate(savedInstanceState: Bundle?) ...(omitted)". This code creates a screen called MainActivity in the Android app and defines a class that has the functionality to handle buttons, etc., that the user can click on the screen. View.OnClickListener is an interface that can be used to handle click events of a view, such as a button click. onCreate is a method that is called when the activity is first launched to perform initialization tasks. In FIG. 2, only a portion of the button creation code (202) is shown for the purpose of creating a simple drawing, but the electronic device (101) can generate button creation code that operates and executes completely.

[0050] The electronic device (101) can receive feedback from the user on whether the automatically generated code was helpful. For example, if the automatically generated code is helpful, the user can enter a "good!" answer (203). Or, if the automatically generated code is not to the user's liking, the user can change the user prompt to re-enter the code they want. In this case, although it is not explicit feedback, if a user prompt of a similar purpose is re-entered, it can be inferred that the user's satisfaction with the code is low.

[0051] The electronic device (101) stores user prompts (201), automatically generated code (202) in response thereto, and user feedback (203) as evaluated history, and can enhance the coding assistant using the history accumulated over a certain period. By retraining the coding assistant from the user's evaluated history rather than using a fixed pre-trained model, it is possible to provide a service that suits the user's environment or preferences. The enhanced coding assistant can provide a code generation service optimized for the user based on user characteristics derived from the prompts of users using the coding assistant. For example, if multiple people working on a development project use the coding assistant and accumulate history to enhance the coding assistant, it can become a coding assistant that is gradually optimized for the people participating in the development project. In other words, when the coding assistant is optimized by reflecting user feedback (203), the likelihood of generating code in the style desired by the user increases.

[0052] FIG. 3 is a schematic diagram illustrating the operation of a coding assistant of an electronic device according to one embodiment of the present disclosure.

[0053] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can automatically generate code through a coding assistant in response to a user request. The user may request the coding assistant to perform tasks related to programming. For example, the user may request code generation, code explanation, code conversion, code modification, code optimization, or code comment generation. The coding assistant may include an LLM that understands the user request and a code generation model that generates target code based on the user request. Alternatively, the LLM may be included in the code generation model. The coding assistant may receive user feedback on the automatically generated code. For example, it may receive an evaluation score. The coding assistant may calculate a score evaluating the quality of the automatically generated code. As history information, the coding assistant may store user prompts, user ratings, and scores for the code together with the automatically generated code.

[0054] Referring to FIG. 3, the electronic device (101) can generate code in response to a user prompt input, receive a user evaluation of the generated code, assign a score to the quality of the code, and store such usage history in a DB. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0055] In operation 301, an electronic device (101) according to one embodiment may receive a user prompt. The electronic device (101) may receive the user prompt through an input device (e.g., keyboard, microphone). The user prompt may be a sentence in the form of natural language containing user requirements. For example, the electronic device (101) may receive prompts such as "Write code to make a button in the Android app" or "Explain how the following code works." Depending on the characteristics of how the user uses the coding assistant, the user request may be classified into several tasks. For example, the user request may be classified into one of code generation, code explanation, code language transformation, code bug fix, code refactoring, and code comment generation. This is merely an example, and the types of tasks may be changed or subdivided depending on the characteristics of how the user uses the coding assistant. For example, user requests may further include code debugging, test code generation, automatic documentation generation, or code auto-completion. User prompts may include the type of operation. The type of operation may be included in the user prompt in the form of a predefined prefix. The prefixes may be, for example, / generation, / explain, / transform, / fix, / refactoring, and / comments.

[0056] In operation 302, an electronic device (101) according to one embodiment may understand a user prompt based on LLM and generate a response to the user prompt using a code generation model (10). The code generation model (10) may be an AI-based learning model trained to perform one or more functions related to programming, such as generating code, converting code, debugging, analyzing and interpreting code, or generating comments for code. The code generation model (10) may automatically generate code when a user input prompt requests code generation. Alternatively, the code generation model (10) may modify the language of the original code when a user input prompt requests a code language modification.

[0057] In operation 311, an electronic device (101) according to one embodiment may transmit a user input prompt and an automatically generated code in response thereto to a task cluster to determine a code generation model adapter (20) for the task. The code generation model adapter (20) may enable a basic code generation model (10) to generate an answer that matches the user input prompt.

[0058] In operation 312, if the electronic device (101) according to one embodiment does not include a prefix designating a task type in the user input prompt, in operation 313, it may classify which task type the prompt and code correspond to using a task classifier and then transmit them to a task cluster. If the task type for the prompt is not identified, task clustering may be difficult because the prompt is too broad. Therefore, before task clustering, if there is no prefix for the task type in the prompt, the type can be specified using a task classifier.

[0059] 314 In operation, an electronic device (101) according to one embodiment can perform task clustering on user input prompts and generated code. Through task clustering, the electronic device (101) can group tasks by analyzing similarities between them. A task group may consist of tasks that have similarities within a task type.

[0060] In operation 315, an electronic device (101) according to one embodiment may select a code generation model adapter (20) corresponding to a task group based on the task clustering result for the corresponding prompt and code. For example, if the user prompt and generated code are clustered into the Nth task group, the Nth code generation model adapter may be selected.

[0061] The basic code generation model (10) generates an answer regardless of the task type of the prompt, but if the code generation model adapter (20) is activated, it can generate an answer optimized for the task of the prompt. A specific code generation model adapter (20) can be gradually optimized for a specific task group by being retrained based on the history included in the specific task group. There may be as many code generation model adapters (20) as there are task groups based on the task clustering results.

[0062] In operation 321, the electronic device (101) according to one embodiment may receive a user rating for the generated code. The user rating may be performed according to a quantitative evaluation method. For example, the user may assign a higher score to the automatically generated code as satisfaction increases, on a scale of 1 to 5 points.

[0063] In operation 322, if the electronic device (101) according to one embodiment does not receive a user evaluation for the generated code, in operation 323, it may request an evaluation from an annotator model (30). The annotator model (30) may have learned previous user evaluation data to infer user prompts and evaluations for the generated code.

[0064] In operation 323, an electronic device (101) according to one embodiment can rate the generated code using a task annotation model (30). The task annotation model (20) can learn the evaluation data of previous users and infer the evaluation of the automatically generated code by processing user prompts.

[0065] In operation 331, an electronic device (101) according to one embodiment may calculate a score for automatically generated code using a code scoring LLM (40). The automatically generated code may be evaluated directly by a user (operation 321) or evaluated by an annotation model (30) (operation 323). The code scoring LLM (40) may be trained to analyze the generated code and determine the quality of the code itself. The code scoring LLM (40) may calculate a code score for the input code by understanding the source code and analyzing how well it meets evaluation criteria. Evaluation criteria may include, for example, instruction following, code explanation, code complexity and efficiency, code readability, and coding style. The code scoring LLM (40) can obtain a final score for automatically generated code based on a code score that analyzes the quality of the code itself and a user rating.

[0066] In operation 341, an electronic device (101) according to one embodiment may store user input prompts, generated code, evaluations, and final scores in a rated history (50). Data stored in the history (50) may be used as retraining data for a code generation model (10) through each code generation model adapter (20).

[0067] An electronic device according to one embodiment of the present disclosure comprises at least one processor including a display; a memory; and a processing circuit; wherein the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: receive a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; select a code generation model adapter through task clustering for the first user prompt; generate an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; obtain a user evaluation of the answer; obtain a quality evaluation based on the result of analyzing the answer based on the LLM; store the first user prompt, the answer, the user evaluation, and the quality evaluation in a history; and perform learning on at least one code generation model adapter related to the code generation model based on the data stored in the history.

[0068] According to one embodiment, the code generation model may be trained to perform at least one of code generation, language conversion, debugging, analysis, explanation, comment generation, error correction, refactoring, optimization, or documentation.

[0069] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: identify whether the first user prompt contains a prefix indicating a task type before performing task clustering for the first user prompt, and, if the first user prompt does not contain the prefix, cause the first user prompt to be classified based on a task classification model.

[0070] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: perform task clustering by referencing the task type included in the first user prompt, identify a first task group to which the first user prompt belongs among one or more task groups composed of similar tasks based on the clustering result, and select the code generation model adapter corresponding to the first task group.

[0071] According to one embodiment, when the memory is executed individually or collectively by the at least one processor, the electronic device may: include a score entered according to a predetermined quantitative evaluation method for the user evaluation of the answer.

[0072] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: identify whether a user evaluation of the answer is entered within a certain time, and, if the user evaluation is not entered, infer an evaluation score for the answer based on an annotator model.

[0073] According to one embodiment, the annotation model may be learned based on user evaluation data directly input by the user accumulated through previous use.

[0074] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: obtain a quality evaluation based on the result of analyzing the answer based on the LLM, transmit a prompt requesting an evaluation of the answer to the LLM, and receive a predetermined quantitative evaluation score from the LLM.

[0075] According to one embodiment, the memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: the prompt to include information about evaluation criteria; and the LLM to include a model learned to understand the answer and analyze how well it meets the evaluation criteria to infer a score for the answer.

[0076] A method of an electronic device according to one embodiment of the present disclosure may include: receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; selecting a code generation model adapter through task clustering for the first user prompt; generating an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; obtaining a user evaluation of the answer; obtaining a quality evaluation based on the result of analyzing the answer based on the LLM; storing the first user prompt, the answer, the user evaluation, and the quality evaluation in a history; and performing learning for at least one code generation model adapter related to the code generation model based on the data stored in the history.

[0077] According to one embodiment, the code generation model may be trained to perform at least one of code generation, language conversion, debugging, analysis, explanation, comment generation, error correction, refactoring, optimization, or documentation.

[0078] According to one embodiment, before performing task clustering for the first user prompt, the method may further include: identifying whether the first user prompt contains a prefix indicating a task type; and, if the first user prompt does not contain the prefix, classifying the first user prompt based on a task classification model.

[0079] According to one embodiment, the operation of selecting a code generation model adapter through task clustering for the first user prompt may include: an operation of performing task clustering by referring to the type of task included in the first user prompt; an operation of identifying a first task group to which the first user prompt belongs among one or more task groups composed of similar tasks based on the clustering result; and an operation of selecting the code generation model adapter corresponding to the first task group.

[0080] According to one embodiment, the user evaluation of the answer may include a score entered according to a predetermined quantitative evaluation method.

[0081] According to one embodiment, the method may further include an operation of identifying whether a user evaluation of the answer is entered within a set time; and, if the user evaluation is not entered, an operation of inferring an evaluation score for the answer based on an annotator model.

[0082] According to one embodiment, the annotation model can be learned based on user evaluation data directly input by users accumulated through previous use.

[0083] According to one embodiment, the operation of obtaining a quality evaluation based on the result of analyzing the answer based on the LLM may include: the operation of transmitting a prompt to the LLM requesting an evaluation of the answer; and the operation of receiving a predetermined quantitative evaluation score from the LLM.

[0084] According to one embodiment, the prompt includes information about evaluation criteria, and the LLM may include a model trained to understand the answer and analyze how well it meets the evaluation criteria to infer a score for the answer.

[0085] A non-transitory storage medium storing at least one computer-readable instruction according to one embodiment of the present disclosure may be provided. The instruction may perform at least one operation when executed by at least one processor comprising a processing circuit of an electronic device. The at least one operation may include receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model. The at least one operation may include selecting a code generation model adapter through task clustering for the first user prompt. The at least one operation may include generating an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied. The at least one operation may include obtaining a user evaluation of the answer. The at least one operation may include obtaining a quality evaluation based on the result of analyzing the answer based on the LLM. The at least one operation may include storing the first user prompt, the answer, the user evaluation, and the quality evaluation in a history. At least one operation may include an operation of performing training on at least one code generation model adapter related to the code generation model based on data stored in the history.

[0086] According to one embodiment, the code generation model may be trained to perform at least one of code generation, language conversion, debugging, analysis, explanation, comment generation, error correction, refactoring, optimization, or documentation.

[0087] FIG. 4 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to automatically generate code according to a user prompt.

[0088] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can automatically generate code based on a coding assistant in response to a user input prompt. The coding assistant can perform one or more tasks related to programming. For example, the coding assistant can perform automatic code generation, code analysis, code modification, and code optimization. Accordingly, the user may request an explanation of the code, code language conversion, code bug fix, or code optimization. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0089] In operation 410, an electronic device (101) according to one embodiment may receive a user prompt. The user prompt may be a natural language keyword, query, or sentence. A coding assistant may understand the user prompt based on LLM. The user prompt may include content requesting a task related to programming. For example, the user prompt may request writing code.

[0090] In operation 420, an electronic device (101) according to one embodiment can identify whether there is a prefix in the user prompt. The user prompt may include a prefix that designates a type of task. The coding assistant can perform task clustering using the prefix within the prompt. Prefixes may be defined in correspondence with specific tasks primarily used by users of the coding assistant, and clustering may be performed on the coding assistant that operates through the prefix. A code generation model adapter (e.g., the code generation model adapter (20) of FIG. 3) can adapt a general code generation model (e.g., the code generation model (10) of FIG. 3) to a specific purpose or environment. The coding assistant may include an adapter that tunes the corresponding code generation model to derive a more suitable answer for each clustered task group. The electronic device (101) can perform task clustering if the user input prompt contains a prefix (operation 440).

[0091] In operation 430, the electronic device (101) according to one embodiment may classify tasks using a task classification model when there is no prefix in the user input prompt. After classifying tasks, the electronic device (101) may perform task clustering (operation 440).

[0092] In operation 440, an electronic device (101) according to one embodiment may perform task clustering to select a code generation model adapter (20). The code generation model adapter (20) may adapt a general code generation model to a task clustering group. The code generation model (10) may be a universally learned model used by an unspecified number of people. The coding assistant may optimize the general code generation model (10) by using the adapter (20) to reflect the usage history of specific users (e.g., multiple people participating in a single project, coding assistants used within an enterprise).

[0093] In operation 450, the electronic device (101) according to one embodiment may generate code for a user prompt using a code generation model (10) to which a selected code generation model adapter (20) is applied. Alternatively, the coding assistant may perform processing on the user prompt if the user prompt requests other tasks, such as code modification or language conversion. For example, in response to a user prompt of "Explain what the following code does," it may analyze and explain the code.

[0094] In operation 460, the electronic device (101) according to one embodiment can output the generated code through an output device (e.g., a display).

[0095] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure acquiring feedback information regarding a response from a coding assistant.

[0096] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) may store usage history information, including feedback on the results of using the coding assistant, in a DB (e.g., the history (50) of FIG. 3). The feedback may include a satisfaction score evaluated directly by the user or a quality analysis score for the response itself. The coding assistant may accumulate usage history and use it to retrain a code generation model (e.g., the code generation model (10) of FIG. 3). For example, in FIG. 4, the electronic device (101) may perform an operation (460 operation) of automatically generating and outputting code in response to a user prompt, followed by the operations of FIG. 5. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0097] In operation 510, the electronic device (101) according to one embodiment may receive user evaluations of automatically generated code. The user may enter an evaluation score indicating satisfaction with the automatically generated code. The evaluation score may be specified to be selected from a set quantitative value (e.g., 1 to 5 points). The user may be included in a specific group that shares the coding assistant. For example, it may be developers participating in a development project or employees using the in-house coding assistant.

[0098] In operation 520, if the electronic device (101) according to one embodiment does not receive an evaluation score from the user, it can evaluate automatically generated code using an annotation model (e.g., the annotation model (30) of FIG. 3). The annotation model (30) may be trained with score data directly evaluated by the user.

[0099] In operation 530, an electronic device (101) according to one embodiment can calculate a score of a code using a code scoring LLM (e.g., the code scoring LLM (40) of FIG. 3). The code scoring LLM (40) can determine the quality of the generated code using a separate LLM. For example, the code scoring LLM (40) can calculate a score for the code for each item of instruction following, code explanation, code complexity and efficiency, code readability, and coding style. The code scoring LLM (40) can obtain an evaluation score by prompting a separate LLM with an explanation of the score calculation criteria and evaluation content for the code.

[0100] In operation 540, the electronic device (101) according to one embodiment may calculate a final score by reflecting user evaluations and code quality scores. For example, the coding assistant may calculate the final score as the average value of the evaluation score obtained through the user or the comment model (30) and the code score evaluated for code quality. Alternatively, the ratio between the evaluation score and the code score may vary depending on the type of task. In the case of code generation, the weight of the code score may be increased, and conversely, in the case of code explanation, the weight of the evaluation score may be increased.

[0101] In operation 550, the electronic device (101) according to one embodiment may store the prompt, generated code, evaluation, and final score in a history DB (e.g., evaluation completed history (50) of FIG. 3).

[0102] FIG. 6 is an example of an electronic device according to one embodiment of the present disclosure evaluating the response of a coding assistant using an LLM.

[0103] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can evaluate the quality of the answer (e.g., automatically generated code, code explanation) itself that the coding assistant has generated in response to a user input prompt. The electronic device (101) may, for example, request an evaluation of the automatically generated code explanation from a separate LLM using an evaluation request prompt from a code scoring LLM (e.g., the code scoring LLM (40) of FIG. 3). On the execution screen (610) of the coding assistant, the code scoring LLM (40) may input a prompt requesting an evaluation of the coding assistant's answer. The prompt may request an evaluation of the clarity and depth of the explanation of the code segment and may present evaluation criteria and evaluation scores. Evaluation criteria may include clarity, depth, relevance, and accessibility. Evaluation scores may be defined, for example, as insufficient (1 point), basic (2 points), average (3 points), very good (4 points), and excellent (5 points) for each evaluation criterion. The electronic device (101) can obtain a quality score for the code by using a prompt that includes an explanation of the evaluation criteria and criteria for calculating the evaluation score in the LLM. For example, regarding the evaluation criteria in the prompt, it may be explained to evaluate: 1. Clarity, how easy the explanation is to understand; 2. Depth, whether the explanation sufficiently covers the logic, structure, and decisions of the code; 3. Relevance, whether the explanation is appropriately connected to the purpose and design philosophy of the code; and 4. Accessibility, whether the explanation can be understood by a broad public regardless of their technical background.In addition, regarding the evaluation scores in the prompt, 1) Insufficient corresponds to cases where the explanation is unclear, superficial, or missing; 2) Basic corresponds to cases where the explanation covers basic points but lacks depth or clarity; 3) Average corresponds to cases where the explanation is clear and covers some details but may miss deeper insights; 4) Very Good corresponds to cases where the explanation is clear and detailed and covers most logic, structure, and decisions well; and 5) Excellent corresponds to cases where the explanation is very clear and in-depth and makes the purpose, design, and logic of the code easy to understand for all users.

[0104] Alternatively, the electronic device (101) may define criteria and evaluation scores for judging the quality of the language conversion when the automatically generated code is a code language conversion. The criteria and scores for judging the output result in response to a user prompt may be defined differently depending on the purpose of using the coding assistant, etc.

[0105] FIG. 7 is a schematic diagram illustrating the learning operation of an electronic device according to one embodiment of the present disclosure.

[0106] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can accumulate a usage history for a coding assistant and perform retraining to optimize each code generation model adapter (e.g., the code generation model adapter (20) of FIG. 3). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0107] In operation 710, an electronic device (101) according to one embodiment receives input from a user including a request to generate code to a code generation model (10) through a coding assistant, and in operation 720, the code generation model (10) can automatically generate code as an answer to the prompt.

[0108] In operation 730, an electronic device (101) according to one embodiment may receive an evaluation of an automatically generated code from a user. Data (701) directly evaluated by the user may be included in the history (50), but may be stored and managed separately.

[0109] In operation 740, an electronic device (101) according to one embodiment can train an annotation model (30) based on data directly evaluated by a user. By learning the user direct evaluation data, the annotation model (30) can infer an evaluation of a prompt-automatically generated code (answer) in which the user evaluation is missing.

[0110] In operation 750, an electronic device (101) according to one embodiment can select a code generation model adapter (20) through task classification / clustering for user prompts and automatically generated code.

[0111] In operation 760, an electronic device (101) according to one embodiment can activate a selected adapter (20) and perform retraining on a code generation model (10) based on data accumulated in a history (50) that has been evaluated. Through retraining, the code generation model adapter (20) can have parameter values ​​optimized for the corresponding task group set. In the inference process, when a user prompt corresponding to the task group is input, the code generation model adapter (20) corresponding to the task group is activated so that the code generation model (10) generates an answer, thereby generating an answer optimized for the task of the user prompt. Although there is only one code generation model (10), it can generate answers optimized for each of various user prompts by using adapters (20) specialized for multiple tasks.

[0112] FIG. 8 is a schematic diagram illustrating the inference operation of an electronic device according to one embodiment of the present disclosure.

[0113] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) can, upon receiving a user input prompt, select a code generation model adapter after task clustering and generate an answer through the code generation model. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0114] In operation 810, an electronic device (101) according to one embodiment can receive prompts from a user to a code generation model (10) through a coding assistant.

[0115] In operation 820, an electronic device (101) according to one embodiment may transmit a prompt to a task cluster that includes a prefix indicating a task type.

[0116] In operation 812, if the electronic device (101) according to one embodiment does not include a prefix in the prompt, the task type can be determined through the task classification model (813).

[0117] In operation 830, an electronic device (101) according to one embodiment can perform task clustering for user prompts. The task cluster can cluster user prompts to group user prompts into task groups that include tasks similar to user prompts among the task groups.

[0118] In operation 840, an electronic device (101) according to one embodiment can select a code generation model adapter (20) corresponding to a task group containing a user prompt.

[0119] In operation 850, an electronic device (101) according to one embodiment can activate a selected code generation model adapter (20) and generate an answer to a user input prompt based on a code generation model (10). The answer may be, for example, a code generation.

[0120] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0121] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0122] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0123] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0124] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; Memory comprising at least one storage medium for storing instructions; and It includes at least one processor including a processing circuit; When the above instructions are executed individually or collectively by the at least one processor, the electronic device: A first user prompt is received through a coding assistant that understands the user prompt using a massive language model (LLM) and generates an answer to the said user prompt based on a code generation model, and Select a code generation model adapter through task clustering for the above first user prompt, and Based on the code generation model to which the code generation model adapter is applied, an answer to the first user prompt is generated, and Obtain user ratings for the above answer, and Obtain a quality evaluation based on the results of analyzing the above answer based on LLM, and The above first user prompt, the above answer, the above user rating, and the above quality rating are stored in history, and An electronic device that causes learning to be performed on at least one code generation model adapter related to the code generation model based on data stored in the above history.

2. In Paragraph 1, The above code generation model is an electronic device trained to perform at least one of code generation, language conversion, debugging, analysis, explanation, comment generation, error correction, refactoring, optimization, or documentation.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Before performing task clustering for the first user prompt, identify whether the first user prompt includes a prefix indicating a task type, and An electronic device that causes the first user prompt to be classified based on a task classification model when the first user prompt does not include the prefix.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The operation of selecting a code generation model adapter through task clustering for the first user prompt above is, Task clustering is performed by referring to the task types included in the first user prompt above, and Based on the clustering result, identify the first task group to which the first user prompt belongs among one or more task groups composed of similar tasks, and An electronic device that causes the selection of the code generation model adapter corresponding to the first task group.

5. In any one of paragraphs 1 through 4, The above user evaluation of the above answer is an electronic device comprising a score entered according to a predetermined quantitative evaluation method.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Identify whether a user evaluation for the above answer is entered within a certain period of time, and An electronic device that causes to infer an evaluation score for the answer based on an annotator model when the above user evaluation is not entered.

7. In any one of paragraphs 1 through 6, The above annotation model is an electronic device that learns based on user evaluation data directly input by users accumulated through previous use.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The operation of obtaining a quality evaluation based on the results of analyzing the above answer based on the above LLM is, Send a prompt to the above LLM requesting an evaluation of the above answer, and An electronic device that causes to receive a predetermined quantitative evaluation score from the above LLM.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: The above prompt includes information regarding evaluation criteria, and An electronic device that causes the above LLM to include a model trained to understand the above answer and analyze how well it meets evaluation criteria to infer a score for the above answer.

10. In a method of an electronic device, An operation of receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; The operation of selecting a code generation model adapter through task clustering for the first user prompt above; An operation to generate an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; An action of obtaining user evaluations for the above answer; An operation to obtain a quality evaluation based on the results of analyzing the above answer based on LLM; The operation of saving the above first user prompt, the above answer, the above user evaluation, and the above quality evaluation to history; and A method comprising the operation of performing training on at least one code generation model adapter related to the code generation model based on data stored in the history above.

11. In Paragraph 10, The above code generation model is a method trained to perform at least one of code generation, language conversion, debugging, analysis, explanation, comment generation, error correction, refactoring, optimization, or documentation.

12. In Paragraph 10 or 11, Before performing task clustering for the first user prompt, an operation to identify whether the first user prompt includes a prefix indicating a task type; and A method further comprising, when the prefix is ​​not included in the first user prompt, classifying the first user prompt based on a task classification model.

13. In any one of paragraphs 10 through 12, The operation of selecting a code generation model adapter through task clustering for the first user prompt above is, An operation to perform task clustering by referring to the task types included in the first user prompt above; An operation to identify, based on clustering results, a first task group to which the first user prompt belongs among one or more task groups composed of similar tasks; and A method comprising the operation of selecting the code generation model adapter corresponding to the first task group.

14. In any one of paragraphs 10 through 13, A method in which the user evaluation of the above answer includes a score entered according to a predetermined quantitative evaluation method.

15. A non-transitory storage medium storing at least one computer-readable instruction, When the above instructions are executed by at least one processor including a processing circuit of an electronic device, the electronic device: An operation of receiving a first user prompt through a coding assistant that understands the user prompt using a large language model (LLM) and generates an answer to the user prompt based on a code generation model; The operation of selecting a code generation model adapter through task clustering for the first user prompt above; An operation to generate an answer to the first user prompt based on the code generation model to which the code generation model adapter is applied; An action of obtaining user evaluations for the above answer; An operation to obtain a quality evaluation based on the results of analyzing the above answer based on LLM; The operation of saving the above first user prompt, the above answer, the above user evaluation, and the above quality evaluation to history; and A storage medium comprising an operation of performing training on at least one code generation model adapter related to the code generation model based on data stored in the above history.

Images