Method for generating feedback information for problem solution

Abstract

Disclosed, according to one embodiment of the present disclosure, is a method, performed by a computing device, for generating feedback information for a problem solution. The method may comprise the steps of: obtaining model answer information items related to a problem; generating reference answer information items on the basis of the model answer information items by using an artificial intelligence model; obtaining user input information items related to a solution of the problem; and generating feedback information for any one of the user input information items by associating the user input information items with the reference answer information items.

Description

How to generate feedback information on problem solutions

[0001] The present invention relates to the field of educational support technology, and more specifically, to a technology that generates a natural language-based reference answer based on model answer information for a problem and provides real-time or step-by-step feedback information by matching it with the learner's solution information.

[0002] Most standard model answers are formula-centric and fail to adequately explain the learner's step-by-step thought process. Consequently, learners find it difficult to self-assess their solution process or identify where their thinking has stalled.

[0003] Furthermore, student answers feature various expression styles, changes in order, the use of units, and natural language descriptions. Conventional technologies often treat these differences in expression as incorrect answers, leading to a problem where they fail to accurately recognize answers that are logically identical but differ in notation.

[0004] Furthermore, while partial credit in mathematics assessments requires considering various criteria such as solution structure, thought process, and concept application, individually setting these standards is inefficient and lacks consistency. Most existing digital learning systems provide feedback only after the correct answer is submitted, and they suffer from a problem in that they lack features to immediately intervene or guide students to the next step even if they linger on the solution process for too long or proceed in the wrong direction.

[0005] Korean Patent Publication No. 10-2021-0094317 (July 29, 2021) discloses a method for automatically providing feedback learning content by utilizing math problem solving.

[0006] The present disclosure aims to provide a method for automatically generating natural language-based reference answer information based on model answer information and generating real-time context-based feedback information by step-by-step matching it with user solution information.

[0007] In addition, the present disclosure aims to provide a method for automatically generating and applying accurate and consistent scoring criteria through the classification of solution types, generation of a weight-based partial score system, and determination of logical identity considering differences in expression methods.

[0008] Meanwhile, the technical problem that the present disclosure aims to solve is not limited to the technical problem mentioned above, and various technical problems may be included within the scope obvious to a person skilled in the art from the contents described below.

[0009] According to one embodiment of the present disclosure for realizing the aforementioned task, a method for generating feedback information for the solution of a problem performed by a computing device is disclosed. The method may include the steps of: obtaining model answer information related to the problem; generating reference answer information based on the model answer information using an artificial intelligence model; obtaining user input information related to the solution of the problem; and generating feedback information for any one of the user input information by linking the user input information and the reference answer information.

[0010] In one embodiment, the step of generating the reference answer information based on the model answer information using the artificial intelligence model may include: a step of sequentially dividing the acquired model answer information; and a step of generating the reference answer information by adding natural language-based descriptive information to each of the sequentially divided model answer information using the artificial intelligence model.

[0011] In one embodiment, the step of generating feedback information for any one of the user input information by linking the user input information and the reference answer information may include: a step of sequentially dividing the acquired user input information; a step of matching the model answer information and the user input information; and a step of generating feedback information for any one of the user input information by utilizing the matching result and the reference answer information.

[0012] In one embodiment, the step of generating feedback information for any one of the user input information using the matching result and the reference answer information may include, if the pre-set time required is not satisfied, generating the feedback information to induce problem solving after the section where the model answer information and the user input information are matched using the reference answer information.

[0013] In one embodiment, the step of generating feedback information for any one of the user input information using the matching result and the reference answer information may further include: a step of generating feedback information by displaying a first highlighting on the user input information containing the condition when any one of the user input information contains a condition determined to be important in advance; and a step of generating feedback information by displaying a second highlighting on the user input information containing the condition when any one of the user input information does not contain a condition determined to be important in advance.

[0014] In one embodiment, the method may further include the step of generating a scoring criterion for the problem based on the model answer information.

[0015] In one embodiment, the step of generating a scoring criterion for the problem based on the model answer information may include: a step of classifying the solution type of each of the model answer information; and a step of generating a partial score system for each of the model answer information by considering the classified solution type.

[0016] In one embodiment, the step of determining a partial score system for each of the model answer information in consideration of the classified solution type may include the step of generating the partial score system by assigning a dynamically adjusted weight to each of the model answer information in consideration of the classified solution type.

[0017] In one embodiment, the step of generating a scoring criterion for the problem based on the model answer information may include: a step of analyzing the expression method included in the model answer information; and a step of generating the scoring criterion to recognize as the correct answer when the expression methods of the model answer information and the user input information are different from each other but have the same logical meaning, taking into account the analyzed expression method.

[0018] A computer program stored on a computer-readable storage medium is disclosed in accordance with one embodiment of the present disclosure for realizing the aforementioned objectives. When the computer program is executed on one or more processors, the one or more processors are configured to perform the following operations to generate feedback information regarding the solution of a problem, wherein the operations may include: an operation of obtaining model answer information related to the problem; an operation of generating reference answer information based on the model answer information using an artificial intelligence model; an operation of obtaining user input information related to the solution of the problem; and an operation of generating feedback information for any one of the user input information by linking the user input information and the reference answer information.

[0019] In one embodiment, the operation of generating the reference answer information based on the model answer information using the artificial intelligence model may include: the operation of sequentially dividing the acquired model answer information; and the operation of generating the reference answer information by adding natural language-based descriptive information to each of the sequentially divided model answer information using the artificial intelligence model.

[0020] In one embodiment, the operation of generating feedback information for any one of the user input information by linking the user input information and the reference answer information may include: the operation of sequentially dividing the acquired user input information; the operation of matching the model answer information and the user input information; and the operation of generating feedback information for any one of the user input information by utilizing the matching result and the reference answer information.

[0021] In one embodiment, the operation of generating feedback information for any one of the user input information using the matching result and the reference answer information may include, when the pre-set time required is not satisfied, generating the feedback information to induce problem solving after the section where the model answer information and the user input information are matched using the reference answer information.

[0022] In one embodiment, the operation of generating feedback information for any one of the user input information using the matching result and the reference answer information may further include: generating feedback information by displaying a first highlighting on the user input information containing the condition when any one of the user input information contains a condition determined to be important in advance; and generating feedback information by displaying a second highlighting on the user input information containing the condition when any one of the user input information does not contain a condition determined to be important in advance.

[0023] In one embodiment, the operation may further include an operation of generating a scoring criterion for the problem based on the model answer information.

[0024] In one embodiment, the operation of generating a scoring criterion for the problem based on the model answer information may include: a operation of classifying the solution type of each of the model answer information; and a operation of generating a partial score system for each of the model answer information by considering the classified solution type.

[0025] In one embodiment, the operation of determining a partial score system for each of the model answer information in consideration of the classified solution type may include the operation of generating the partial score system by assigning a dynamically adjusted weight to each of the model answer information in consideration of the classified solution type.

[0026] In one embodiment, the operation of generating a scoring criterion for the problem based on the model answer information may include: an operation of analyzing the expression method included in the model answer information; and an operation of generating the scoring criterion to recognize as the correct answer when the model answer information and the user input information have different expression methods but the same logical meaning, taking into account the analyzed expression method.

[0027] A computing device according to one embodiment of the present disclosure for realizing the aforementioned task is disclosed. The device comprises at least one processor; and memory, wherein the at least one processor may be configured to acquire model answer information related to a problem; generate reference answer information based on the model answer information using an artificial intelligence model; acquire user input information related to the solution of the problem; and generate feedback information for any one of the user input information by linking the user input information and the reference answer information.

[0028] In one embodiment, the at least one processor may be configured to sequentially divide the acquired model answer information; and to generate the reference answer information by utilizing the artificial intelligence model to add natural language-based descriptive information to each of the sequentially divided model answer information.

[0029] In one embodiment, the at least one processor may be configured to sequentially divide the acquired user input information; match the model answer information and the user input information; and generate feedback information for any one of the user input information by utilizing the matching result and the reference answer information.

[0030] In one embodiment, the at least one processor may be configured to generate feedback information to induce problem solving after the section where the model answer information and the user input information are matched, by utilizing the reference answer information when the preset time required is not satisfied.

[0031] In one embodiment, the at least one processor may be further configured to generate feedback information by displaying a first highlighting on the user input information containing the condition when any of the user input information contains a condition determined to be important in advance; and to generate feedback information by displaying a second highlighting on the user input information containing the condition when any of the user input information does not contain a condition determined to be important in advance.

[0032] In one embodiment, the at least one processor may be further configured to generate a scoring criterion for the problem based on the best answer information.

[0033] In one embodiment, the at least one processor may be further configured to classify the solution type of each of the model answer information; and to generate a partial score system for each of the model answer information in consideration of the classified solution type.

[0034] In one embodiment, the at least one processor may be configured to generate the partial scoring system by assigning a dynamically adjusted weight to each of the best answer information in consideration of the classified solution type.

[0035] In one embodiment, the at least one processor may be configured to analyze the expression method included in the model answer information; and, taking into account the analyzed expression method, generate the scoring criteria to recognize the model answer information and the user input information as correct answers when their expression methods are different but have the same logical meaning.

[0036] The present disclosure analyzes model answer information using an artificial intelligence model to convert it into natural language-based reference answer information, and matches user input solution information step-by-step with the reference answer information to generate feedback information in real time, such as highlighting important conditions, guiding to the next step when the solution is stalled, and identifying incorrect answer sections. This enables the immediate identification of errors, omissions, and stall points occurring during the learner's solution process, and the provision of necessary hints and explanations, thereby improving the learner's understanding of the problem, enhancing self-checking ability, and increasing learning efficiency.

[0037] The present disclosure classifies solution types included in model answer information, automatically generates a partial score system by assigning dynamically adjustable weights to the corresponding types, and automatically constructs and applies scoring criteria to recognize various answers with logically identical meanings as correct by analyzing differences in expression methods. This enables the accurate evaluation of the logical validity of concept application, etc., without being affected by formal differences such as the calculation order, notation method, or use of natural language in learner answers, thereby providing consistent and highly reliable scoring results.

[0038] Meanwhile, the effects of the present disclosure are not limited to those mentioned above, and various effects may be included within the scope obvious to a person skilled in the art from the contents described below.

[0039] FIG. 1 is a block diagram of a computing device for generating feedback information on the solution of a problem according to one embodiment of the present disclosure.

[0040] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0041] FIG. 3 is a flowchart illustrating a method for generating feedback information for the solution of a problem according to one embodiment of the present disclosure.

[0042] FIG. 4 is a diagram showing reference answer information generated based on model answer information according to one embodiment of the present disclosure.

[0043] FIG. 5 is a diagram showing an example of generating feedback information by displaying a first highlighting and a second highlighting on user input information according to one embodiment of the present disclosure.

[0044] FIG. 6 is a diagram showing an example of generating feedback information to induce another problem-solving in one embodiment of the present disclosure.

[0045] FIG. 7 is a diagram showing an example of generating scoring criteria for a problem based on model answer information according to one embodiment of the present disclosure.

[0046] FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0047] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0048] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0049] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0050] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0051] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0052] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0053]

[0054] FIG. 1 is a block diagram of a computing device for generating feedback information on the solution of a problem according to one embodiment of the present disclosure.

[0055] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0056] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0057] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0058] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0059] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0060] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0061] In addition, the network unit (150) presented in this specification may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0062] In the present disclosure, the network unit (150) can be configured regardless of the communication mode, such as wired and wireless, and can be configured as various communication networks such as a Local Area Network (LAN), a Personal Area Network (PAN), and a Wide Area Network (WAN). In addition, the network may be a known World Wide Web (WWW) and may utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

[0063] The technologies described in this specification can be used not only in the networks mentioned above but also in other networks.

[0064]

[0065] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0066] Throughout this specification, artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0067] A neural network can be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting neural networks may be interconnected by one or more links.

[0068] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0069] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight can be variable and can be varied by the user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0070] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0071] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0072] In one embodiment of the present disclosure, a set of neurons or nodes may be defined by the expression a layer.

[0073] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0074] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0075] An artificial intelligence-based model according to one embodiment of the present disclosure may include a deep neural network (DNN). A deep neural network may refer to a neural network that includes a plurality of hidden layers in addition to an input layer and an output layer. By using a deep neural network, latent structures of data can be identified. That is, latent structures of photos, text, video, voice, protein sequence structures, gene sequence structures, peptide sequence structures, music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the voice, etc.), and / or binding affinity between peptides and MHCs can be identified. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), Transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0076] The artificial intelligence-based model of the present disclosure may be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0077] A neural network that can be used in an artificial intelligence-based model of the present disclosure may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of a neural network may be a process of applying knowledge to the neural network to perform a specific operation.

[0078] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training point is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node being updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0079] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed using the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a type of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0080]

[0081] This disclosure relates to a technology that utilizes a Large Language Model (LLM) to convert model answer information into detailed, natural language-centered reference answer information, and to generate customized feedback information by analyzing the learner's solution steps and input characteristics. To improve the learning performance of the LLM model, this disclosure may additionally utilize a dataset specialized for solving mathematical problems (e.g., MATH Dataset), thereby pre-training the model to naturally understand and generate various mathematical expressions such as systems of equations, transposition, right-hand side, and variable substitution. Furthermore, if the correctness of specific steps in the learner's solution is indicated in a highlighting format (e.g., color display using the LaTeX / textcolor command) and provided as input to the LLM, this disclosure can generate more sophisticated and contextually appropriate feedback sentences based on this structural information. Accordingly, the LLM can precisely grasp the learner's solution context and improve the accuracy and explanatory power of the feedback. Accordingly, the present disclosure implements a problem-solving feedback generation technology capable of providing higher quality customized feedback information to learners by combining natural language expansion of model answers, analysis of solution steps, highlighting-based input processing, and understanding of mathematical concepts.

[0082]

[0083] FIG. 3 is a flowchart illustrating a method for generating feedback information for a solution to a problem according to one embodiment of the present disclosure. The method for generating feedback information for a solution to a problem can be performed by a computing device (100).

[0084] According to one embodiment of the present disclosure, a computing device (100) can obtain information on model answers related to a problem (S110). The computing device (100) can obtain information on a problem and model answers related to the problem. For example, a problem (10) refers to a mathematical question or calculation instruction that a learner must solve, and may include all types of problem data such as problem sentences in text form, formulas or function expressions, problems in a form mixed with text and formulas, or descriptive, multiple-choice, or short-answer types.

[0085] In addition, model answer information refers to a formula-centered recommended solution procedure provided in response to a problem, and may be composed of a set of structured solution units (steps) that include an operation expression, a value substitution process, a calculation procedure, and a final answer at each step. For example, referring to Fig. 4, the model answer information may be expressed as information that lists the solution to the problem step by step, such as checking the function expression, substituting values, step-by-step calculation, and deriving the final answer.

[0086] For example, the computing device (100) can acquire the problem (10) and model answer information (20) in various ways. The computing device (100) can acquire the problem and model answer information related to the problem in various forms, such as a problem sheet in PDF format, a scanned image, or an online learning platform UI screen. Additionally, the computing device (100) can convert the problem and model answer information related to the problem into an analyzable form for processing. For example, when the problem and model answer are input from a PDF file or an image file, the computing device (100) can acquire the problem and model answer information by converting them into structured data by extracting characters and formulas using an OCR (Optical Character Recognition) module or a formula recognition module. Additionally, the computing device (100) can communicate with a question bank database or an external learning server to directly query problem and model answer records through API calls or data queries, or receive problem and model answer information in JSON or XML format from a cloud-based problem providing server. Additionally, the computing device (100) can obtain the information when a teacher or administrator directly inputs the problem and model answer information using text, a formula editor, an image upload function, etc., through a user interface (UI). The computing device (100) can obtain the problem (10) and model answer information (20) through various channels, such as a file input method (PDF, image), a database query method, a data reception method from an external server, or a user direct input method.

[0087]

[0088] FIG. 4 is a diagram showing reference answer information generated based on model answer information according to one embodiment of the present disclosure.

[0089] According to one embodiment of the present disclosure, a computing device (100) can generate reference answer information based on model answer information by utilizing an artificial intelligence model (S120). For example, the artificial intelligence model may include a natural language generation model including a GPT (generative pre-trained transformer), T5 (Text-to-Text Transformer), a BERT-based generative model, or an LLM-based model specialized for solving mathematical problems. Such an artificial intelligence model can receive formulas, operation structures, variable relationships, etc., of the model answer information as input, interpret the meaning of each solution step, and generate a natural language explanation. For example, if the model answer includes a step of "substitute x=12," the artificial intelligence model can generate reference answer information by analyzing the problem and solution method and automatically generating a natural language-based sentence such as "Since there are 12 classes, we substitute 12 for the value of x and calculate." The computing device (100) can convert the model answer information into a richly expanded form based on natural language by utilizing an artificial intelligence model, such as a Large Language Model (LLM). This transformation process is intended to improve the accuracy and naturalness of subsequently generated feedback information by reconstructing simple formula- or equation-centered solutions into a more explanatory and descriptive form.

[0090] Alternatively, AI models can be fine-tuned using datasets specialized for solving mathematical problems. For example, AI models can perform supervised fine-tuning using mathematics-specific datasets (e.g., public MATH datasets, in-house datasets built by educational institutions) that cover various topics such as systems of equations, linear and quadratic functions, geometry and vectors, probability and statistics, and sequences. In this case, each training sample may include the problem text, the step-by-step solution of the model answer, the correct answer, and, if necessary, examples of natural language explanations for reference answer information. The AI ​​model can be trained to generate step-by-step natural language explanations (reference answer information) by taking the structure of the given problem and model answer as input. Additionally, the AI ​​model can perform instruction-tuning fine-tuning. Specifically, training samples can be composed of "prompts" and "responses." The prompt may include the problem, the formula steps of the model answer, the student's solution steps, highlighting information, and grading criteria metadata, while the response may include feedback sentences regarding the corresponding steps, hint sentences, results of classifying incorrect answer types, and feedback types such as positive, negative, or abandonment. Through this, the AI ​​model can learn a multi-task output format that simultaneously performs reference answer generation, feedback generation, and incorrect answer type determination from the same input context.

[0091]

[0092] According to one embodiment, the computing device (100) can sequentially divide the acquired model answer information. Referring to FIG. 4 for example, the computing device (100) can sequentially divide the acquired model answer information. For example, if the model answer information consists of the following formula-based steps, it can be sequentially divided into: ① f(x)=12x+6, ② x=12, ③ f(12)=12Х12+6, ④ 144 + 6 = 150, ⑤ Answer: 150 people. The computing device (100) can divide the model answer information into solution units (steps) separated by each step based on line units, semicolon units, formula units, or step delimiter tokens. Additionally, during the division process, an algorithm can be applied to analyze internal formula components (e.g., terms, operators, left-hand and right-hand terms of an equation) and automatically correct if step separation is required. For example, the computing device (100) may perform a subdivision of "12X12+6" into "multiplication steps" and "addition steps." These subdivided solution units can then be used as input tokens for an artificial intelligence model to generate a natural language explanation.

[0093] Additionally, the computing device (100) can generate reference answer information by utilizing an artificial intelligence model to add natural language-based explanatory information to each of the sequentially divided model answer information. For example, when the sequentially divided model answer information is as follows: ① f(x)=12x+6, ② x=12, ③ f(12)=12X12+6, ④ 144 + 6 = 150, ⑤ Answer: 150 people, the computing device (100) can generate the following natural language explanation after analyzing the meaning of each step through the artificial intelligence model. Referring again to FIG. 4, the computing device (100) generates reference answer information ① “The total number of participants is represented by the function f(x)=12x+6.” Reference answer information ② “Since there are 12 classes, 12 is substituted into the input value x of the function.” Reference answer information ③ “f(12)=12X12+6, and 12X12=144.” Reference answer information ④ “Since 6 additional people are added to 144, the total number of participants is 150.” Reference answer information ⑤ “Therefore, the correct answer is 150.” Reference answer information (30) can be generated as follows.

[0094] In this way, the computing device (100) can automatically generate context-centered reference answer information that the learner can actually understand by adding descriptive and contextual natural language sentences to each step of the model answer. In addition, the computing device (100) can generate reference answers that not only generate simple sentences but also adjust the difficulty of the sentences according to the student's level or include additional background explanations (e.g., definition of concepts, reason for operations).

[0095]

[0096] FIG. 5 is a diagram showing an example of generating feedback information by displaying a first highlighting and a second highlighting on user input information according to one embodiment of the present disclosure, and FIG. 6 is a diagram showing an example of generating feedback information to induce another problem-solving according to one embodiment of the present disclosure.

[0097] According to one embodiment of the present disclosure, a computing device (100) can acquire user input information related to the solution of an artificial intelligence problem (S130). For example, "user input information" includes all forms of data generated throughout the problem-solving process, such as formulas, calculation steps, explanatory sentences, option selection information, and handwritten solution processes, which are input by a learner to solve the problem. The computing device (100) can acquire the user input information in various ways. For example, if a learner directly inputs a formula or sentence using a keyboard, touchscreen, or stylus pen, the computing device (100) can acquire the input in real time through a text input module or a handwriting recognition module. Additionally, if a learner uploads a solution written on paper by taking a picture with a camera, the computing device (100) can extract formulas, symbols, and sentences from the image through an image recognition module and a formula OCR module and convert them into user input information. In addition, when using an online learning platform, information such as multiple-choice selections submitted by the student, drag-and-drop sorting problem-solving information, and calculation tool input values ​​can also be automatically recorded as user input information. Furthermore, if the learner explains the solution steps by voice, the computing device (100) can convert the voice into text through a voice recognition module and use it as problem-solving information. For example, as illustrated in FIG. 5, the computing device (100) can obtain user input information (40) related to the solution of the problem from various user interfaces such as text input, handwriting input, voice input, image-based input, and option input.

[0098] According to one embodiment of the present disclosure, a computing device (100) may generate feedback information for any one of the user input information by linking user input information and reference answer information (S140). For example, the feedback information may include information indicating the results of the computing device (100) comparing and analyzing the user input information and reference answer information to determine the progress of the learner's problem-solving process, whether there is an error, whether the answer is correct, missing operation steps, direction of solution, and logical appropriateness of the expression method. For example, the feedback information may include information on determining whether the answer is correct, information on the type of error, information on the progress of the solution step, information on the matching result with the model answer and reference answer, information on determining the logical equivalence of the expression method, guidance information on the appropriateness of the direction of solution, information on the appropriateness of concept application, and a judgment result for generating highlighting information or hint information.

[0099] According to one embodiment, a computing device (100) can sequentially divide the acquired user input information (40). The user input information may consist of various elements such as text, formulas, symbols, handwriting data, and image-based formulas that constitute the entire solution process written by the learner, and the computing device (100) can analyze this by separating it into steps. For example, the computing device (100) can sequentially divide the acquired user input information by performing at least one of line, statement, operator (Х, +, =), handwriting area analysis, line segmentation, stroke clustering, character recognition (OCR), formula recognition (Math-OCR), and block unit element detection (Layout analysis).

[0100] Additionally, the computing device (100) can match model answer information and user input information. Matching is a process for determining whether the model answer information and user input information have the same meaning as the corresponding solution step, and may be performed in the following manner. For example, the computing device (100) can match model answer information and user input information through text and formula structure-based matching. If the first model answer step is f(x)=12x+6 and the first user input information is f(x)=12x+6, the computing device (100) can determine that it is an exact match because the formula structure (operator, operand, equation structure) is identical. Additionally, the computing device (100) can match model answer information and user input information through logical equivalence-based matching. For example, if the user input is 144, which is 12X12, and the model answer is "12X12=144," the expressions are different but the meanings are identical; therefore, the computing device (100) can determine a logical equivalence match by performing formula normalization, term reordering, and equivalence transformation. Additionally, the computing device (100) may perform a partial match. For example, if the user input is "12X12 = ?" and the model answer step is "12X12=144," and the structure of the operation is correct but there is no result, the computing device (100) may match it as a "partial alignment." The computing device (100) may also perform a match of natural language-based step descriptions by utilizing an artificial intelligence model. The model answer information is in the form of natural language as follows, and "Since there are 12 classes, we substitute x=12.When the user input is “since there are 12 and a half, x=12”, the computing device (100) may perform a sentence embedding-based similarity evaluation to perform a semantic match with the reference answer step.

[0101] Additionally, the computing device (100) can generate feedback information for any one of the user input information by utilizing matching results and reference answer information. Based on the matching results between the user input information and the model answer information and the reference answer information, the computing device (100) can determine the accuracy, error, logical appropriateness, or progress of the solution step for any one of the solution steps entered by the learner, and generate feedback information corresponding to the determination result. For example, if the user input information is matched to exactly match the corresponding step of the model answer information, the computing device (100) can generate "positive feedback information" indicating that the step was performed correctly. For example, the computing device (100) can generate feedback information including a message such as "You have calculated correctly up to this step," thereby guiding the learner that the solution is proceeding normally. Additionally, if user input information only partially matches the formula structure or solution form of the model answer information and the result value is not recorded, the computing device (100) may generate feedback information indicating that the solution contains some correct structure and that the next step needs to be performed. For example, it may provide feedback such as, "You have set up the multiplication expression correctly. The next step is to proceed with the calculation." Additionally, if user input information is structurally or logically identical to the model answer information but differs only in its method of expression, the computing device (100) may determine that both expressions have the same meaning and generate normalization-based feedback information. For example, if "144 is 12 x 12" and "12 x 12 = 144" are recognized as having the same meaning, the computing device (100) may generate feedback information such as, "The method of expression is different, but they have the same meaning. You may proceed to the next step."

[0102] On the other hand, if the matching result indicates that the user input information is clearly inconsistent with the model answer information and is determined to contain calculation errors or conceptual errors, the computing device (100) may generate feedback information pointing out the error itself. For example, if a learner inputs an incorrect calculation result such as "12 x 12 = 124," the computing device (100) may provide feedback information such as "The multiplication result is incorrect at this stage. Please check the operation from the previous stage again." Additionally, if the user input information does not include an operation or description corresponding to a specific stage of the model answer information, that is, if it is determined that the solution is missing an intermediate stage, the computing device (100) may generate feedback information indicating that the corresponding stage has not yet been performed. For example, if only "144" is entered without the step "144 + 6 = 150" being performed, the computing device (100) may generate feedback information such as "The operation required to proceed to the next stage has not yet been performed." Additionally, if it is determined that user input information proceeds in a manner different from the solution order or logical structure listed in the reference answer information, the computing device (100) may generate feedback information that guides the appropriateness of the solution flow. For example, if the reference answer presents the step "Substitute x = 12," but the learner calculates a different formula first, the computing device (100) may generate feedback information such as "The problem-solving process does not match the order of the reference solution. You must set the formula first and then substitute the value."

[0103] According to one embodiment, if any of the user input information contains a condition determined to be important in advance, the computing device (100) may generate feedback information by displaying a first highlighting on the user input information containing the condition. For example, “condition determined to be important in advance” refers to a concept, relational expression, precondition, or conclusion that contributes critically to deriving the correct answer during the problem-solving process, and may include elements in which whether such condition is appropriately specified directly affects the structural validity of the problem-solving and the calculation of partial scores. For example, if any of the user input information does not contain a condition determined to be important in advance, the computing device (100) may generate feedback information by displaying a second highlighting on the user input information containing the condition.

[0104] According to one embodiment, the conditions determined to be important in advance may be automatically set by a computing device (100) by analyzing key concepts included in model answer information, mathematical relationships essential for step development, or conclusions defining the logical flow of problem solving. For example, if a specific item is determined to be an element to which a high weight is assigned, such as a “conclusion derivation step,” “key concept application step,” or “prerequisite specification step,” according to the solution type classification or student solution classification criteria defined in the partial score system generation process, that element may be set as a condition determined to be important in advance.

[0105] Additionally, the computing device (100) can apply a first highlighting to the user input information when important conditions are appropriately specified or correctly applied during the student's solution process, thereby visually indicating that the key step has been performed successfully. For example, in a geometric proof problem, if a condition that serves as a decisive basis for solving the problem is presented, such as "AB = DE, BC = EF, angle B = angle E, therefore SAS are congruent," the computing device (100) can display the said condition as a first highlighting (e.g., blue highlighting).

[0106] On the other hand, if a critical condition is incorrectly specified or a conclusion contains an error, the computing device (100) may apply a second highlighting (e.g., red highlighting, underlining, warning indication, etc.) to the user input information to guide the learner to intuitively recognize the error and review it. For example, if an incorrect angle correspondence is written in a step where SAS congruence must be claimed, the computing device (100) may apply a second highlighting to that part to clearly indicate that it is a critical condition containing an error.

[0107] Accordingly, the conditions determined to be important in advance serve as criteria for identifying concepts and conclusions that play a key role in the problem-solving process, and the computing device (100) can determine the importance of each solution step based on this and generate appropriate highlighting-based feedback information for user input information.

[0108] For example, referring to FIG. 5, when a step-by-step solution of user input information (40) is provided corresponding to each step of the model answer information (20), the computing device (100) can determine whether a specific step among the user input information corresponds to a condition that has been determined to be important in advance. For example, referring to FIG. 5, since the formula "f(12)=12X12+6" entered by the user contains core operations (value substitution and expression construction) of the problem-solving process, the computing device (100) can highlight that the formula is an important step by displaying it as a first highlighting (52). On the other hand, the computing device (100) may display a second-1 highlighting (51-1) if a user makes a calculation error (e.g., "124 + 6") during the calculation process performed by the user, a second-2 highlighting (51-2) if an intermediate calculation error (e.g., "130") is made, or a second-3 highlighting (51-3) if a final result error (e.g., "130 people") is made. The computing device (100) may determine that such user input information does not correspond to important conditions or does not match the reference answer, and may display a second highlighting (51) on the information. The second highlighting (51) may be a display format for visually highlighting incorrect steps or sections requiring correction, such as a red mark, a dotted box, an error marker, etc.

[0109] Accordingly, as illustrated in FIG. 5, the computing device (100) can provide feedback information so that the learner can distinguish between key steps and error steps in the solution process by applying a first highlighting or a second highlighting to the user input information at each step, by comprehensively considering whether the user input information includes important conditions, whether it matches the reference answer, and the validity of the calculation result.

[0110] Alternatively, the computing device (100) may apply a first highlighting to sections of user input information (40) that contain conditions determined to be important in advance, apply a second highlighting to sections that require error or correction, and then display them using special tokens before transmitting them to an artificial intelligence model. For example, text sections to which the first highlighting is applied may be converted to be surrounded by special tokens such as [H1_START] and [H1_END], and text sections to which the second highlighting is applied may be converted to be surrounded by special tokens such as [H2_START] and [H2_END]. The computing device (100) may normalize the highlighting information expressed using LaTeX formulas or the / textcolor command into abstract tags that are easy for the artificial intelligence model to process. For example, if the original formula is expressed as / textcolor{blue}{f(12)=12 / times12+6}, the computing device (100) can internally convert it into a tag-based string such as / [H1_START / ]f(12)=12 / times12+6 / [H1_END / ]. Similarly, an error formula such as / textcolor{red}{124+6} is converted into / [H2_START / ]124+6 / [H2_END / ], thereby enabling the artificial intelligence model to explicitly distinguish and recognize important steps and error steps within a single text sequence. The AI ​​model can recognize structural information regarding which parts are core steps that have already been performed correctly and which parts contain errors through special tokens such as [H1_START] and [H2_START] within the aforementioned prompt, and accordingly, can generate a feedback sentence that includes a balance of "a sentence praising that the core steps were performed well" and "a sentence pointing out errors and guiding in the right direction."

[0111]

[0112] According to one embodiment, if the computing device (100) does not satisfy a pre-set time, it may generate feedback information to induce problem solving after the section where the model answer information and user input information are matched by utilizing reference answer information. For example, if no additional input occurs for a predetermined time after the user input information is successfully matched up to a specific stage, the computing device (100) may determine a core operation or logical combination process to be performed after that stage by utilizing reference answer information and mathematical concept-based solution classification criteria. At this time, the computing device (100) may generate and provide hint-type feedback information in real time to guide the direction of the next determined step of the solution. For example, if a learner writes a solution up to the fourth line and there is no additional input for 20 seconds or more, the computing device (100) may determine that the fourth line is a stage where one calculation process is completed and recognize that the next step of the reference answer information is a process of deriving a conclusion by combining problem conditions. Accordingly, the computing device (100) can generate and display feedback information on the screen that includes a hint message in the form of "Shall we use the conditions of the question?", thereby guiding the learner to naturally continue solving the problem.

[0113] For example, referring to FIG. 6, the computing device (100) can normally match the model answer information and the user input information in the first step (41) and the second step (42). In the third step (43), there may be cases where the user input information is not entered for a certain period of time or longer. In such cases, the computing device (100) can analyze the core operation or description of the third step (43) included in the reference answer information and generate feedback information to induce the user to proceed with the solution of the corresponding step. For example, if the computing device (100) does not input an additional solution for a certain period of time after the solution of the second step (42) is entered, it can induce the user to solve the problem after the solution of the second step (42). The computing device (100) can generate guidance-type feedback information (50) in the form of "Shall we try substituting 12 for x?" by confirming that the next step of the reference answer information after the second step (42) solution is "f(12)=12X12+6". This feedback information is intended to guide the learner to naturally move to the next step of the solution and can be automatically generated based on the flow of the reference answer information. Therefore, the computing device (100) can detect in real time when the learner's solution is stalled or delayed at a specific point, and by providing feedback information that guides the problem-solving after that point based on the reference answer information, it can support the learner to continue the solution process continuously without interrupting it. In addition, the computing device (100) can not stop at merely pointing out errors or determining whether the solution is correct, but can effectively support the learner's thought process and problem-solving process by detecting the learner's solution progress and delay in real time and generating feedback that guides the performance of the next step based on the reference answer information and the results of the mathematical concept analysis.

[0114] According to another embodiment, the computing device (100) can store hint information by linking each step of the reference answer information with the hint information. In particular, since the model answer information may have one logical step expressed across two or more formulas or multiple lines, the computing device (100) can map specific hint information to correspond to multiple lines of the model answer information. For example, the computing device (100) may be configured to map hint information corresponding to Step 1 to the first line of the model answer information, and to map hint information corresponding to Step 2 to the second and third lines of the model answer information simultaneously. Through this, even when the detailed development process of the model answer information is expressed across multiple lines, hint information suitable for the corresponding step can be consistently provided.

[0115] The computing device (100) can determine which stage of the model answer information the user input information has reached. The computing device (100) can match the user input information with the model answer information to calculate the step number currently performed by the learner, and based on the calculated step number, select reference answer information or hint information corresponding to the next stage. For example, if it is determined that the learner's solution has reached the fifth stage of the model answer information, the computing device (100) recognizes that the sixth stage of the reference answer information must be performed and can provide hint information corresponding to the sixth stage or feedback information guiding the direction of the next solution. At this time, considering that the expression method may differ for each learner, the computing device (100) can generate feedback information that is paraphrased into various sentence forms while maintaining the mathematical meaning of the reference answer information. Through this, the computing device (100) can provide customized feedback that adapts to the learner's expression method and solution characteristics, even for the same problem.

[0116]

[0117] According to another embodiment, the computing device (100) analyzes the content of each user input information corresponding to the student's solution to determine what type of solution step (simple calculation, application of mathematical concepts, logical reasoning, etc.) the content corresponds to, and furthermore, can infer the mathematical concept or formula used in that step in reverse. The computing device (100) can identify whether each line of the student's solution is a simple calculation step, a concept application step, or a verification and reasoning step using given conditions by synthesizing the solution type classification results based on the model answer and the partial score criteria. Subsequently, the computing device (100) can infer specific mathematical concepts or formulas, such as whether the distributive law was used, whether the Pythagorean theorem was applied, or whether verification was performed using the conditions presented in the problem, by analyzing the formula structure, symbols used, variable relationships, etc. included in the line. The computing device (100) can utilize the inferred mathematical concept or formula to generate feedback information. For example, regarding a step where the student correctly used the distributive law, positive feedback such as "The distributive law was properly applied in this step" can be generated, and if a different formula was incorrectly used in a step requiring the Pythagorean theorem, supplementary feedback such as "In this step, the lengths of the sides of the right triangle must be calculated using the Pythagorean theorem" can be provided. The computing device (100) can automatically back-infer the mathematical concept or formula used in each line of the student's solution and reflect the result in the feedback content, thereby providing high-quality feedback information that goes beyond simple correct / incorrect judgment to specifically suggest which concept was used well and which concept needs to be supplemented.

[0118]

[0119] According to another embodiment, the computing device (100) can generate feedback information to be provided to the learner by classifying it into various types based on user input information and reference answer information. The computing device (100) can classify the feedback type into three types—positive, negative, and abandonment—depending on whether the learner has reached the solution content and the final answer. Positive feedback may be provided when the learner has accurately written both the solution process and the final answer to the problem, and the computing device (100) can generate a positive feedback message that emphasizes the correctly performed steps or the use of key concepts. On the other hand, negative feedback may be provided when either the learner's solution or the answer is insufficient or an error occurs. In this case, even if the learner has correctly written the final answer, negative feedback may be provided if the solution process is insufficient or if there are formal errors such as the omission of units. Additionally, abandonment feedback may be provided when the solution written by the learner is accurate but the final conclusion or answer has not been reached, and the computing device (100) can generate guidance-type feedback information that induces the performance of the next step.

[0120] For example, the computing device (100) may classify the types of errors into two stages—higher error types and lower error types—to more precisely identify the cause when a learner makes a mistake on a problem. For instance, if the higher error type corresponds to a lack of understanding, the lower error types may include a lack of concept understanding, omission of problem conditions, etc. If the higher error type is a calculation error, the lower error types may include arithmetic errors (errors in four basic operations, rounding errors, etc.) or errors in signs or inequalities (e.g., incorrectly writing ">" as "<", etc.). If the higher error type is a representation error, the lower error types may include format errors (e.g., omission of irreducible fraction notation, omission of curly braces when notating sets, etc.) or simple typos. Additionally, errors that do not fall into the above categories may be classified as other types. The computing device (100) may store the types of errors identified as above together with specific lines of the student's solution that serve as the basis, and may provide customized feedback information to individual students based on this. For example, if the type of incorrect answer is determined to be due to a lack of understanding of concepts, the computing device (100) can generate feedback information that provides additional concept explanation materials or supplementary lectures associated with the problem. If the cause is the omission of problem conditions, it can provide feedback in the form of highlighting the omitted conditions or elaborating on the relevant conditions. If the cause is an expression error, it can generate feedback information that guides the correct expression method and induces the correction of the relevant part. Additionally, the computing device (100) can accumulate and store the results of the analysis of the types of incorrect answers and utilize them for generating subsequent questions provided to learners, providing supplementary learning to prevent similar errors, or generating analysis reports for teachers.

[0121]

[0122] FIG. 7 is a diagram showing an example of generating scoring criteria for a problem based on model answer information according to one embodiment of the present disclosure.

[0123] According to one embodiment of the present disclosure, a computing device (100) can generate scoring criteria for a problem based on model answer information. The computing device (100) can classify the solution type of each of the model answer information. By classifying the solution type of each of the model answer information, the computing device (100) can identify what nature each step has, such as a calculation process, a logical development process, or a condition review process.

[0124] For example, referring to FIG. 7, the model answer consists of multiple steps including various types such as interpretation of problem conditions, transformation of algebraic expressions, calculation processes, and processes for deriving logical conclusions. For instance, Step 1 of FIG. 7 can be classified as a type of “organizing the presented conditions,” and Step 2 can be classified as a “step of mathematically expressing the conditions.” Additionally, Steps 3 and 4 can be classified as a “calculation process” and a “logical development process,” respectively, and Steps 5 and 6 can be classified as steps of examining conditions and performing logical reasoning. Finally, Step 7 can correspond to the “calculation process and final conclusion derivation step.”

[0125] The computing device (100) can generate a partial score system for each of the model answer information by considering the classified solution types. For example, the computing device (100) can assign high weights to types that make a key contribution to the problem-solving process, such as logical development processes (step 4) or condition-based reasoning processes (steps 5, 6). On the other hand, types corresponding to simple substitution or arithmetic calculations (steps 3, 7) can be assigned low weights by judging them to be relatively less important steps.

[0126] For example, the computing device (100) can generate a partial score system by assigning dynamically adjusted weights to each of the model answer information, taking into account the classified solution type. The computing device (100) can dynamically adjust said weights according to the nature of the problem, the purpose of evaluation, or the settings of the educational institution. For example, if the educational institution sets “ability to present logical grounds” as a core element of evaluation, the computing device (100) can adjust the weights of the logical development process (step 4) and the condition review and reasoning steps (steps 5 and 6) to be relatively high. Conversely, in an environment where computational accuracy is evaluated primarily, the weights of the calculation steps (steps 3 and 7) can be increased. In this way, the computing device (100) can generate a sophisticated partial score system for each of the model answer information by evaluating the importance and difficulty of each step based on the step-by-step solution type illustrated in FIG. 7 and assigning dynamically adjusted weights according to the classified solution type. The computing device (100) can set the importance or weight of each step based on the solution types classified in this way, thereby creating a partial score system or automatically configuring the scoring criteria for each step. For example, the scoring criteria can be set by giving a higher weight to the core logical development process of problem solving (step 4, step 6, etc.) and a relatively lower weight to the simple calculation steps (step 3, step 7, etc.). In addition, the computing device (100) can analyze the extent to which each solution type contributes to the student's final answer derivation and automatically determine not only the correct answer but also whether to award partial scores. For example, if the student accurately performed the condition organization and expression in steps 1 and 2, but an error occurred during the condition review process after step 5, the computing device (100) can apply the scoring criteria in a way that awards partial scores for the initial steps.

[0127] Accordingly, as described through FIG. 7, according to one embodiment of the present disclosure, a computing device (100) can automatically generate scoring criteria for a problem by identifying and classifying the solution type of each of the best answer information and setting weights and importance criteria according to the type.

[0128] According to another embodiment, the computing device (100) can analyze the solution content of each step included in the model answer information and classify the steps by type according to standards reasonably accepted in mathematics education and evaluation. More specifically, the computing device (100) can determine the solution type by determining whether each step corresponds to a simple calculation process, a process of applying mathematical concepts, or a process of deriving new facts by combining multiple conditions. These classification criteria can be set to align with descriptive or argumentative evaluation criteria generally used in domestic and international mathematics education. The computing device (100) can construct a partial score system by calculating weights for each step in consideration of the classified solution types. For example, the computing device (100) may assign high partial scores to steps that accurately apply mathematical concepts or logically derive conclusions, and relatively low partial scores to simple calculation steps. Additionally, the computing device (100) may set weights by assigning negative partial scores or reflecting them as deduction factors when arithmetic errors (calculation errors) are included. The computing device (100) may configure the scoring criteria in a way that assigns a high weight to the step that directly contributes to the derivation of the final answer.

[0129] Additionally, the computing device (100) can automatically generate a partial score system based on a predefined weight calculation logic, but can dynamically adjust the weights according to the nature of the problem or the characteristics of the evaluation institution. For example, if the evaluation institution prioritizes the accuracy of the final answer, the computing device (100) may be set to assign a high weight to the final conclusion derivation stage. Conversely, in an environment that emphasizes logical thinking and process-oriented evaluation, the weight system may be adjusted to increase the weight of the condition analysis and logical development stages. Furthermore, by setting the evaluation personnel to directly modify the partial score system or redefine the weights for individual questions, customized scoring criteria reflecting the characteristics of each problem may be generated. In this way, the computing device (100) can generate a sophisticated partial score system that reflects the characteristics of the problem by combining reasonable classification criteria for solution types with a weight adjustment function based on the intentions of the educational institution or the evaluator.

[0130]

[0131] According to one embodiment, the computing device (100) can analyze the expression method included in the model answer information. This is because the model answer information can be composed in various ways, such as mathematical formulas, symbols, geometric representations, and natural language descriptions, so the computing device (100) identifies the structural or semantic characteristics of the expression and uses them for comparison with user input information. The computing device (100) may include its own customized module to support various mathematical expressions used in the domestic curriculum. For example, it may be configured to define a custom class for mathematical expressions not supported by general mathematical expression processing libraries (such as ±, dot product, composite function symbol, number of elements in a set n(A), proportion symbol, etc.), or to compile and process LaTeX expressions not natively supported by SymPy into a Java-based system.

[0132] The computing device (100) can generate a scoring criterion to recognize the model answer information and user input information as correct answers when they have different expression methods but the same logical meaning, taking into account the analyzed expression method. For example, the computing device (100) can perform various text preprocessing logic to improve the accuracy of the correct answer comparison. For example, it can perform an alignment process so that the following characters in geometric representations, such as triangle symbols (△), line segment symbols (), and square symbols (□), can be processed with the same meaning even if the order of the characters following them is changed (e.g., △ABC=△BCA). In addition, it may include a process of simplifying original characters such as ㉠ and ㉡ into ㄱ and ㄴ, or unifying different geometric notation methods such as “triangle ↔ △”, “angle ↔ ∠”, and “line segment ↔ ”. Furthermore, symbols that have meaning only during the rendering process and do not have actual mathematical meaning may be removed. The computing device (100) can perform preprocessing on polynomial-type answers so that they can be judged as identical even if the order of the left and right terms, or the order of items within the terms, changes. Additionally, for questions requiring a natural language-based answer, it can perform a natural language scoring function that determines whether a specific phrase is included in the user's answer according to the institution's settings, or compares sentences by exactly matching them. Meanwhile, the computing device (100) can apply a predefined synonym dictionary to recognize similar expressions as correct answers. For example, various expressions having the same meaning, such as “black / black color,” “sky / sky,” and “pi / pi,” can be normalized into the same form and reflected in the scoring. Furthermore, for questions containing units, the computing device (100) can separate the units included in the answer from the numerical value and compare them, thereby branching the correct or incorrect answer based on the presence or absence of the unit or the appropriateness of the unit.

[0133] The computing device (100) can classify problem types according to problem characteristics into sequential listing problems, unordered listing problems, multiple-choice problems, or problems containing multiple sub-items, and can determine whether the answer is correct by applying different scoring rules for each problem type. The computing device (100) generates scoring criteria so that even if the expression methods are different, if they have the same logical meaning, they can be recognized as correct answers. In addition, the degree of identity recognized as correct answers can be adjusted according to the needs of each institution (e.g., only similar expressions are accepted, natural language is accepted, order is irrelevant is accepted, etc.), thereby applying more flexible scoring criteria. The computing device (100) can consider all diversity of mathematical expressions, symbols, language, and sentence structures to eliminate simple discrepancies caused by differences in expression methods and perform a determination of the correct answer based on actual logical meaning.

[0134]

[0135] The steps mentioned in the foregoing description may be further subdivided into additional steps or combined into fewer steps, depending on the embodiment of the present disclosure. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0136]

[0137] Meanwhile, a computer-readable medium storing a data structure is disclosed according to an embodiment of the present disclosure.

[0138] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data for solving specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of its resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0139] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0140] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form a loop in a graph data structure.

[0141] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, the term neural network will be used consistently. A data structure may include a neural network. Furthermore, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the configurations described above, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated during the computational process of the neural network, and is not limited to the foregoing. A computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0142] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0143] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0144] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0145] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0146] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0147]

[0148] FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0149] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0150] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0151] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0152] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0153] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0154] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1104).

[0155] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0156] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the floppy disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0157] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0158] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0159] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0160] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0161] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0162] When used in a LAN networking environment, the computer (1102) is connected to the LAN (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means of establishing communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via an input device interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0163] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0164] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0165] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0166] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0167] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0168] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0169] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0170] As described above, the relevant details have been described in the best mode for carrying out the invention.

Claims

1. A method for generating feedback information on the solution to a problem, performed by a computing device, Step of obtaining model answer information related to the problem; A step of generating reference answer information based on the above model answer information using an artificial intelligence model; A step of obtaining user input information related to the solution of the above problem; and A step of generating feedback information for any one of the user input information by linking the user input information and the reference answer information. including, method.

2. In Paragraph 1, The step of generating the reference answer information based on the model answer information using the artificial intelligence model described above is: A step of sequentially dividing the above-mentioned model answer information; and A step of generating the reference answer information by adding natural language-based descriptive information to each of the sequentially divided model answer information using the artificial intelligence model; including, method.

3. In Paragraph 2, The step of generating feedback information for any one of the user input information by linking the user input information and the reference answer information is: A step of sequentially dividing the above-mentioned acquired user input information; A step of matching the above model answer information and the above user input information; and A step of generating feedback information for any one of the user input information using the above matching result and the above reference answer information including, method.

4. In Paragraph 3, The step of generating feedback information for any one of the user input information by utilizing the matching results and reference answer information is: If the pre-set time required is not satisfied, the step of generating the feedback information to induce problem solving after the section where the model answer information and the user input information are matched, by utilizing the reference answer information. including, method.

5. In Paragraph 3, The step of generating feedback information for any one of the user input information by utilizing the matching results and reference answer information is: If any of the above user input information includes a condition determined to be important in advance, a step of generating the feedback information by displaying a first highlighting on the user input information including the condition; and If any of the above user input information does not include a condition determined to be important in advance, the step of generating the feedback information by displaying a second highlighting on the user input information containing the condition. including, method.

6. In Paragraph 3, The above method is, A method further comprising the step of generating scoring criteria for the above problem based on the above model answer information, method.

7. In Paragraph 6, The step of generating scoring criteria for the above problem based on the above model answer information is, A step of classifying the solution type for each of the above model answer information; and A step of generating a partial score system for each of the above model answer information by considering the above classified solution types; including, method.

8. In Paragraph 7, The step of determining a partial scoring system for each of the above model answer information by considering the above-classified solution types is, A step of generating the partial scoring system by assigning dynamically adjusted weights to each of the model answer information, taking into account the above classified solution types. including, method.

9. In Paragraph 5, The step of generating scoring criteria for the above problem based on the above model answer information is, A step of analyzing the expression method included in the above model answer information; and A step of generating the scoring criteria to recognize the above model answer information and the above user input information as the correct answer when, considering the above analyzed expression methods, the expression methods are different from each other but have the same logical meaning; including, method.

10. A computer program stored on a computer-readable storage medium, wherein, when the computer program is executed on one or more processors, the one or more processors are configured to perform the following operations to generate feedback information regarding the solution to a problem, and said operations are: The action of obtaining model answer information related to the problem; The operation of generating reference answer information based on the above model answer information using an artificial intelligence model; An operation to obtain user input information related to the solution of the above problem; and The operation of generating feedback information for any one of the user input information by linking the user input information and the reference answer information. including, A computer program stored on a computer-readable storage medium.

11. In Paragraph 10, The operation of generating the reference answer information based on the model answer information using the artificial intelligence model described above is, The operation of sequentially dividing the above-mentioned acquired model answer information; and The operation of generating the reference answer information by adding natural language-based descriptive information to each of the sequentially divided model answer information using the above artificial intelligence model; including, A computer program stored on a computer-readable storage medium.

12. In Paragraph 11, The operation of generating feedback information for any one of the user input information by linking the above user input information and the above reference answer information is, The operation of sequentially dividing the above-mentioned acquired user input information; An operation of matching the above model answer information and the above user input information; and The operation of generating feedback information for any one of the user input information using the above matching results and the above reference answer information. including, A computer program stored on a computer-readable storage medium.

13. In Paragraph 12, The operation of generating feedback information for any one of the user input information using the above matching results and the above reference answer information is, If the pre-set time required is not satisfied, the operation of generating the feedback information to induce problem solving after the section where the model answer information and the user input information are matched, by utilizing the reference answer information. including, A computer program stored on a computer-readable storage medium.

14. In Paragraph 13, The operation of generating feedback information for any one of the user input information using the above matching results and the above reference answer information is, If any of the above user input information includes a condition determined to be important in advance, the operation of generating the feedback information by displaying a first highlighting on the user input information containing the condition; and If none of the above user input information includes a condition determined to be important in advance, the operation of generating the feedback information by displaying a second highlighting on the user input information containing the condition. including more, A computer program stored on a computer-readable storage medium.

15. In Paragraph 12, The above operation is, Further including the operation of generating scoring criteria for the above problem based on the above model answer information, A computer program stored on a computer-readable storage medium.

16. In Paragraph 15, The operation of generating scoring criteria for the above problem based on the above model answer information is, An action of classifying the solution type for each of the above model answer information; and The operation of generating a partial scoring system for each of the above model answer information by considering the above classified solution types; including, A computer program stored on a computer-readable storage medium.

17. In Paragraph 16, The operation of determining a partial scoring system for each of the above model answer information by considering the above-classified solution types is, The operation of generating the partial scoring system by assigning dynamically adjusted weights to each of the model answer information, taking into account the above classified solution types; including, A computer program stored on a computer-readable storage medium.

18. In Paragraph 14, The operation of generating scoring criteria for the above problem based on the above model answer information is, An operation to analyze the expression method included in the above model answer information; and An operation to generate the scoring criteria to recognize the above-mentioned model answer information and the above-mentioned user input information as the correct answer when, considering the above-mentioned analyzed expression methods, the expression methods of the above-mentioned model answer information and the above-mentioned user input information are different from each other but have the same logical meaning; including, A computer program stored on a computer-readable storage medium.

19. As a computing device, At least one processor; and Memory; Includes, The above-mentioned at least one processor is, Obtain model answer information related to the problem; Generate reference answer information based on the above model answer information using an artificial intelligence model; Obtain user input information related to the solution of the above problem; and Configured to generate feedback information for any one of the user input information by linking user input information and the above reference answer information, device.

20. In Paragraph 19, The above-mentioned at least one processor is, Sequentially divide the above-mentioned model answer information; and Configured to generate the reference answer information by utilizing the above artificial intelligence model to add natural language-based descriptive information to each of the sequentially divided model answer information, device.

21. In Paragraph 20, The above-mentioned at least one processor is, Sequentially dividing the above-mentioned acquired user input information; Matching the above model answer information and the above user input information; and Configured to generate feedback information for any one of the user input information by utilizing the above matching results and the above reference answer information, device.

22. In Paragraph 21, The above-mentioned at least one processor is, If the pre-set time required is not satisfied, the above reference answer information is utilized to generate the above feedback information to induce problem solving after the section where the above model answer information and the above user input information are matched. device.

23. In Paragraph 21, The above-mentioned at least one processor is, If any of the above user input information includes a condition determined to be important in advance, the feedback information is generated by displaying a first highlighting on the user input information containing the condition; and If any of the above user input information does not include a condition determined to be important in advance, the method is further configured to generate the feedback information by displaying a second highlighting on the user input information that includes the condition. device.

24. In Paragraph 21, The above-mentioned at least one processor is, Further configured to generate scoring criteria for the above problem based on the above model answer information, device.

25. In Paragraph 24, The above-mentioned at least one processor is, Classify the solution types for each of the above model answer information; and Additionally configured to generate a partial scoring system for each of the above model answer information, taking into account the above classified solution types, device.

26. In Paragraph 25, The above-mentioned at least one processor is, Configured to generate the partial scoring system by assigning dynamically adjusted weights to each of the model answer information, taking into account the above classified solution types, device.

27. In Paragraph 23, The above-mentioned at least one processor is, Analyze the expression methods included in the above model answer information; and Considering the above-analyzed expression methods, the scoring criteria are configured to be generated so as to recognize the above-analyzed model answer information and the above-analyzed user input information as the correct answer when their expression methods are different but they have the same logical meaning. device.

Images