Method for generating correction information about mathematical problem solving

Abstract

According to an embodiment of the present disclosure, disclosed is a method, performed by a computing device, for generating correction information about mathematical problem solving. The method may comprise the steps of: acquiring, from a first user terminal, first user input information related to mathematical problem solving; analyzing the first user input information related to the mathematical problem solving by utilizing a pre-trained artificial intelligence model; providing a second user terminal with the first user input information related to the mathematical problem solving when a confidence score for an analysis result of the artificial intelligence model is less than a preset threshold value; and generating correction information about the mathematical problem solving on the basis of second user input information acquired from the second user terminal.

Description

How to generate feedback information for math problem solutions

[0001] The present invention relates to a method and a data pipeline for training logical thinking at the level of the process of solving mathematical problems, and more specifically, to a technology for generating correction information on the solution of mathematical problems by utilizing an artificial intelligence model and verification by multiple users.

[0002] In general AI-based learning systems, a method is mainly used in which an AI model automatically grades or corrects problem solutions submitted by users.

[0003] However, in subjects centered on logical thinking, such as mathematics, simple automatic grading alone is not sufficient because it is necessary to evaluate not only whether the answer is correct but also the logical consistency of the solution, the validity of the calculation procedure, and the accuracy of the application of concepts.

[0004] In particular, artificial intelligence models find it difficult to make accurate judgments when solution structures or mathematical expressions are diverse, and there is a risk of providing incorrect feedback when the confidence of the correction results generated by the model is low.

[0005] In addition, existing automated feedback systems made uniform judgments even when the model's reliability was low or simply relied on the intervention of reviewers (e.g., teachers). As a result, it was difficult to provide immediate feedback in large-scale user environments, and there was a problem with the limited accumulation of training data for artificial intelligence models.

[0006] Korean Patent Publication No. 10-2021-0094317 (July 29, 2021) discloses a method for automatically providing feedback learning content by utilizing mathematical problem solving.

[0007] The present disclosure aims to provide a method for automatically determining the reliability of an AI correction result for a math problem solution, correcting the correction information by reflecting the judgments of other users when the reliability is low, and utilizing the result as training data for an AI model.

[0008] Meanwhile, the technical problem that the present disclosure aims to solve is not limited to the technical problem mentioned above, and various technical problems may be included within the scope obvious to a person skilled in the art from the contents described below.

[0009] According to one embodiment of the present disclosure for realizing the aforementioned task, a method for generating correction information for a solution to a mathematical problem performed by a computing device is disclosed. The method may include: a step of obtaining first user input information related to a solution to a mathematical problem from a first user terminal; a step of analyzing the first user input information related to a solution to a mathematical problem using a pre-trained artificial intelligence model; a step of providing the first user input information related to a solution to a mathematical problem to a second user terminal when the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold; and a step of generating correction information for a solution to a mathematical problem based on the second user input information obtained from the second user terminal.

[0010] In one embodiment, the step of analyzing the first user input information related to the solution of the math problem using the pre-trained artificial intelligence model may include the step of dividing the process of solving the math problem into steps to generate a plurality of solution items.

[0011] In one embodiment, the second user terminal may include at least one other user terminal that performs a solution to the mathematical problem; or a terminal of an inspector that performs an inspection of the solution to the mathematical problem.

[0012] In one embodiment, when the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold, the step of providing the first user input information related to the solution of the mathematical problem to the second user terminal may include: the step of selecting a plurality of user terminals to provide the mathematical problem and the plurality of solution items; and the step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals.

[0013] In one embodiment, the step of selecting a plurality of user terminals for providing the mathematical problem and the plurality of solution items may include the step of selecting user terminals among the plurality of user terminals whose learning achievement level for the mathematical problem is above a preset standard.

[0014] In one embodiment, the step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals includes, for each of the plurality of solution items, the step of providing a plurality of error type items for selecting an error type, wherein the plurality of error type items may include a selectable error type item configured to select one of a predefined error type; and a descriptive error type item configured to input an undefined error type as text.

[0015] In one embodiment, the step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals may further include, for each of the plurality of solution items, a step of verifying the suitability of the data included in each solution item.

[0016] In one embodiment, the step of generating correction information for the solution of the mathematical problem based on the second user input information obtained from the second user terminal may include at least one of the steps of: obtaining second user input information that selects the first error solution item present in the plurality of solution items; or obtaining second user input information that selects an item corresponding to no error.

[0017] In one embodiment, the step of generating correction information for the solution of the mathematical problem based on second user input information obtained from the second user terminal may include: a step of calculating a judgment agreement for the first error solution item that has an error among the plurality of solution items by considering a plurality of second user input information obtained from a plurality of user terminals, or a step of calculating a judgment agreement for an item corresponding to having no error; and a step of generating correction information for the mathematical problem solution item by considering the calculated judgment agreement.

[0018] In one embodiment, the method may further include the step of utilizing the generated annotation result as training data for the artificial intelligence model.

[0019] In one embodiment, the method may further include the step of generating annotation information for the math problem solution item based on the analysis result of the artificial intelligence model, when the reliability score of the analysis result of the artificial intelligence model is greater than or equal to a preset threshold, without providing the first user input information related to the solution of the math problem to the second user terminal.

[0020] A computer program stored on a computer-readable storage medium is disclosed in accordance with one embodiment of the present disclosure for realizing the aforementioned task. When the computer program is executed on one or more processors, the one or more processors are configured to perform the following operations to generate correction information for the solution of a mathematical problem, wherein the operations may include: an operation of obtaining first user input information related to the solution of a mathematical problem from a first user terminal; an operation of analyzing the first user input information related to the solution of the mathematical problem using a pre-trained artificial intelligence model; an operation of providing the first user input information related to the solution of the mathematical problem to a second user terminal when the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold; and an operation of generating correction information for the solution of the mathematical problem based on the second user input information obtained from the second user terminal.

[0021] In one embodiment, the operation of analyzing the first user input information related to the solution of the math problem using the pre-trained artificial intelligence model may include the operation of dividing the process of solving the math problem into steps to generate a plurality of solution items.

[0022] In one embodiment, the second user terminal may include at least one other user terminal that performs a solution to the mathematical problem; or a terminal of an inspector that performs an inspection of the solution to the mathematical problem.

[0023] In one embodiment, when the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold, the operation of providing the first user input information related to the solution of the mathematical problem to the second user terminal may include: the operation of selecting a plurality of user terminals to provide the mathematical problem and the plurality of solution items; and the operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals.

[0024] In one embodiment, the operation of selecting a plurality of user terminals for providing the mathematical problem and the plurality of solution items may include the operation of selecting user terminals among the plurality of user terminals whose learning achievement level for the mathematical problem is above a preset standard.

[0025] In one embodiment, the operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals includes, for each of the plurality of solution items, the operation of providing a plurality of error type items for selecting an error type, and the plurality of error type items may include a selectable error type item configured to select one of a predefined error type; and a descriptive error type item configured to input an undefined error type as text.

[0026] In one embodiment, the operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals may further include, for each of the plurality of solution items, an operation of verifying the suitability of the data included in each solution item.

[0027] The operation of generating correction information for the solution of the math problem based on the second user input information obtained from the second user terminal may include at least one of the operation of obtaining second user input information that selects the first error solution item existing in the plurality of solution items; or the operation of obtaining second user input information that selects an item corresponding to no error.

[0028] In one embodiment, the operation of generating correction information for the solution of the mathematical problem based on second user input information obtained from the second user terminal may include: an operation of calculating a judgment agreement for the first error solution item that has an error among the plurality of solution items by considering a plurality of second user input information obtained from a plurality of user terminals, or an operation of calculating a judgment agreement for an item corresponding to that there is no error; and an operation of generating correction information for the mathematical problem solution item by considering the calculated judgment agreement.

[0029] A computing device according to one embodiment of the present disclosure for realizing the aforementioned tasks is disclosed. The device comprises at least one processor; and a memory, wherein the at least one processor may be configured to acquire first user input information related to the solution of a mathematical problem from a first user terminal; analyze the first user input information related to the solution of the mathematical problem using a pre-trained artificial intelligence model; provide the first user input information related to the solution of the mathematical problem to a second user terminal when the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold; and generate correction information for the solution of the mathematical problem based on the second user input information acquired from the second user terminal.

[0030] In one embodiment, the at least one processor may be configured to generate a plurality of solution items by dividing the process of solving the mathematical problem into steps.

[0031] In one embodiment, the second user terminal may include at least one other user terminal that performs a solution to the mathematical problem; or a terminal of an inspector that performs an inspection of the solution to the mathematical problem.

[0032] In one embodiment, the at least one processor may be configured to select a plurality of user terminals for providing the mathematical problem and the plurality of solution items; and to provide the mathematical problem and the plurality of solution items to the selected plurality of user terminals.

[0033] In one embodiment, the at least one processor may be configured to select user terminals among the plurality of user terminals whose learning achievement level regarding the mathematical problem is above a preset standard.

[0034] In one embodiment, the at least one processor is configured to provide a plurality of error type items for selecting an error type for each of the plurality of solution items, and the plurality of error type items may include a selectable error type item configured to select one of predefined error types; and a descriptive error type item configured to input an undefined error type as text.

[0035] In one embodiment, the at least one processor may be further configured to verify the suitability of the data included in each of the plurality of solution items for each of the solution items.

[0036] In one embodiment, the at least one processor may include at least one of the operation of obtaining second user input information that selects the first error solution item present in the plurality of solution items; or the operation of obtaining second user input information that selects an item corresponding to no error.

[0037] In one embodiment, the at least one processor may be configured to calculate a judgment agreement for the first error solution item that has an error among the plurality of solution items, or to calculate a judgment agreement for an item that has no error, by considering a plurality of second user input information obtained from a plurality of user terminals; and to generate correction information for the math problem solution item by considering the calculated judgment agreement.

[0038] The present disclosure automatically detects uncertainty in analysis results based on the reliability score of an artificial intelligence model, and when reliability is low, provides math problem-solving items to multiple user terminals to collect and analyze error types and judgment consistency, thereby complementing the limitations of judgment made solely by artificial intelligence and generating highly reliable correction information, which can provide immediate and accurate feedback to users while continuously improving the performance of the artificial intelligence model.

[0039] Meanwhile, the effects of the present disclosure are not limited to those mentioned above, and various effects may be included within the scope obvious to a person skilled in the art from the contents described below.

[0040] FIG. 1 is a block diagram of a computing device for generating correction information for the solution of a mathematical problem according to one embodiment of the present disclosure.

[0041] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0042] FIG. 3 is a configuration diagram of a computing device for generating correction information for the solution of a mathematical problem according to one embodiment of the present disclosure.

[0043] FIG. 4 is a flowchart illustrating a method for generating correction information for the solution of a mathematical problem according to one embodiment of the present disclosure.

[0044] FIG. 5 is a user interface including a mathematical problem and a plurality of solution items provided to a second user terminal according to one embodiment of the present disclosure.

[0045] FIG. 6 is a user interface that provides a plurality of error type items for selecting an error type for each of a plurality of solution items provided to a second user terminal according to one embodiment of the present disclosure.

[0046] FIG. 7 is a user interface showing optional error type and descriptive error type items according to one embodiment of the present disclosure.

[0047] FIG. 8 is a user interface showing an operation in which a learner, who has been provided with a solution in which the ground truth has already been determined according to one embodiment of the present disclosure, receives immediate feedback based on the result of their choice.

[0048] FIG. 9 is a user interface for any one of the operations of generating correction information for a mathematical problem-solving item by considering the calculated judgment agreement according to one embodiment of the present disclosure.

[0049] FIG. 10 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0050] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0051] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0052] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0053] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0054] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0055] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0056]

[0057] FIG. 1 is a block diagram of a computing device for generating correction information for the solution of a mathematical problem according to one embodiment of the present disclosure.

[0058] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0059] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0060] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0061] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0062] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0063] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0064] In addition, the network unit (150) presented in this specification may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0065] In the present disclosure, the network unit (150) can be configured regardless of the communication mode, such as wired and wireless, and can be configured as various communication networks such as a Local Area Network (LAN), a Personal Area Network (PAN), and a Wide Area Network (WAN). In addition, the network may be a known World Wide Web (WWW) and may utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

[0066] The technologies described in this specification can be used not only in the networks mentioned above but also in other networks.

[0067]

[0068] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

[0069] Throughout this specification, artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0070] A neural network can be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting neural networks may be interconnected by one or more links.

[0071] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0072] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight can be variable and can be varied by the user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0073] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0074] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0075] In one embodiment of the present disclosure, a set of neurons or nodes may be defined by the expression a layer.

[0076] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0077] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0078] An artificial intelligence-based model according to one embodiment of the present disclosure may include a deep neural network (DNN). A deep neural network may refer to a neural network that includes a plurality of hidden layers in addition to an input layer and an output layer. By using a deep neural network, latent structures of data can be identified. That is, latent structures of photos, text, video, voice, protein sequence structures, gene sequence structures, peptide sequence structures, music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the voice, etc.), and / or binding affinity between peptides and MHCs can be identified. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), Transformers, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0079] The artificial intelligence-based model of the present disclosure may be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0080] A neural network that can be used in an artificial intelligence-based model of the present disclosure may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of a neural network may be a process of applying knowledge to the neural network to perform a specific operation.

[0081] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training point is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node being updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0082] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed using the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a type of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0083]

[0084] The present invention relates to a technology for automatically generating and correcting correction information through interaction between an artificial intelligence model and a user group during the process of solving mathematical problems. Specifically, a pre-trained artificial intelligence model analyzes a solution submitted by a user step by step, and if the reliability of the analysis result is below a threshold, the solution is provided to another user terminal to allow the user to select and input whether there is an error and the type of error (e.g., calculation error, lack of conceptual understanding, lack of logical basis, etc.) at each step. Subsequently, the final correction result is determined by calculating the judgment agreement rate through the synthesis of multiple user inputs, and the confirmed correction information is utilized as training data for the artificial intelligence model to continuously improve performance.

[0085] Furthermore, the provided learning data is filtered through a suitability verification module to ensure that it does not include personal information, profanity, or irrelevant content. When selecting users, the reliability of the correction results is enhanced by applying constraints and weights based on the achievement level of the relevant unit, as well as procedures to secure statistically significant responses. Through this, the present invention can complement the limitations of artificial intelligence while simultaneously achieving improved thinking skills through peer evaluation and enhanced correction quality.

[0086]

[0087] FIG. 3 is a configuration diagram of a computing device for generating correction information for the solution of a mathematical problem according to one embodiment of the present disclosure.

[0088] According to one embodiment of the present disclosure, the first user terminal (200) is a device used by a user and can write a solution to a mathematical problem and transmit the solution to a computing device (100). The first user terminal (200) may be a terminal possessed by a user who performs the solution to a mathematical problem. The first user terminal (200) may be implemented in the form of a smartphone, tablet, laptop, desktop computer, etc., and performs the function of the user checking the problem and inputting or submitting the solution.

[0089] A computing device (100) can receive solution data transmitted from a first user terminal (200), analyze the solution process using a pre-trained artificial intelligence model, and calculate a reliability score of the analysis result. If the reliability score of the analysis result is less than a preset threshold, the computing device (100) can transmit the solution data obtained from the first user to a second user terminal (300) so that other users can review the solution and select an error item or input an error type.

[0090] The second user terminal (300) is a device used by multiple users or reviewers and can be implemented in various forms such as a tablet, smartphone, desktop computer, or laptop. The second user terminal (300) may be a terminal used by another user or reviewer who reviews the solution submitted by the first user and determines whether there is an error. The second user terminal (300) can display mathematical problems and step-by-step solution items provided by the computing device (100), and perform the function of selecting or inputting whether there is an error, the type of error (e.g., calculation error, lack of conceptual understanding, lack of logical basis), and the direction of improvement. The second user terminal (300) can display mathematical problems and step-by-step solution items provided by the computing device (100), select whether there is an error or specify the type of error (e.g., calculation error, lack of conceptual understanding, lack of logical basis) for each step, and input the cause of the error or the direction of improvement as text as necessary. In addition, the second user terminal (300) can be configured so that multiple users can participate simultaneously and submit responses, and the input results are transmitted to the computing device (100) and statistically aggregated and analyzed to increase the reliability of the correction information.

[0091] The computing device (100) can provide the solution submitted by the first user to the second user terminal (300) in the form of a learning solution. The computing device (100) calculates the degree of agreement in judgment based on responses collected from multiple second user terminals (300), and if a statistically significant result is derived, it can determine the final correction information based on this. The determined correction information is provided to the first user terminal (200) again in the form of feedback, and is also stored as training data for an artificial intelligence model so that it can be used to improve correction performance in the future.

[0092]

[0093] According to one embodiment, the computing device (100) may refer to existing verified problem, solution, and correction data stored in memory (130), or obtain solution data generated by modifying a model answer (correct solution), or solution data submitted in real time by another learner. The computing device (100) may analyze these data, process them to suit the learner's level and learning purpose, and provide them to the first user terminal (200) as a math problem in the form of a learning problem.

[0094] The first user terminal (200) displays a problem of a learning type provided by the computing device (100), and the first user can input a solution or correction input for the problem as first user input information in a descriptive form. For example, the first user terminal (200) can transmit first user input information including a descriptive answer that includes a calculation formula, a step-by-step solution, or grounds for error judgment, and the computing device (100) can analyze this to detect whether it is correct or incorrect, or the location of a solution error. The computing device (100) calculates a reliability score of the correction result corresponding to the input solution information, and if the reliability score is below a preset threshold, it can provide the problem and solution items to the second user terminal (300) or a plurality of other learner terminals based on the solution data collected from the first user terminal (200).

[0095] The second user terminal (300) can display the solution submitted by the first user in a learning format, select a solution step that contains an error, or select "correct answer" if it determines that the step is correct. Additionally, the second user terminal (300) can generate second user input information by selecting error types such as calculation errors, lack of concept understanding, or lack of logical basis, or by entering additional comments in a descriptive format.

[0096] The computing device (100) can aggregate input information collected from a plurality of second user terminals (300) to calculate the judgment consistency of the correction result, and if the judgment consistency is greater than or equal to a preset standard, it can finalize the correction result and return the correct and incorrect judgment results to the first user terminal (200). On the other hand, if the judgment consistency is less than the standard, the computing device (100) can automatically reassign the problem to an additional reviewer or send a review request to a human worker to withhold finalization of the result.

[0097] In this way, the computing device (100) according to the present embodiment can provide personalized learning problems to learners by linking existing verification data, model answer modification data, and real-time learner solution data, and can reliably determine correction results by utilizing multiple user judgments.

[0098]

[0099] FIG. 4 is a flowchart illustrating a method for generating correction information for a solution to a mathematical problem according to one embodiment of the present disclosure, FIG. 5 is a user interface including a mathematical problem and a plurality of solution items provided to a second user terminal according to one embodiment of the present disclosure, FIG. 6 is a user interface providing a plurality of error type items for selecting an error type for each of the plurality of solution items provided to the second user terminal according to one embodiment of the present disclosure, and FIG. 7 is a user interface illustrating a selectable error type and a descriptive error type item according to one embodiment of the present disclosure. For reference, the method for generating correction information for a solution to a mathematical problem may be performed by a computing device (100).

[0100]

[0101] According to one embodiment of the present disclosure, a computing device (100) may obtain first user input information related to the solution of a mathematical problem from a first user terminal (200) (S110). The computing device (100) may provide a mathematical problem of a suitable difficulty and range to the user by referring to information regarding the first user's learning level, learning unit, or past solution history. For example, the computing device (100) may automatically select a problem corresponding to the unit currently being studied based on the first user's learning unit information, or present a customized problem among a plurality of problems by considering the user's correct answer rate, solution speed, or understanding of concepts.

[0102] The first user terminal (200) can display a problem provided by the computing device (100) and transmit first user input information related to the solution of a math problem entered by the user to the computing device (100). The computing device (100) can obtain first user input information consisting of at least one of an input formula, a solution formula, a calculation procedure, a descriptive sentence, and a multiple-choice response for a math problem.

[0103] The computing device (100) can provide a math problem to the first user terminal (200) by utilizing existing problem, solution, and correction data stored in memory (130). Additionally, it can provide a math problem to the first user terminal (200) in a modified form of the existing problem, solution, and correction data. The computing device (100) can convert the data into a learning-type problem format that induces the learner to construct the solution process directly by modifying some calculation formulas, conditions, or logical developments based on existing verified correct solutions or model answers. The first user terminal (200) displays the modified math problem, and the first user can directly input first user input information in the form of a descriptive answer to the problem. The input descriptive answer may include processes such as mathematical grounds, formula development, unit conversion, or condition setting for each step of problem solving, and the computing device (100) can analyze this through an artificial intelligence model to determine the validity, logical consistency, and presence of errors of the solution.

[0104] Additionally, the math problem provided to the first user terminal (200) may include a problem constructed by modifying a model answer (correct solution). For example, the computing device (100) may intentionally change some steps of the solution process of the model answer, or modify some of the calculation formulas, conditions, or logical developments to reconstruct the problem into a form containing partial errors. Such modified problems are intended to induce the learner to review the solution process step by step to find the line where the first error occurred or to determine that there are no errors. Additionally, the user may directly input first user input information in the form of a descriptive answer to this math problem.

[0105]

[0106] According to one embodiment of the present disclosure, a computing device (100) can analyze first user input information related to the solution of a mathematical problem by utilizing a pre-trained artificial intelligence model (S120). For example, the computing device (100) can generate multiple solution items by dividing the process of solving a mathematical problem into steps. The computing device (100) can generate multiple solution items by dividing the first user input information related to the solution of a mathematical problem into sentence units, formula units, or logical development units. Such step division is performed by considering the flow of mathematical operations or logical dependencies within the sentence, and each item can be identified as a single logical unit.

[0107] For each of the generated multiple solution items, the artificial intelligence model can evaluate whether the answer is correct, logical validity, and the appropriateness of concept application at each step of the mathematical problem-solving process, and calculate a confidence score for the analysis results. Additionally, the computing device (100) can analyze the causal relationships or sequential dependencies between solution items to track the impact of an error at a specific step on subsequent solution steps. For example, if a calculation error occurs initially, it can determine whether all subsequent operations depend on it and lead to an incorrect result. Meanwhile, during this analysis process, the computing device (100) can improve the accuracy of the analysis by referencing existing verified solution data, intentionally modified model solution data, or solution data from other users, and comparing them with identical or similar solution structures. Through this, the computing device (100) can structurally analyze the first user's solution and quantitatively evaluate the logical consistency and mathematical validity at each step, thereby providing reliable judgment-based data for subsequent editing steps.

[0108] Alternatively, the computing device (100) may use a pre-trained artificial intelligence model to analyze the step-by-step validity of a solution to a math problem for first user input information received from a first user terminal (200) in order to verify the solution. The artificial intelligence model may have reduced reliability in analysis if the user's solution is not written in the same format as the model answer, if the logical explanation expressed in descriptive form deviates from formula-centered data, or if the model has not learned a type of problem (e.g., non-standard approach, creative solution, mixed unit problem, etc.). In such cases, the artificial intelligence model calculates a confidence score for the analysis result of each step, and if the score is below a preset threshold, it determines that it is difficult to guarantee the accuracy of the analysis result.

[0109] When the reliability of the AI ​​model's judgment is low, the computing device (100) provides first user input information to the second user terminal (300) in a learning format so that multiple users can directly review the solution and select or input the step where an error occurred (first incorrect line), the type of error (calculation error, lack of understanding of concepts, etc.), or the direction of improvement, thereby supplementing the AI ​​model's uncertain judgment through a collective verification process. The computing device (100) can solve the problem of insufficient reliability of AI-only judgment by combining the intuitive judgment of human users with the area where the AI ​​model judged it to be uncertain, and can determine statistically significant correction results by utilizing the agreement of responses from multiple users (majority voting, decision tree, etc.), and can gradually improve the model's judgment accuracy by utilizing the verified results again as training data for the AI ​​model.

[0110] The computing device (100) can simultaneously improve the feedback reliability and learning efficiency of the entire system by automatically operating a human-in-the-loop based on the judgment reliability of the artificial intelligence model.

[0111]

[0112] According to one embodiment of the present disclosure, a computing device (100) may provide first user input information related to the solution of a mathematical problem to a second user terminal when the reliability score of the analysis result of an artificial intelligence model is less than a preset threshold (S130). The computing device (100) may obtain a reliability score for each of a plurality of solution items calculated by the artificial intelligence model. If the reliability score for each of the plurality of solution items is lower than the preset threshold, the computing device (100) may determine that the artificial intelligence model is not confident in the correctness or logical validity of the solution. In this case, the computing device (100) may provide (distribute) the first user's solution information to a plurality of second user terminals (300) so that other users or reviewers can review it.

[0113] The second user terminal (300) may include at least one other user terminal that performs a solution to a mathematical problem. The computing device (100) can implement a cooperative learning environment based on peer review among users by converting the solution data submitted by the first user into a learning format and providing it to the second user terminal (300). Multiple users possessing the second user terminal (300) can review the presented solution process and select the first step where an error is determined to exist or an item corresponding to no error. Meanwhile, in the embodiment described below, the second user terminal (300) may be at least one other user terminal that performs a solution to a mathematical problem. The at least one other user terminal that performs a solution to a mathematical problem may be used interchangeably with the second user terminal (300) or multiple user terminals.

[0114] Additionally, the second user terminal (300) may include a terminal of an inspector who performs professional inspection of the math problem solution. The inspector's terminal may review response data collected from the first user terminal (200) or at least one other user terminal that performs the math problem solution, or perform an intervention function to finally confirm the correction results of the AI ​​model. The inspector can ensure the quality and reliability of the correction results by verifying discrepancies between the correction results produced by the AI ​​and the judgment results of the users, and by inputting correction data if necessary. The second user terminal (300) may function not merely as a simple response device, but as a multi-layered verification node that directly participates in the process of supplementing the reliability of the AI ​​model and refining the training data, including the roles of the user and the inspector.

[0115] The computing device (100) may provide solutions to users selected based on criteria such as unit-specific learning achievement, response reliability, or past evaluation accuracy, rather than randomly selecting a second user terminal (300). For example, it may provide solutions only to users whose achievement level in the unit is above a certain level, or adjust the reliability of the response results by assigning weights proportional to each user's achievement level.

[0116] The computing device (100) performs a suitability verification module to verify whether personal information, profanity, or mathematically irrelevant content is included during the distribution process, thereby filtering the solution data provided to the second user so that it is not unsuitable for learning purposes. As a result, the configuration according to the present embodiment supplements the uncertain judgment of the artificial intelligence model with the collective judgment of multiple users to increase the reliability of the correction results and minimize the possibility of errors in the artificial intelligence model alone.

[0117]

[0118] According to one embodiment, the computing device (100) can verify the suitability of the data included in each of the plurality of solution items for each of the solution items. The computing device (100) can verify the suitability of the data included in each of the solution items for each of the plurality of solution items in order to provide the first user input information to the second user terminal (300) in a learning form. The computing device (100) can determine whether the first user information (the plurality of solution items) received from the first user terminal (200) contains personal information, profanity, or sentences unrelated to mathematical solutions. Through this, the solution data provided to other users can be filtered in advance so that it aligns with educational purposes and does not contain inappropriate information.

[0119] Additionally, the computing device (100) can determine whether multiple solution items included in the first user input information can be used as normal learning data by comparing them with previously verified problem, solution, and correction data. For example, the computing device (100) can verify whether the first user solution shows the same formula structure or logical development by matching each step of the model answer in the existing database with the first user solution, or check the consistency and logic of the solution by comparing it with standard solution data in which errors have been intentionally inserted.

[0120]

[0121] According to one embodiment, a computing device (100) can select a plurality of user terminals to provide mathematical problems and a plurality of solution items. For example, the computing device (100) can select user terminals among the plurality of user terminals whose learning achievement level regarding the mathematical problem is above a preset standard. For example, the computing device (100) can select user terminals among the plurality of user terminals whose achievement level regarding the same unit or learning topic as the mathematical problem is above a preset standard by referring to the profile data and learning history data of a first user stored in advance. The computing device (100) can calculate the learning achievement level of each user by unit based on factors such as the correct answer rate, solution time, concept understanding evaluation score, or past correction accuracy. The computing device (100) may evaluate the above factors in the form of a weight by combining them. In addition, if a plurality of users belong to the same achievement level range or response results are derived at similar rates, the computing device (100) can derive a statistically significant judgment result by correcting the response results by reflecting a weight corresponding to the learning achievement level of each of the plurality of users. The computing device (100) can improve the quality of the response by selecting users to provide solutions to math problems for the first user input information based on metadata such as unit achievement, past evaluation accuracy, and learning reliability, rather than simply distributing them to users who perform solutions to arbitrary math problems.

[0122] Meanwhile, if the computing device (100) is unable to calculate reliable statistical values ​​because the number of respondents is insufficient below a certain level, it can expand the range of eligible users to select at least one additional user terminal that performs the solution to a new math problem until the sample size exceeds a preset threshold. According to this configuration, the computing device (100) can improve the quality and reliability of the correction results through a user selection and weight reflection mechanism based on learning achievement.

[0123]

[0124] According to one embodiment, a computing device (100) may provide mathematical problems and multiple solution items to a plurality of selected user terminals. The computing device (100) may provide mathematical problems and multiple solution items whose suitability has been verified to a plurality of selected user terminals. Additionally, the computing device (100) may filter data containing personal information, profanity, or mathematically irrelevant content and provide only training data whose suitability has been verified. For example, referring to FIG. 5, the computing device (100) may provide mathematical problems (20) and multiple solution items (30) to each of the plurality of selected user terminals. The computing device (100) may confirm through a suitability verification module that the solution data obtained from a first user does not contain personal information, profanity, or mathematically irrelevant content, and then configure a learning-type interface based on the verified data and provide it to a plurality of user terminals (300). When a computing device (100) provides a math problem (20) and multiple solution items (30), it may also provide problem instructions (10) to induce thinking training centered on 'finding the incorrect line' rather than a descriptive solution method. The problem instructions (10) guide the user not to simply write a solution, but to analyze the presented solution process step by step to find the line where the first error occurred. By providing the problem instructions (10), the computing device (100) allows multiple users to logically review the solution for each step and select the first step where an error exists, or 'no incorrect line' if there is no error.

[0125] The computing device (100) may provide multiple error type items for selecting an error type for each of the multiple solution items. When displaying the solution items for each step, the computing device (100) may provide multiple error type items so that the user can specify the type of error that occurred at that step, rather than simply selecting "incorrect line." In this process, the error type classification can be used as meta-feedback data to improve the reliability of the artificial intelligence model. If multiple users make a consistent judgment regarding the same solution step as "calculation error" or "lack of conceptual understanding," the computing device (100) may statistically aggregate this to strengthen the reliability of the correction results.

[0126] According to one embodiment, a computing device (100) may obtain second user input information that selects the first error solution item among a plurality of solution items. For example, referring to FIG. 6, one of a plurality of second user terminals (300) may display a plurality of solution items (31–36) divided into steps along with a presented mathematical problem (20). Each solution item (31–36) corresponds to a specific step of the mathematical problem-solving process and may be composed of process steps such as setting conditions for the problem, transforming expressions, substituting numbers, and calculating results. At least one of a plurality of users possessing a second user terminal (300) may select the first solution item among the presented plurality of solution items (31–36) that is determined to have an error. At this time, the user may make a judgment based on the mathematical validity, logical connectivity, or calculation accuracy of each step, and the selected item may be recorded as an error point.

[0127] Additionally, the computing device (100) may obtain second user input information in which an item corresponding to "no error" is selected. This means that multiple second users have determined that the solution for all steps is logically valid, and if multiple users select "no error" in the same way, the computing device (100) may confirm the solution as the correct solution.

[0128] For example, referring to FIG. 7, the computing device (100) may provide a plurality of error type items for selecting an error type for each of the plurality of solution items. The plurality of error type items may include a selectable error type item (301) and a descriptive error type item (302). The selectable error type item (301) may be configured to select any one of the predefined error types (e.g., calculation error, lack of conceptual understanding, lack of logical basis, etc.). On the other hand, the descriptive error type item (302) may be configured to allow the user to directly input text for an unpredefined cause of error or direction for improvement. For example, instead of simply selecting "calculation error," the user may input a specific reason and direction for correction, such as "incorrect variable substitution during the application of the greatest common divisor."

[0129] The computing device (100) calculates an error judgment ratio or a degree of agreement for each solution item based on selection information collected from at least one of the second user terminals (300), and if there is a step that multiple users judged as an error in the same way, it can be determined as an error step. Conversely, if multiple users select the "no incorrect lines" item (37), the solution can be determined as the correct solution.

[0130] The computing device (100) can statistically classify the type and frequency of errors for each solution item by comprehensively analyzing multiple-choice and descriptive error type data received from multiple user terminals. In addition, by utilizing these analysis results as additional training data for an artificial intelligence model, the accuracy of the AI's correction judgment in similar problem types can be improved in the future. The computing device (100) can induce participation among users and structure various forms of error recognition data to continuously improve the reliability and adaptability of AI-based correction. By having the multiple-choice items (301) contribute to aggregation based on quantitative statistics and the descriptive items (302) contribute to the accumulation of qualitative training data, the accuracy and explainability of the AI ​​correction model can be increased simultaneously.

[0131] Alternatively, the computing device (100) may process the input as a partial match if the input information obtained from the second user terminal (300) does not match the "first error solution item," that is, if the learner selects a subsequent step (e.g., the second incorrect line) rather than the first error. The computing device (100) may calculate the degree to which the user's selection item is close to the actual error section and assign a partial score. For example, if the actual error occurred in the third line among multiple solution items but the learner selected the fourth line, the system may assign a partial score by "determining it as a logically adjacent error detection" or present a sentence such as "need to review the operation or condition of the previous step" as feedback information. Additionally, this data may be classified as a "close error detection case" during artificial intelligence model training and utilized to improve the accuracy of the error location prediction model.

[0132] The computing device (100) may process the data as an invalid response if it detects unclear input, such as when a response is not entered within a certain time from one of the multiple user terminals or when multiple items are selected simultaneously. If invalid responses accumulate from one of the multiple user terminals (e.g., 2-1 user terminal), the computing device (100) may lower the reliability of the response accuracy for the same user (e.g., a user possessing the 2-1 user terminal) or reassign the problem to another learner or reviewer to correct the result.

[0133]

[0134] According to one embodiment, if the computing device (100) cannot finalize the correction result even after undergoing a collective verification process by an artificial intelligence model and a plurality of user terminals (300), or if the result is not finalized by a designated time for providing the correction result, it can automatically assign the content to the terminal of a validator performing the verification of the math problem solution and proceed with the verification. If the consistency ratio of the judgment of the correction result or the sum of the response weights is below a preset confidence standard, the computing device (100) can automatically assign the problem to a human validator. The validator can view the solution of the first user, the judgment results of multiple learners, and the analysis results of the AI ​​model together through a management interface within the system, and can finalize or modify the final correction result by synthesizing this information. If the result is ambiguous or statistically insufficient even after undergoing both automatic correction by the artificial intelligence model and user-based collective verification, the computing device (100) can perform a three-stage correction procedure that finally involves the judgment of a human worker. The computing device (100) can form a multi-layered verification loop leading from AI to user to inspector to ensure reliability, accountability, and quality stability for the editing results.

[0135] According to one embodiment of the present disclosure, a computing device (100) can generate correction information for a solution to a math problem based on second user input information obtained from a second user terminal (300) (S140). The computing device (100) can statistically analyze error judgment results collected from a plurality of user terminals to calculate the judgment consistency for each solution item. For example, if multiple users select the same type of error or determine that an error occurred at the same stage for the same solution item, the computing device (100) can determine the item as an error stage. Conversely, if multiple users select the item "no incorrect lines," the solution can be determined as the correct solution.

[0136] According to one embodiment, the computing device (100) may calculate a judgment consistency for the first error solution item among the plurality of solution items that has an error, or calculate a judgment consistency for an item corresponding to no error, by considering a plurality of second user input information obtained from a plurality of user terminals. The computing device (100) may statistically aggregate the second user input information from each of the second user terminals (300). The computing device (100) may derive a judgment consistency ratio by calculating the ratio of users who made the same judgment regarding a plurality of solution items. For example, if 80% or more of the total respondents judged a solution item of a specific stage (e.g., the fifth solution item (35)) to have an “error”, the computing device (100) may determine that item as the first error solution stage. Conversely, if the majority of users selected “no error” for a plurality of solution items, the computing device (100) may determine the entire solution as the correct solution.

[0137] Alternatively, the computing device (100) may correct the judgment agreement by assigning different weights to each response, taking into account the learning achievement, past response reliability, and problem difficulty suitability of each of the multiple second user terminals (300). Through this, a weighted agreement reflecting the level of expertise of the user can be calculated, rather than a statistical result based on simple frequency.

[0138] The computing device (100) can generate correction information for math problem-solving items by considering the calculated judgment agreement. For example, items with a high judgment agreement may be confirmed as highly reliable errors, and items with a low agreement may be marked as “AI review” or “additional learning needed,” thereby assigning a step-by-step reliability indicator to the correction result. Additionally, by analyzing error types selected by multiple users (e.g., calculation errors, misunderstanding of concepts, etc.) together, the correction information may include error types, representative incorrect answer patterns, directions for improvement, etc. The computing device (100) can go beyond simply identifying the location of errors and generate correction information by integrating and reflecting the judgment agreement and response reliability of the user group, thereby deriving a corrective correction result that has higher accuracy and reliability than correction performed by an AI model alone.

[0139] The computing device (100) can increase the reliability of the correction results through statistical judgment based on majority voting among users and can generate corrected correction results that reflect the judgments of multiple users even when the reliability score of the AI ​​model is low. The computing device (100) can implement a collective intelligence-based correction framework that combines automatic correction and collaborative learning verification. The computing device (100) can aggregate multiple user responses and calculate the degree of judgment agreement through majority voting or decision tree logic, and can determine the location of the error or whether it is the correct answer when this degree of agreement is above a certain standard. Through this, the computing device (100) can generate corrected correction results that reflect the judgments of multiple users even when the judgment reliability of a single AI model is low.

[0140]

[0141] According to one embodiment of the present disclosure, if the reliability score of the analysis result of the artificial intelligence model is above a preset threshold, the computing device (100) may not provide the first user input information related to the solution of the math problem to the second user terminal, and may generate correction information for the math problem solution item based on the analysis result of the artificial intelligence model. The computing device (100) may perform detailed analysis, such as determining correct or incorrect answers, classifying error types, and recommending improvement directions, for each of the multiple solution items produced by the pre-trained artificial intelligence model. If the reliability score of the analysis result of the artificial intelligence model is higher than the threshold, the computing device (100) may perform an auto-correction procedure without human verification or the participation of multiple users. For example, if the artificial intelligence model has secured high accuracy based on a sufficiently large amount of training data for the same type of math problem (e.g., solving linear equations, finding the greatest common divisor, etc.), the computing device (100) may confirm the result of the model as correction information. Accordingly, the system can reduce unnecessary user feedback requests and improve the speed of user response processing.

[0142] According to one embodiment of the present disclosure, a computing device (100) can utilize the generated annotation results as training data for an artificial intelligence model. The computing device (100) can convert a plurality of response information collected from a second user terminal (300) and annotation results determined based thereon (e.g., error location, error type, improvement direction, etc.) into data and utilize them for retraining or fine-tuning of the artificial intelligence model. In this process, the computing device (100) can ensure the quality of the training data by selecting only data with a high degree of judgment agreement among a plurality of user responses, or by adopting only annotation results with a high reliability score as training targets. Additionally, the computing device (100) can correct problem types or solution patterns that showed low reliability in the previous training cycle while integrating the newly generated annotation results into an existing training data set. For example, if judgment errors in an AI model frequently occur in specific math units (e.g., multiplication of fractions, quadratic equations, etc.), the model's accuracy can be improved by intensively training it with verification results from multiple users collected in those units.

[0143] FIG. 8 is a user interface showing an operation in which a learner, who has been provided with a solution in which the ground truth has already been determined according to one embodiment of the present disclosure, receives immediate feedback based on the result of their choice.

[0144] For example, referring to FIG. 8, a computing device (100) can display problem-solving data, for which the correct answer has already been determined, on a first user terminal (200) that has received the data in a learning format. The problem has a Ground Truth (correct solution) that exists in advance, and the learner can be guided to select the first line containing an error among the solution items for each step based on this. When the item selected by the learner is transmitted from the first user terminal (200), the computing device (100) can immediately determine whether the selected solution item is correct or incorrect by comparing it with the previously determined correct answer item (Ground Truth) and display the result in the form of visual feedback.

[0145] For example, if the learner correctly selects the line where the first error actually occurred (e.g., the 5th line), the computing device (100) can visually highlight the line (e.g., green mark) with a feedback phrase such as “You have selected correctly” to inform the learner that it is the correct answer. On the other hand, if the learner selects an incorrect line (e.g., if the 6th line is selected even though the first error is the 5th line), the computing device (100) can determine that the learner's selection is inconsistent with the correct solution and provide a feedback phrase such as "There is an error before the selected line" to induce the learner to review the previous step where the error actually occurred. Additionally, if the learner selects the item "Solution without incorrect lines," the computing device (100) can check whether there is an actual error in the Ground Truth and, if there is no actual error, display feedback such as "It was a solution without incorrect lines." However, if the learner selects "No incorrect lines" even though an error exists, the device can output feedback such as "It has been confirmed as a solution with errors" to guide the learner to re-examine the basis of their judgment.

[0146] According to this configuration, the computing device (100) can implement an immediate feedback learning process by utilizing problem-solving data for which the correct answer has already been verified, so that the learner can directly determine the error. Therefore, instead of simply memorizing the correct answer, the learner can repeatedly perform training to analyze and correct the location and cause of errors within the solution process on their own, and the computing device (100) can continuously accumulate statistics on the learner's problem understanding and types of incorrect answers based on this data.

[0147]

[0148] FIG. 9 is a user interface for any one of the operations of generating correction information for a mathematical problem-solving item by considering the calculated judgment agreement according to one embodiment of the present disclosure.

[0149] For example, referring to FIG. 9, when the reliability score of the analysis result calculated by the artificial intelligence model is below a preset threshold, the computing device (100) cannot immediately determine the correction result based solely on the first user input information related to the solution of the math problem. In this case, the computing device (100) may be configured to distribute the solution items generated based on the first user input information to a plurality of second user terminals (300) to aggregate the error judgment results through the judgment of other users. At this time, the correct / incorrect result is not immediately displayed on the first user terminal (200), and the user may be notified that the current correction result is in a waiting state for external user judgment by outputting a notice such as “Gathering thoughts from other learners” as in FIG. 9.

[0150] The computing device (100) collects error item selection information or “no incorrect line” item selection information received from a plurality of second user terminals (300), and may continue the data collection procedure until a predetermined judgment agreement criterion (e.g., majority agreement, weighted statistical criterion, etc.) is reached. When the judgment agreement is secured at or above the criterion value, the computing device (100) may finalize the final correction information based on the collected judgment results and provide the finalized correction information to the first user terminal (200). After the result is finalized, the screen displayed on the first user terminal (200) may be configured in the same form as the correct answer feedback screen or incorrect answer feedback screen described above in FIG. 8.

[0151] For example, the computing device (100) can produce highly reliable correction results by utilizing the collective judgment of multiple users even when the analysis reliability of the artificial intelligence model is insufficient. Through this, it is possible to provide stable and accurate correction results compared to existing systems that rely on a single model prediction value, and at the same time, the effect of improving data quality based on the participation of learners and multiple users can also be obtained.

[0152]

[0153] The steps mentioned in the foregoing description may be further subdivided into additional steps or combined into fewer steps, depending on the embodiment of the present disclosure. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0154]

[0155] Meanwhile, a computer-readable medium storing a data structure is disclosed according to an embodiment of the present disclosure.

[0156] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data to solve specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of the device's resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0157] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0158] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form loops in a graph data structure.

[0159] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, the term neural network will be used consistently. A data structure may include a neural network. Furthermore, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the configurations described above, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated during the computational process of the neural network, and is not limited to the foregoing. A computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0160] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0161] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0162] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0163] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0164] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0165]

[0166] FIG. 10 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0167] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0168] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0169] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0170] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0171] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0172] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1104).

[0173] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0174] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the floppy disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0175] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0176] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0177] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0178] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0179] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0180] When used in a LAN networking environment, the computer (1102) is connected to the LAN (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means of establishing communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via an input device interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0181] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0182] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0183] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0184] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0185] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0186] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0187] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0188] As described above, the relevant details have been described in the best mode for carrying out the invention.

Claims

1. A method for generating annotation information for the solution of a mathematical problem, performed by a computing device, A step of obtaining first user input information related to the solution of a mathematical problem from a first user terminal; A step of analyzing the first user input information related to the solution of the above mathematical problem using a pre-trained artificial intelligence model; If the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold, the step of providing the first user input information related to the solution of the mathematical problem to the second user terminal; and A step of generating correction information for the solution of the math problem based on second user input information obtained from the second user terminal; including, method.

2. In Paragraph 1, The step of analyzing the first user input information related to the solution of the math problem using the aforementioned pre-trained artificial intelligence model is: A step of generating multiple solution items by dividing the solution process of the above mathematical problem into steps. including, method.

3. In Paragraph 2, The above-mentioned second user terminal is, At least one other user terminal that performs the solution to the above mathematical problem; or A terminal of an inspector performing an inspection of the solution to the above math problem; including, method.

4. In Paragraph 3, If the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold, the step of providing the first user input information related to the solution of the math problem to the second user terminal is A step of selecting a plurality of user terminals to provide the above mathematical problem and the above plurality of solution items; and A step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals; including, method.

5. In Paragraph 4, The step of selecting a plurality of user terminals to provide the above mathematical problem and the above plurality of solution items is, A step of selecting user terminals among the plurality of user terminals whose learning achievement level regarding the math problem is above a preset standard. including, method.

6. In Paragraph 4, The step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals is, For each of the above plurality of solution items, the method includes the step of providing a plurality of error type items for selecting an error type, and The above multiple error type items are, Selectable error type item configured to select any one of predefined error types; and Descriptive error type item configured to input non-predefined error types as text including, method.

7. In Paragraph 4, The step of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals is, For each of the above plurality of solution items, a step of verifying the suitability of the data included in each solution item. including, method.

8. In Paragraph 2, The step of generating correction information for the solution of the math problem based on the second user input information obtained from the second user terminal is: A step of obtaining second user input information that selects the first error solution item existing among the plurality of solution items; or Step of obtaining second user input information by selecting an item corresponding to no error including at least one of, method.

9. In Paragraph 8, The step of generating correction information for the solution of the math problem based on the second user input information obtained from the second user terminal is, A step of calculating a judgment agreement for the first error solution item having an error among the plurality of solution items, or calculating a judgment agreement for an item corresponding to having no error, by considering a plurality of second user input information obtained from a plurality of user terminals; and A step of generating correction information for the above math problem-solving item by considering the judgment agreement calculated above. including, method.

10. In Paragraph 8, The above method is, Step of utilizing the generated editing results as training data for the artificial intelligence model including, method.

11. In Paragraph 3, The above method is, If the reliability score of the analysis result of the artificial intelligence model is greater than or equal to a preset threshold, the first user input information related to the solution of the math problem is not provided to the second user terminal, and the analysis result of the artificial intelligence model generates correction information for the math problem solution item. including, method.

12. A computer program stored on a computer-readable storage medium, wherein, when the computer program is executed on one or more processors, the one or more processors are configured to perform the following operations to generate annotation information for the solution of a mathematical problem, and said operations are: An operation to obtain first user input information related to the solution of a mathematical problem from a first user terminal; An operation of analyzing the first user input information related to the solution of the above mathematical problem using a pre-trained artificial intelligence model; An operation of providing the first user input information related to the solution of the mathematical problem to the second user terminal when the confidence score of the analysis result of the artificial intelligence model is less than a preset threshold; and Operation of generating correction information for the solution of the math problem based on second user input information obtained from the second user terminal. including, A computer program stored on a computer-readable storage medium.

13. In Paragraph 12, The operation of analyzing the first user input information related to the solution of the math problem using the aforementioned pre-trained artificial intelligence model is, The operation of generating multiple solution items by dividing the solution process of the above mathematical problem into steps. including, A computer program stored on a computer-readable storage medium.

14. In Paragraph 13, The above-mentioned second user terminal is, At least one other user terminal that performs the solution to the above mathematical problem; or A terminal of an inspector performing an inspection of the solution to the above math problem; including, A computer program stored on a computer-readable storage medium.

15. In Paragraph 13, When the reliability score of the analysis result of the artificial intelligence model is less than a preset threshold, the operation of providing the first user input information related to the solution of the math problem to the second user terminal is, An operation of selecting a plurality of user terminals to provide the above mathematical problem and the above plurality of solution items; and The operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals; including, A computer program stored on a computer-readable storage medium.

16. In Paragraph 15, The operation of selecting multiple user terminals to provide the above mathematical problem and the above multiple solution items is, The operation of selecting user terminals among the plurality of user terminals whose learning achievement level regarding the math problem is above a preset standard. including, A computer program stored on a computer-readable storage medium.

17. In Paragraph 15, The operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals is, For each of the above plurality of solution items, the operation includes providing a plurality of error type items for selecting an error type, and The above multiple error type items are, Selectable error type item configured to select any one of predefined error types; and Descriptive error type item configured to input non-predefined error types as text including, A computer program stored on a computer-readable storage medium.

18. In Paragraph 15, The operation of providing the mathematical problem and the plurality of solution items to the selected plurality of user terminals is, For each of the above plurality of solution items, an operation to verify the suitability of the data included in each solution item. including more, A computer program stored on a computer-readable storage medium.

19. In Paragraph 13, The operation of generating correction information for the solution of the math problem based on the second user input information obtained from the second user terminal is, An operation to obtain second user input information that selects the first error solution item existing among the plurality of solution items above; or The action of obtaining second user input information that selects an item corresponding to no errors including at least one of, A computer program stored on a computer-readable storage medium.

20. In Paragraph 19, The operation of generating correction information for the solution of the math problem based on the second user input information obtained from the second user terminal is, An operation of calculating a judgment agreement for the first error solution item having an error among the plurality of solution items, or calculating a judgment agreement for an item corresponding to having no error, by considering a plurality of second user input information obtained from a plurality of user terminals; and The operation of generating correction information for the above math problem-solving item by considering the judgment agreement calculated above. including, A computer program stored on a computer-readable storage medium.

21. As a computing device, At least one processor; and Memory; Includes, The above-mentioned at least one processor is, Obtaining first user input information related to the solution of a mathematical problem from a first user terminal; Analyzing the first user input information related to the solution of the above mathematical problem using a pre-trained artificial intelligence model; If the confidence score of the analysis result of the artificial intelligence model is below a preset threshold, the first user input information related to the solution of the mathematical problem is provided to the second user terminal; and Configured to generate correction information for the solution of the math problem based on second user input information obtained from the second user terminal, device.

22. In Paragraph 21, The above-mentioned at least one processor is, A method configured to generate multiple solution items by dividing the solution process of the above mathematical problem into steps, device.

23. In Paragraph 22, The above-mentioned second user terminal is, At least one other user terminal that performs the solution to the above mathematical problem; or A terminal of an inspector performing an inspection of the solution to the above math problem; including, device.

24. In Paragraph 23, The above-mentioned at least one processor is, Selecting multiple user terminals to provide the above mathematical problem and the above plurality of solution items; and Configured to provide the mathematical problem and the plurality of solution items to the selected plurality of user terminals, device.

25. In Paragraph 24, The above-mentioned at least one processor is, A configuration for selecting user terminals among the plurality of user terminals whose learning achievement level regarding the math problem is above a preset standard, device.

26. In Paragraph 24, The above-mentioned at least one processor is, For each of the above plurality of solution items, it is configured to provide a plurality of error type items for selecting an error type, and The above multiple error type items are, Selectable error type item configured to select any one of predefined error types; and Descriptive error type item configured to input non-predefined error types as text including, device.

27. In Paragraph 24, The above-mentioned at least one processor is, For each of the above plurality of solution items, additionally configured to verify the suitability of the data included in each solution item, device.

28. In Paragraph 22, The above-mentioned at least one processor is, An operation to obtain second user input information that selects the first error solution item existing among the plurality of solution items above; or The action of obtaining second user input information that selects an item corresponding to no errors including at least one of, device.

29. In Paragraph 28, The above-mentioned at least one processor is, Considering multiple second user input information obtained from multiple user terminals, calculate a judgment agreement for the first error solution item among the multiple solution items that has an error, or calculate a judgment agreement for the item corresponding to having no error; and Configured to generate correction information for the above math problem-solving item by considering the above-calculated judgment agreement, device.

Images