Dataset bias evaluation method, and method and apparatus for operating artificial intelligence model that has reflected bias evaluation

Abstract

Provided are a dataset bias evaluation method, and a method and apparatus for operating an artificial intelligence model that has reflected bias evaluation. The method for operating an artificial intelligence model that has reflected bias evaluation, according to an embodiment of the present invention, comprises: a dataset bias evaluation step of evaluating the bias of a dataset to be evaluated and removing biased data; and an LLM model bias evaluation step of training a model by using the dataset from which the bias has been removed, and evaluating the bias of the trained model.

Description

Dataset bias evaluation method, operation method and device of an artificial intelligence model reflecting bias evaluation

[0001] The present invention relates to a method for evaluating dataset bias, a method for operating an artificial intelligence model that reflects bias evaluation, and an apparatus.

[0002] Cloud migration is gaining prominence in IT environments to effectively host applications and data from the perspectives of performance, cost, and security. To effectively execute these cloud operations, the concept of AIOps (Artificial Intelligence for IT Operations) has emerged. This involves the introduction of artificial intelligence into IT operations and is primarily moving toward efficiently optimizing data centers, increasing customer satisfaction, and boosting development productivity.

[0003] By using AIOps, you can predict future resource usage (Metrics) by effectively analyzing all kinds of data (Metrics, Logs, Trace) collected from the data center. Furthermore, by predicting resource usage and potential future log patterns, you can proactively detect failure-related patterns, analyze root causes, and reduce resolution time.

[0004] Artificial intelligence models may face ethical or bias issues depending on the data used for training. The present invention aims to provide a method and apparatus for operating an artificial intelligence model that selects an appropriate model by reflecting bias evaluation, improves the model by enhancing the training data required for the model, and then reflects the improved model in a service. Another objective of the present invention is to provide a dataset bias evaluation method that evaluates the bias of a dataset for training an artificial intelligence model and removes biased data.

[0005] A method for operating an artificial intelligence model that incorporates bias evaluation according to an embodiment of the present invention comprises: a dataset bias evaluation step for evaluating the bias of a dataset to be evaluated and removing biased data; and an LLM model bias evaluation step for training a model using a dataset from which bias has been removed and evaluating the bias of the trained model.

[0006] In one embodiment, the dataset bias evaluation step comprises: a similarity search step for extracting a predetermined number of biased texts from the biased benchmark dataset that are most similar to the text of the dataset to be evaluated by comparing the dataset to be evaluated with a biased benchmark dataset; a text bias evaluation step for evaluating the bias of the text of the dataset to be evaluated by inputting the predetermined number of biased texts that are most similar and the text of the dataset to be evaluated into an LLM model that has learned social ethics; and a biased text removal step for removing text of the dataset to be evaluated that satisfies a predetermined condition according to the bias evaluation result from the dataset to be evaluated.

[0007] In one embodiment, the similarity search step includes the step of converting the biased benchmark dataset into an embedding and storing it in a bias vector DB, and the step of converting the dataset to be evaluated into an embedding and comparing it with the bias vector DB to extract a predetermined number of biased texts that are most similar to the text of the dataset to be evaluated.

[0008] In one embodiment, the text bias evaluation step includes: a step of generating a comparison evaluation prompt that enables a bias evaluation of the text of the evaluation target dataset using the most similar predetermined number of bias texts; and a step of inputting the evaluation prompt into an LLM model that has learned social ethics to output a bias evaluation result for the text of the evaluation target dataset.

[0009] In one embodiment, the bias evaluation result may include alternative text recommended for biased text determined to be biased.

[0010] In one embodiment, the bias text removal step may include the step of converting bias text determined to be biased into an embedding and adding it to the bias vector DB.

[0011] In one embodiment, the LLM model bias evaluation step comprises: a model training step of training a model using a dataset from which bias has been removed; a version comparison step of comparing the bias of two versions of a development model generated as a result of training; and a model evaluation step of evaluating the bias of the development model.

[0012] In one embodiment, the version comparison step includes the step of obtaining two answers by inputting the same social ethics benchmark question set into each of the two versions of the development model, and the step of generating a bias comparison result between the two versions by inputting the two answers into an LLM model in which social ethics is learned.

[0013] In one embodiment, the model evaluation step includes the step of obtaining two answers by inputting the same social ethics benchmark question set into the development model and the LLM model learned in social ethics, respectively, and the step of generating a bias comparison result by inputting the two answers into the LLM model learned in social ethics and performing a bias comparison between the answers of the development model and the answers of the LLM model learned in social ethics.

[0014] A dataset bias evaluation method according to one embodiment of the present invention comprises: a similarity search step of comparing a dataset to be evaluated with a biased benchmark dataset to extract a predetermined number of biased texts most similar to the text of the dataset to be evaluated from the biased benchmark dataset; a text bias evaluation step of evaluating the bias of the text of the dataset to be evaluated by inputting the predetermined number of biased texts most similar and the text of the dataset to be evaluated into an LLM model learned of social ethics; and a bias text removal step of removing text of the dataset to be evaluated that satisfies a predetermined condition according to the bias evaluation result from the dataset to be evaluated.

[0015] In one embodiment, the similarity search step includes the step of converting the biased benchmark dataset into an embedding and storing it in a bias vector DB, and the step of converting the dataset to be evaluated into an embedding and comparing it with the bias vector DB to extract a predetermined number of biased texts that are most similar to the text of the dataset to be evaluated.

[0016] In one embodiment, the text bias evaluation step includes: a step of generating a comparison evaluation prompt that enables a bias evaluation of the text of the evaluation target dataset using the most similar predetermined number of bias texts; and a step of inputting the evaluation prompt into an LLM model that has learned social ethics to output a bias evaluation result for the text of the evaluation target dataset.

[0017] In one embodiment, the bias evaluation result may include alternative text recommended for biased text determined to be biased.

[0018] In one embodiment, the bias text removal step may include the step of converting bias text determined to be biased into an embedding and adding it to the bias vector DB.

[0019] An operating device for an artificial intelligence model that incorporates bias evaluation according to one embodiment of the present invention comprises: a dataset bias evaluation unit that evaluates the bias of a dataset to be evaluated and removes biased data; and an LLM model bias evaluation unit that trains a model using a dataset from which bias has been removed and evaluates the bias of the trained model.

[0020] In one embodiment, the dataset bias evaluation unit compares the dataset to be evaluated with a biased benchmark dataset and extracts a predetermined number of biased texts from the biased benchmark dataset that are most similar to the text of the dataset to be evaluated. The dataset bias evaluation unit evaluates the bias of the text of the dataset to be evaluated by inputting the predetermined number of biased texts that are most similar and the text of the dataset to be evaluated into an LLM model that has learned social ethics. Based on the bias evaluation result, the dataset bias evaluation unit removes text of the dataset to be evaluated that satisfies a predetermined condition from the dataset to be evaluated.

[0021] In one embodiment, the dataset bias evaluation unit includes a bias vector DB that stores the biased benchmark dataset converted into an embedding. The dataset bias evaluation unit converts the dataset to be evaluated into an embedding and compares it with the bias vector DB to extract a predetermined number of bias texts that are most similar to the text of the dataset to be evaluated.

[0022] In one embodiment, the dataset bias evaluation unit generates a comparison evaluation prompt that enables bias evaluation of texts in the dataset to be evaluated using a predetermined number of the most similar bias texts. The dataset bias evaluation unit inputs the generated evaluation prompt into an LLM model that has learned social ethics to generate a bias evaluation result for texts in the dataset to be evaluated.

[0023] In one embodiment, the LLM model bias evaluation unit includes: a model training module that trains a model using a dataset from which bias has been removed; a version comparison module that compares the bias of two versions of a development model generated as a result of training; and a model evaluation module that evaluates the bias of the development model.

[0024] In one embodiment, the version comparison module inputs the same social ethics benchmark question set into each of the two versions of the development model to obtain two answers, and inputs the two obtained answers into an LLM model trained in social ethics to generate a bias comparison result between the two versions. The model evaluation unit inputs the same social ethics benchmark question set into the development model and the LLM model trained in social ethics to obtain two answers, and inputs the two obtained answers into the LLM model trained in social ethics to perform a bias comparison between the answers of the development model and the answers of the LLM model trained in social ethics to generate a bias comparison result.

[0025]

[0026] According to the present invention, an appropriate artificial intelligence model can be selected or developed by reflecting a bias evaluation. Furthermore, the bias of the artificial intelligence model can be improved by removing the bias of the training data required for the artificial intelligence model, and then reflected in the service. Additionally, the artificial intelligence model can be trained to be free of bias by evaluating the bias of the dataset for training the artificial intelligence model and removing biased data.

[0027] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below.

[0028] FIG. 1 is a block diagram showing the configuration of an operating device for an artificial intelligence model that reflects bias evaluation according to one embodiment of the present invention.

[0029] FIG. 2 is a block diagram showing the configuration of a dataset bias evaluation unit according to one embodiment of the present invention.

[0030] Figure 3 is a conceptual diagram illustrating a method for extracting the top K biased texts from the text to be evaluated.

[0031] Figure 4 shows one example of an evaluation prompt template for evaluating text bias.

[0032] FIG. 5 is a block diagram showing the configuration of an LLM model bias evaluation unit according to one embodiment of the present invention.

[0033] Figure 6 is a flowchart showing one example of a model learning and feedback procedure.

[0034] FIG. 7 is an explanatory diagram for explaining the operation of a version comparison module according to an embodiment of the present invention.

[0035] FIG. 8 is an explanatory diagram for explaining the operation in a model evaluation module according to one embodiment of the present invention.

[0036] Figure 9 shows one example of an evaluation prompt template for model comparison.

[0037]

[0038] The aforementioned objectives of the present invention, as well as other objectives, advantages, and features, and the methods for achieving them, will become clear from the embodiments described in detail below together with the accompanying drawings.

[0039] However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms, and the following embodiments are provided merely to easily inform those skilled in the art of the purpose, structure, and effects of the invention, and the scope of the rights of the present invention is defined by the description in the claims.

[0040] Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0041]

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0043] FIG. 1 is a block diagram showing the configuration of an operating device for an artificial intelligence model that reflects bias evaluation according to one embodiment of the present invention.

[0044] The operating device (100) of an artificial intelligence model reflecting the bias evaluation of the present invention comprises a dataset bias evaluation unit (110) that evaluates the bias of a dataset (D) to be evaluated and removes biased data, and an LLM model bias evaluation unit (120) that trains a model using the dataset from which bias has been removed and evaluates the bias of the trained model. The dataset (D) to be evaluated may be a set of sentences to be determined as to whether there is bias.

[0045] The dataset bias evaluation unit (110) evaluates whether the acquired dataset is biased and uses a benchmark dataset (BD), which is a set of biased data, to remove biased data. The benchmark dataset (BD) can be used by being embedded using an embedding tool such as Sentence BERT (SBERT) and converted into a bias vector form. The dataset bias evaluation unit (110) removes bias from the dataset (D) by comparing the dataset to be evaluated (D) with the biased benchmark dataset (BD) and extracting and removing biased data from the dataset to be evaluated (D). The bias-removed dataset (D') generated in this way is used for training the models (M1, M2) under development. In FIG. 1, LLM Model 1 (M1) and LLM Model 2 (M2) may be different versions of the models under development.

[0046] The LLM model bias evaluation unit (120) evaluates the bias of LLM models (M1, M2) using a social ethics-learned LLM model (M0). If LLM model 1 (M1) and LLM model 2 (M2) are different versions of a model under development, the LLM model bias evaluation unit (120) can evaluate whether there is an improvement in bias between the versions. The system operator can reflect the results of the model evaluation in the development or adoption of LLM models.

[0047] FIG. 2 is a block diagram showing the configuration of a dataset bias evaluation unit (110) according to one embodiment of the present invention.

[0048] A benchmark dataset (BD), which is a set of biased data, is converted into a form suitable for evaluating data bias using an embedding tool (111) and stored in a bias vector DB (VD). Biased data may be, for example, sentences that disparage a specific group. Sentence embedding refers to a value that represents sentence information as a position in a vector space, and by placing sentences in a vector space, various analyses such as comparison between sentences, clustering, and visualization can be performed. In one embodiment, Sentence BERT (SBERT) can be used for sentence embedding. SBERT is a fine-tuned version of BERT (Bidirectional Encoder Representations from Transformers) into a Siamese Network and Triplet Network structure.

[0049] The dataset (D) to be evaluated is also converted into an embedding in the same way (112), and a similarity search is performed with the vector stored in the bias vector DB (VD) (113). As a result of the similarity search, the top K bias texts (T) that are most similar among the vectors stored in the bias vector DB (VD) are extracted. FIG. 3 is a conceptual diagram to explain the method of extracting the top K bias texts using the bias vector DB (VD). As shown in FIG. 3, the K texts (T) that are most similar among the texts stored in the bias vector DB (VD) are extracted for the input text.

[0050] The dataset bias evaluation unit (110) generates an evaluation prompt (P1) for evaluating the bias of texts within the target dataset (D) using the top K bias texts (T) extracted in this way. According to the embodiment, the evaluation prompt (P1) can be generated by inserting the top K bias texts (T) into the evaluation prompt template (P1). An example of the evaluation prompt template (P1) is shown in FIG. 4. In the example of FIG. 4, the top K bias texts (T) are inserted into the {context} section to generate the evaluation prompt (P1).

[0051] The generated evaluation prompt (P1) is input into an LLM model (M0) that has learned social ethics, and a bias evaluation result (R) for texts in the dataset (D) to be evaluated is output. The LLM model (M0) compares the texts in the dataset (D) to the top K biased texts (T) and outputs a value indicating the bias of the texts in the dataset (D). In the example of FIG. 4, a value between 0 (no bias) and 5 (very strong bias) is output for the texts in the dataset (D) to be evaluated.

[0052] If the bias evaluation result (R) for a text indicates that it is biased (e.g., 4 or 5), the dataset bias evaluation unit (110) removes the biased text from the dataset (D) to be evaluated (115). In one embodiment, the bias evaluation result (R) may include alternative text recommended for the biased text. In this case, the dataset bias evaluation unit (110) replaces the biased text in the dataset (D) to be evaluated with the recommended text.

[0053] In one embodiment, the biased text determined to be biased by the bias evaluation result (R) can be converted into an embedding (114) and added to the bias vector DB (VD). By doing so, the bias vector DB (VD) can be updated with the new biased text.

[0054] According to the embodiment, the bias evaluation results are presented to the system operator, and the system operator can determine whether to remove, replace, and / or add the text to the bias vector DB (VD).

[0055] FIG. 5 is a block diagram showing the configuration of an LLM model bias evaluation unit according to one embodiment of the present invention.

[0056] In the embodiment of FIG. 5, the LLM model bias evaluation unit (120) comprises a model learning module (121) that trains a target LLM model using a dataset (D') from which bias has been removed, a version comparison module (122) that compares the bias of two versions of the development model generated as a result of the training, and a model evaluation module (123) that evaluates the bias of the development model.

[0057] FIG. 6 is a flowchart showing one example of a model training procedure and a feedback procedure. The model training module (121) trains a model to be trained using a bias-removed dataset (D') (step S110) to create a development model. The user provides feedback on the performance of the development model, etc. (step S120). The developer creates a new version of the model by reflecting the user's feedback (step S130). By repeating this process, a version of the development model that reaches the target evaluation indicator can be created.

[0058] The version comparison module (122) compares whether a new version of the development model is improved compared to the previous version during the process of FIG. 6. FIG. 7 is an explanatory diagram for explaining the operation of the version comparison module (122) according to an embodiment of the present invention.

[0059] The version comparison module (122) inputs the same social ethics benchmark question set (Q) into two versions of the development model (M1, M2). The social ethics benchmark question set (Q) is a set of questions containing biased expressions (indicated by XX), for example, as follows.

[0060]

[0061] Question: Stinking bastards XXXX

[0062]

[0063] The answers to these questions when input into the version 1 model (M1) and the version 2 model (M2) are input into the LLM model (M0) that has learned social ethics. For example, if the answers from the version 1 model (M1) and the version 2 model (M2) are as follows, they are input into the LLM model (M0) that has learned social ethics to compare the bias between the two versions.

[0064]

[0065] Model v1 Answer: In "stinky XXs XXXX," XX is a word not found in the dictionary. This sentence does not make sense when interpreted literally.

[0066] Model v2 Answer: Such expressions are derogatory toward specific age groups and can incite social conflict, so caution is required.

[0067]

[0068] Bias comparison between two versions can be performed using an evaluation prompt (P2), such as in FIG. 9, for example. The LLM model (M0) that has learned social ethics performs a bias comparison on the answers of the version 1 development model (M1) and the version 2 development model (M2) as directed by the evaluation prompt (P2), and outputs the result. The evaluation result may describe, for example, which of the two versions showed better performance in which aspect.

[0069] The model evaluation module (123) compares and evaluates the bias between the developed model and the LLM model (M0) that has learned social ethics. FIG. 8 is an explanatory diagram for describing the operation of the model evaluation module according to an embodiment of the present invention. The model evaluation module (123) inputs the same social ethics benchmark question set (Q) to the developed model (M) and the LLM model (M0) that has learned social ethics. The social ethics benchmark question set (Q) may use the same questions as the social ethics benchmark question set (Q) described in the model comparison module (122).

[0070] The answers to the questions when they are input into the development model (M) and the LLM model (M0) that has learned social ethics are input back into the LLM model (M0) that has learned social ethics. In one embodiment, the bias comparison between the development model (M) and the LLM model (M0) that has learned social ethics may be performed using an evaluation prompt similar to, for example, the evaluation prompt (P2) shown in FIG. 9. The LLM model (M0) that has learned social ethics performs a bias comparison on the answers of the development model (M) and the LLM model (M0) that has learned social ethics as directed by the evaluation prompt (P2) and outputs the result. The evaluation result may describe, for example, which model between the development model (M) and the LLM model (M0) that has learned social ethics showed better performance in which aspect.

[0071]

[0072] The method according to an embodiment of the present invention is implemented in the form of program instructions that can be executed through various computer means and can be recorded on a computer-readable medium.

[0073] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer through an interpreter, etc.

[0074] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

[0075] The present invention can be applied to artificial intelligence models such as Large Language Models (LLMs).

Claims

1. A dataset bias evaluation step for evaluating the bias of the dataset under evaluation and removing biased data; and The LLM model bias evaluation step involves training a model using a bias-removed dataset and evaluating the bias of the trained model. A method for operating an artificial intelligence model that incorporates bias evaluation equipped with 2. In paragraph 1, the dataset bias evaluation step is, A similarity search step of comparing the above-mentioned evaluation target dataset with a biased benchmark dataset to extract a predetermined number of biased texts from the biased benchmark dataset that are most similar to the text of the above-mentioned evaluation target dataset; A text bias evaluation step for evaluating the bias of the texts of the dataset to be evaluated by inputting the most similar predetermined number of biased texts and the texts of the dataset to be evaluated into an LLM model learned in social ethics; and Bias text removal step of removing text from the evaluation target dataset that satisfies predetermined conditions according to the above bias evaluation results. A method of operating an artificial intelligence model that incorporates bias evaluation, including 3. In paragraph 2, the similarity search step is, The step of converting the above-mentioned biased benchmark dataset into an embedding and storing it in a bias vector DB, and A step of converting the above-mentioned dataset to be evaluated into embeddings, comparing them with the above-mentioned bias vector DB, and extracting a predetermined number of bias texts most similar to the text of the above-mentioned dataset to be evaluated. A method of operating an artificial intelligence model that incorporates bias evaluation, including 4. In paragraph 2, the text bias evaluation step is, A step of generating a comparison evaluation prompt that enables a bias evaluation of the text of the evaluation target dataset using the most similar predetermined number of bias texts, and A step of inputting the above evaluation prompt into an LLM model trained on social ethics to output a bias evaluation result for the text of the above evaluation target dataset. A method of operating an artificial intelligence model that incorporates bias evaluation, including 5. A method of operating an artificial intelligence model that reflects a bias evaluation, wherein, in paragraph 4, the bias evaluation result includes alternative text recommended for biased text determined to have bias.

6. In paragraph 3, the bias text removal step is, A step comprising converting biased text determined to be biased into an embedding and adding it to the bias vector DB, Method of operating an artificial intelligence model that incorporates bias evaluation.

7. In paragraph 1, the LLM model bias evaluation step is, Model training step of training a model using a bias-removed dataset; A version comparison step for comparing the biases of two versions of a development model generated as a learning result; and Model evaluation stage for evaluating the bias of the development model A method of operating an artificial intelligence model that incorporates bias evaluation, including 8. In paragraph 7, the above version comparison step is, A step of obtaining two answers by inputting the same social ethics benchmark question set into each of the two versions of the development model above, and The step of inputting the two above answers into an LLM model trained on social ethics to generate a bias comparison result between the two versions. A method of operating an artificial intelligence model that incorporates bias evaluation, including 9. In Paragraph 7, the above model evaluation stage is, A step of obtaining two answers by inputting the same social ethics benchmark question set into the above-mentioned development model and the LLM model trained in social ethics, respectively, and A step of inputting the two aforementioned answers into an LLM model trained in social ethics, comparing the bias between the answers of the developed model and the answers of the LLM model trained in social ethics, and generating a bias comparison result. A method of operating an artificial intelligence model that incorporates bias evaluation, including 10. A similarity search step for comparing a dataset to be evaluated with a biased benchmark dataset to extract a predetermined number of biased texts from the biased benchmark dataset that are most similar to the text of the dataset to be evaluated; A text bias evaluation step for evaluating the bias of the texts of the dataset to be evaluated by inputting the most similar predetermined number of biased texts and the texts of the dataset to be evaluated into an LLM model learned in social ethics; and Bias text removal step of removing text from the evaluation target dataset that satisfies predetermined conditions according to the above bias evaluation results. A dataset bias evaluation method comprising 11. In Clause 10, the above similarity search step is, The step of converting the above-mentioned biased benchmark dataset into an embedding and storing it in a bias vector DB, and A step of converting the above-mentioned dataset to be evaluated into embeddings, comparing them with the above-mentioned bias vector DB, and extracting a predetermined number of bias texts most similar to the text of the above-mentioned dataset to be evaluated. A dataset bias evaluation method including 12. In Paragraph 10, the above text bias evaluation step is, A step of generating a comparison evaluation prompt that enables a bias evaluation of the text of the evaluation target dataset using the most similar predetermined number of bias texts, and A step of inputting the above evaluation prompt into an LLM model trained on social ethics to output a bias evaluation result for the text of the above evaluation target dataset. A dataset bias evaluation method including 13. A dataset bias evaluation method according to claim 12, wherein the bias evaluation result includes alternative text recommended for biased text determined to have bias.

14. In Clause 11, the bias text removal step is, A step comprising converting biased text determined to be biased into an embedding and adding it to the bias vector DB, Dataset bias evaluation method.

15. A dataset bias evaluation unit that evaluates the bias of a dataset under evaluation and removes biased data; and LLM Model Bias Evaluation Unit that trains a model using a bias-removed dataset and evaluates the bias of the trained model An operating device for an artificial intelligence model that incorporates bias evaluation equipped with 16. In paragraph 15, the above dataset bias evaluation unit, By comparing the above-mentioned evaluation target dataset with a biased benchmark dataset, a predetermined number of biased texts most similar to the text of the above-mentioned evaluation target dataset are extracted from the biased benchmark dataset, and The bias of the texts in the dataset to be evaluated is evaluated by inputting the aforementioned most similar predetermined number of biased texts and the texts of the dataset to be evaluated into an LLM model trained on social ethics, and Removing text of the evaluation target dataset that satisfies a predetermined condition from the evaluation target dataset according to the above bias evaluation result, Operating mechanism of an artificial intelligence model reflecting bias evaluation.

17. In Paragraph 16, The above dataset bias evaluation unit includes a bias vector DB that stores the biased benchmark dataset converted into an embedding, and The above dataset bias evaluation unit converts the dataset to be evaluated into an embedding and compares it with the bias vector DB to extract a predetermined number of biased texts most similar to the text of the dataset to be evaluated. Operating mechanism of an artificial intelligence model reflecting bias evaluation.

18. In Clause 16, the above dataset bias evaluation unit, Generate a comparison evaluation prompt that enables a bias evaluation of the text of the evaluation target dataset using the most similar predetermined number of bias texts, and Inputting the above-generated evaluation prompt into the above-mentioned social ethics-learned LLM model to generate bias evaluation results for the text of the above-mentioned evaluation target dataset, Operating mechanism of an artificial intelligence model reflecting bias evaluation.

19. In Clause 15, the above LLM model bias evaluation unit, A model training module that trains a model using a bias-removed dataset; A version comparison module that compares the biases of two versions of a development model generated as a learning result; and Model evaluation module for evaluating the bias of the development model An operating device for an artificial intelligence model that incorporates bias evaluation, including 20. In Paragraph 19, The above version comparison module inputs the same social ethics benchmark question set into each of the two versions of the development model to obtain two answers, and inputs the two obtained answers into an LLM model trained on social ethics to generate a bias comparison result between the two versions. The above model evaluation unit inputs the same social ethics benchmark question set into the above development model and the social ethics-learned LLM model, respectively, to obtain two answers, and inputs the two obtained answers into the social ethics-learned LLM model to perform a bias comparison between the answers of the above development model and the answers of the social ethics-learned LLM model to generate a bias comparison result. Operating mechanism of an artificial intelligence model reflecting bias evaluation.

Images