Big data analysis-based method and system for ai model performance evaluation
By constructing task graphs and generating optimization prompt templates through big data analysis, the adaptability of existing AI model performance evaluation methods in different task scenarios and the integration of user feedback are solved. This enables comprehensive evaluation and dynamic optimization of AI models, improving model execution quality and data security.
Patent Information
- Application Number
- PCT/CN2025/131527
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-14
- Filing Date
- 2025-10-31
- Publication Date
- 2026-11-19
AI Technical Summary
Existing AI model performance evaluation methods rely on manually set evaluation criteria and limited datasets, which cannot fully cover the model's performance in different tasks and application scenarios, ignore the model's adaptability when facing unknown tasks, and fail to fully incorporate user feedback and problems in practical applications, making it difficult to achieve dynamic optimization.
Through big data analysis, the UMAP algorithm is used for dimensionality reduction and clustering to construct a task graph. Graph neural networks are used to extract task risk scores, generate optimization prompt templates, and combine adversarial networks and reinforcement learning to generate multiple optimization prompt templates. Similarity and semantic matching are calculated, user feedback is monitored, and log records are encrypted and stored for continuous optimization.
It enables comprehensive evaluation of AI model performance, ensuring the model's adaptability to different tasks and application scenarios. It also allows for dynamic optimization based on user feedback, improving the model's execution quality and stability, and ensuring data security and traceability.
Smart Images

Figure CN2025131527_19112026_PF_FP_ABST
Abstract
Description
A method and system for evaluating the performance of AI models based on big data analytics
[0001] This application claims priority to Chinese patent application No. CN202510614972.8, filed on May 14, 2025, entitled “A Method and System for Evaluating the Performance of AI Models Based on Big Data Analysis”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This disclosure relates to the field of big data analytics technology, specifically, to the design of a method and system for evaluating the performance of AI models based on big data analytics. Background Technology
[0003] AI models, especially deep learning models, learn underlying patterns by processing large amounts of data and are used to perform various tasks. However, how to effectively evaluate the performance of these AI models and ensure their continuous optimization has always been an important issue in research and practice. With the rapid development of big data technology, data-driven AI performance evaluation methods have gradually become mainstream. Through big data analysis, multiple factors can be considered to evaluate the generalization ability, accuracy, and stability of AI models, thereby providing support for model improvement and optimization. Existing AI performance evaluation methods usually rely on standard evaluation datasets or task-specific evaluation metrics.
[0004] However, traditional methods often rely on manually set evaluation criteria and limited datasets, which cannot fully cover the performance of models in different tasks and application scenarios. In addition, many existing evaluation methods fail to effectively consider the diversity and complexity of tasks, ignore the adaptability of models when facing unknown tasks, and current AI model evaluation methods also fail to fully incorporate user feedback and problems encountered in practical applications, making it difficult to achieve dynamic optimization based on user interaction. Summary of the Invention
[0005] According to some embodiments, in a first aspect, this disclosure provides a method for evaluating the performance of AI models based on big data analysis, comprising:
[0006] The task text data is obtained for the target AI model. The word segmentation is performed by the embedding model to output text vectors. The UMAP algorithm is used to reduce the dimensionality of the task vectors and perform clustering to obtain task clusters. The cosine similarity between task text vectors is calculated to construct a task graph. The task risk score is extracted by graph neural network. Prompt templates are created for learning and optimization. Different optimized prompt templates are generated by adversarial network.
[0007] Based on the optimized prompt template, calculate the Cosine similarity and Jaccard similarity between the target task model output and the standard output, filter the optimized prompt template, perform BERT encoding on the target task model output text and the standard output text respectively, calculate the semantic matching degree and quantify the score, perform LCS comparison on the model output text and the standard text, generate the text quality dataset, and perform cluster divergence calculation on different time windows based on BERT encoding to provide divergence warning.
[0008] Obtain user feedback records, calculate the n-gram overlap between the erroneous text output and the standard text, and calculate the sliding deviation of the word distribution based on the frequency of occurrence of tag words, and issue score warnings and deviation warnings respectively;
[0009] Generate log records for regular analysis, encrypt and store encrypted data items, and back up data through secure transmission.
[0010] In one optional embodiment of this disclosure, the generation of different optimization prompt templates includes,
[0011] Based on the query task text dataset of the target AI model, data cleaning was performed and word segmentation was carried out using the NLTK library;
[0012] The text for each task is dimensionality-reduced using the pre-trained embedding model BERT to embed the text into text vectors, and the UMAP algorithm is used to reduce the dimensionality of the task vectors.
[0013] The K-means clustering algorithm is used to cluster the dimensionality-reduced vectors to obtain task clusters. Each cluster contains a set of semantically similar task text vectors. The cosine similarity between task text vectors is calculated as a measure of the similarity between tasks.
[0014] Based on the similarity between tasks and task clusters, a task graph is constructed using the graph database Neo4j. Each task in a task cluster is treated as a node, and the edge weights between nodes are the similarity between tasks. A graph neural network (GNN) is used to pass information between nodes through graph convolution operations to update the feature representation of each node.
[0015] The node representation is updated layer by layer through multi-layer convolution. The risk score of each task is output through a fully connected layer. The tasks are prioritized in descending order based on the risk score. The task objectives of the task clusters are clarified, and prompt templates are created, including task objectives, input examples, and output formats.
[0016] Using the reinforcement learning (RL) algorithm, the prompts for the task cluster are initialized with a standard prompt template. The intelligent agent generates model output based on the initial prompts and interacts with the input and output requirements of the task cluster as the environment. The accuracy and completeness of the output text compared with the standard output are calculated. The prompt template is adjusted according to the reward function. The intelligent agent iterates through multiple interactions with the environment until the changes in accuracy and completeness are no longer significant. Then the iteration stops and an optimized prompt template is generated.
[0017] Using the Generative Adversarial Network (GAN) algorithm, adversarial training is performed between the generator and the discriminator to generate multiple versions of optimized prompt templates.
[0018] In one optional embodiment of this disclosure, the quality dataset of the generated text is used for divergence warning, including:
[0019] Based on the target task model input optimization prompt template, the text similarity value between the output text and the standard output is calculated using the Jaccard similarity algorithm. The completeness score between the output text and the standard output is calculated using the Cosine similarity algorithm. The weights of the text similarity value and the completeness score are adaptively adjusted using historical task data, and the quality score is calculated by weighted summation. The quality score threshold is calculated based on the sum of the historical mean and twice the standard deviation of the quality score. If the quality score is less than or equal to the quality score threshold, the corresponding optimization prompt template is removed.
[0020] Using a pre-trained BERT Score model, the text output by the target task model and the standard output text are encoded using BERT. The similarity between the encoded text vectors is calculated, and each word is compared to obtain word-by-word similarity. The word-by-word similarity is weighted and averaged to obtain the overall output similarity score. The semantic matching degree between the output text and the standard text is quantitatively scored.
[0021] Using the ROUGE-L tool, the LCS of the model output text and the standard text is compared to find the longest common subsequence. Based on the length and matching degree of the LCS, the recall, precision and F1 score are calculated to obtain a dataset that comprehensively evaluates the quality of the generated text.
[0022] The target task model is labeled by generating an evaluation signature vector based on the quality dataset and quantitative scores;
[0023] Based on BERT encoding, the input BERT encoding within a time window is clustered to obtain the probability distribution of each input belonging to each cluster center. For the cluster distribution of two time windows, the change is calculated using KL divergence.
[0024] A trigger threshold is set based on historical experience. If the divergence between two time windows is greater than or equal to the trigger threshold, re-verification is performed and the evaluation signature vector is updated.
[0025] In one optional embodiment of this disclosure, the step of performing score warning and deviation warning includes:
[0026] Implement real-time feedback ratings from users who are monitoring target task models, identify user feedback ratings based on preset standards, identify user-marked error text output content, and calculate the n-gram overlap between error text output content and standard text;
[0027] Group label recognition is performed based on user attributes. The frequency of occurrence of different label words in the output text of the target task model is calculated, and the deviation change of the continuous window is calculated as the sliding deviation of the word distribution.
[0028] The score threshold and deviation threshold are respectively based on the sum of the historical mean and standard deviation of the BLEU score and the sliding deviation. If the BLEU score is greater than or equal to the score threshold and the sliding deviation is greater than or equal to the deviation threshold, a deviation warning is triggered.
[0029] In one optional embodiment of this disclosure, the generation of log records is periodically analyzed, including:
[0030] Determine the user query text data for deviation warning and the text output data of the target model, and record the timestamp, model version and alarm information. Use a log generation tool to generate log files, use ELK Stack to store the logs and perform regular analysis on the log files.
[0031] In one optional embodiment of this disclosure, the encrypted storage of the encrypted data item includes:
[0032] Identify the encrypted data items, including log files, model inputs and outputs, and any related user information, and securely encrypt the data items using the AES-256 encryption algorithm;
[0033] Role-based access control (RBAC) mechanisms are used to set up audit logs that record the user, time, and operation type of each operation, ensuring the traceability of the data access process and performing data anonymization and desensitization on user personal data in input and output.
[0034] In one optional embodiment of this disclosure, the data backup via secure transmission includes:
[0035] Regularly back up encrypted data items, audit logs, and log files, use TLS / SSL protocols for secure transmission, and perform offline environment backups.
[0036] Secondly, this disclosure provides a system for evaluating the performance of AI models based on big data analysis, including:
[0037] The data segmentation module segments the text data of the target task into words, converts the text into vector representations using an embedding model, reduces the dimensionality of the task vectors using the UMAP algorithm, and clusters the data to calculate task clusters.
[0038] The task graph construction module calculates the cosine similarity between task text vectors, constructs the task graph, and further extracts the risk score of the task.
[0039] The prompt template optimization module generates different optimized prompt templates through an adversarial network, and learns and optimizes the target task model based on the optimized prompt templates;
[0040] The model output comparison module calculates the similarity between the target task model output and the standard output and scores them.
[0041] The text quality assessment module compares the quality dataset of the generated text.
[0042] The clustering divergence warning module calculates clustering divergence for different time windows based on BERT encoding and provides divergence warnings.
[0043] The user feedback module calculates the n-gram overlap between the erroneous text output and the standard text, analyzes the sliding deviation of word distribution, and provides score warnings and deviation warnings.
[0044] The log backup module generates log records for periodic analysis and encrypts and stores encrypted data items, backing up data through secure transmission.
[0045] Thirdly, this disclosure provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the AI model performance evaluation method based on big data analysis as described in the first aspect of this disclosure.
[0046] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the AI model performance evaluation method based on big data analysis as described in the first aspect of this disclosure.
[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0048] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 is a flowchart illustrating the AI model performance evaluation method based on big data analysis in Example 1.
[0050] Figure 2 is a schematic diagram of the AI model performance evaluation system based on big data analysis in Example 1. Detailed Implementation
[0051] To make the above-mentioned objectives, features and advantages of this disclosure more apparent and understandable, the specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.
[0052] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure. However, this disclosure may be implemented in other ways different from those described herein, and those skilled in the art may make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific embodiments disclosed below.
[0053] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this disclosure. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0054] Example 1, referring to Figures 1 and 2, is the first embodiment of this disclosure. This embodiment provides a method for evaluating the performance of AI models based on big data analysis, including the following steps:
[0055] S1. Obtain task text data for the target AI model, perform word segmentation processing through the embedding model to output text vectors, use the UMAP algorithm to reduce the dimensionality of the task vectors and perform clustering calculations to obtain task clusters, calculate the cosine similarity between task text vectors to construct a task graph, use graph neural networks to extract task risk scores, create prompt templates for learning and optimization, and generate different optimized prompt templates through adversarial networks.
[0056] Preferably, different optimization prompt templates are generated, including:
[0057] Based on the query task text dataset of the target AI model, data cleaning was performed and word segmentation was carried out using the NLTK library;
[0058] The text for each task is transformed into a text vector by using the pre-trained embedding model BERT to embed the text into an embedding vector dimension.
[0059] The UMAP algorithm is used to reduce the dimensionality of task vectors (using strategies of local and global preservation to reduce the dimensionality of high-dimensional data). It preserves the relative distance and similarity between task vectors, ensuring that semantically similar task vectors remain close in the low-dimensional space, while semantically different task vectors are separated.
[0060] The K-means clustering algorithm is used to cluster the dimensionality-reduced vectors to obtain task clusters. Each cluster contains a group of semantically similar task text vectors.
[0061] Calculate the cosine similarity between task text vectors as a measure of task similarity;
[0062] Based on task similarity and task clusters, a task graph is constructed using the graph database Neo4j. Each task within a task cluster is treated as a node, and the edge weights between nodes represent the task similarity. A graph neural network (GNN) is used to update the feature representation of each node by passing information between nodes through graph convolution operations. Each graph convolution operation is represented as follows:
[0063] in, Let represent the convolutional representation of task i at layer L, σ represent the activation function, and N(i) and N(j) represent the neighboring nodes of task i and j, respectively. Let b represent the convolutional representation of task j at layer L. (L) S represents the bias term of the Lth convolution layer. ij This represents the similarity between task i and task j;
[0064] The node representation is updated layer by layer according to the multi-layer convolution, and the risk score of each task is output through the fully connected layer. The tasks are then sorted in descending order of priority based on the risk score.
[0065] Define the task objectives of the task cluster (e.g., the objective of the "text classification" cluster is to assign text to the most appropriate category, and the objective of the "text generation" cluster is to generate relevant text based on given prompts), and create prompt templates that include task objectives (a clear task description based on the task objectives of the task cluster), input examples (a unified input example designed based on the task cluster), and output format (provide a standard output format requirement for each task to ensure that the output structure of all tasks is consistent).
[0066] Using the reinforcement learning (RL) algorithm, the prompts for the task cluster are initialized with a standard prompt template. The intelligent agent generates model output based on the initial prompts and interacts with the input and output requirements of the task cluster as the environment. The accuracy and completeness of the output text compared with the standard output are calculated. The prompt template is adjusted according to the reward function. The intelligent agent iterates through multiple interactions with the environment until the changes in accuracy and completeness are no longer significant. Then the iteration stops and an optimized prompt template is generated.
[0067] Using Generative Adversarial Networks (GANs) algorithms, multiple diverse versions of prompt templates are generated by a generator based on reinforcement learning-based optimized prompt templates. A discriminator judges whether the generated templates are accurate, complete, and operable. The generator and discriminator are trained adversarially (the generator continuously adjusts the generated prompt templates, while the discriminator continuously improves its ability to recognize the generated templates, ensuring that the generated templates meet certain quality standards). Multiple different versions of optimized prompt templates are generated.
[0068] Word segmentation makes each text data more structured, laying a solid foundation for subsequent embedding and model training. This ensures that subsequent steps can process each individual word and its grammatical function in the sentence, resulting in more accurate semantic understanding. By embedding the text using a pre-trained BERT model and converting each task text into a vector, powerful semantic information representations can be provided for each task text. The BERT model captures contextual relationships, enabling each text vector to accurately reflect the semantic features of the text, providing rich semantic dimensionality support for subsequent dimensionality reduction and clustering analysis. UMAP, through local and global preservation strategies, ensures that semantically similar task texts remain in the lower-dimensional space after dimensionality reduction. However, task texts that are close in distance but semantically different are separated, ensuring the relative relationship and separability between tasks. By constructing a task graph and updating node information through a graph neural network (GNN), and updating the features of task nodes through graph convolution operations, the feature representation of each task is made more accurate. The graph structure fully considers the interrelationship between tasks. GNN can propagate information between nodes and learn the similarity between tasks, thus providing more informative feature representations for task risk scoring. By utilizing the hierarchy of graph convolution to strengthen the risk assessment of each task, the risk scoring is made more accurate and reliable, providing a priority reference for subsequent task management and optimization.
[0069] By defining clear objectives for each task cluster and creating standardized prompt templates, the uniformity and operability of tasks are ensured. Continuous adjustment of prompts through interaction with the environment ensures that the generated prompt templates achieve optimal accuracy and completeness. The RL algorithm continuously optimizes the generation strategy based on a reward mechanism, enabling prompt templates to better align with task objectives and improve task execution quality. Multiple high-quality optimized prompt templates are generated through generative adversarial networks, providing more diverse inputs for the target task and ensuring execution quality in different task scenarios.
[0070] S2, calculate the Cosine similarity and Jaccard similarity between the target task model output and the standard output based on the optimized prompt template, filter the optimized prompt template, perform BERT encoding on the target task model output text and the standard output text respectively, calculate the semantic matching degree and quantify the score, perform LCS comparison on the model output text and the standard text, generate the text quality dataset, perform clustering divergence calculation on different time windows based on BERT encoding, and perform divergence warning.
[0071] Preferably, a high-quality dataset of generated text is used for divergence warning, including:
[0072] Based on the target task model input optimization prompt template, the text similarity value between the output text and the standard output is calculated using the Jaccard similarity algorithm, and the completeness score between the output text and the standard output is calculated using the Cosine similarity algorithm. (The weights of the text similarity value and the completeness score are adaptively adjusted using historical task data, and the quality score is calculated by weighted summation. The quality score threshold is calculated based on the sum of the historical mean and twice the standard deviation of the quality score. If the quality score is less than or equal to the quality score threshold, the corresponding optimization prompt template is removed.)
[0073] Using a pre-trained BERT Score model, the text output by the target task model and the standard output text are encoded using BERT. The similarity between the encoded text vectors is calculated, and each word is compared to obtain word-by-word similarity. The word-by-word similarity is weighted and averaged to obtain the overall output similarity score. The semantic matching degree between the output text and the standard text is quantitatively scored.
[0074] Using the ROUGE-L tool, the LCS of the model output text and the standard text is compared to find the longest common subsequence. Based on the length and matching degree of the LCS, the recall, precision and F1 score are calculated to obtain a dataset that comprehensively evaluates the quality of the generated text.
[0075] The target task model is labeled by generating an evaluation signature vector based on the quality dataset and quantitative scores;
[0076] Based on BERT encoding, clustering is performed on the input BERT codes within a time window to obtain the probability distribution of each input belonging to each cluster center. For the cluster distributions of two time windows, the change is calculated using KL divergence, expressed as:
[0077] Among them, D(P) t ||P t+1 ) represents the divergence between two time windows, K represents the number of clusters, and P represents the number of clusters. t (k) and P t+1 (k) represent the k-th probability distributions of time windows t and t+1, respectively;
[0078] A trigger threshold is set based on historical experience. If the divergence between two time windows is greater than or equal to the trigger threshold, re-verification is performed and the evaluation signature vector is updated.
[0079] By comparing the optimized prompt template input to the task model with the standard output, and using Jaccard similarity and Cosine similarity to calculate the similarity and completeness of the output text, it is possible to ensure that the model's output maintains consistency in semantic matching with the standard output. Through adaptive adjustment of task data, the evaluation weights can be dynamically adjusted for different task types, thereby improving the accuracy of quality scores. By using the BERT Score model to encode the output of the target task model and the standard output, the text can be transformed into a deep semantic embedding. The quality score of the output is obtained by comparing the similarity between word vectors. By combining the ROUGE-L tool to perform the longest common subsequence (LCS) comparison between the generated text and the standard text, the content similarity of the text can be further quantitatively evaluated. By clustering the input text within two time windows based on BERT encoding and KL divergence calculation, and analyzing the changes in cluster distribution, it is possible to effectively detect changes in the model's input over continuous time periods. By quantifying changes in data, potential drift or pattern changes can be captured.
[0080] By using methods such as Jaccard similarity, Cosine similarity, BERT Score, and ROUGE-L to perform multi-level quality assessment of the model's output text, we can not only accurately measure the similarity, completeness, and semantic matching between the model's output and the standard output, but also ensure that the optimized prompt templates meet the requirements of accuracy and content completeness during the generation process. Combining dimensionality reduction clustering and data drift monitoring with BERT encoding and KL divergence calculation, the model can track changes in the distribution of input data in real time and identify potential drifts based on different features of task clusters. This process provides strong support for the continuous validation of the task model: timely revalidation is triggered when the data distribution changes, ensuring that the model can adapt to changes in input patterns in different time windows and preventing the model performance from degrading due to data bias.
[0081] S3: Obtain user feedback records, calculate the n-gram overlap between the erroneous text output content and the standard text, and calculate the sliding deviation of the word distribution based on the frequency of occurrence of tag words, and issue score warnings and deviation warnings respectively.
[0082] Preferably, score warnings and deviation warnings are implemented, including:
[0083] The real-time feedback ratings of users implementing the monitoring target task model are identified based on preset standards. The error text output content marked by users is identified, and the n-gram overlap between the error text output content and the standard text is calculated, expressed as:
[0084] Where BE represents the BLEU score, pr n The value represents the n-gram precision between the original text and the standard text, and te represents the length of the erroneous text output.
[0085] Group label recognition is performed based on user attributes. The frequency of occurrence of different label words in the output text of the target task model is calculated, and the deviation change of the continuous window is calculated as the sliding deviation of the word distribution.
[0086] The score threshold and deviation threshold are respectively based on the sum of the historical mean and standard deviation of the BLEU score and the sliding deviation. If the BLEU score is greater than or equal to the score threshold and the sliding deviation is greater than or equal to the deviation threshold, a deviation warning is triggered.
[0087] By monitoring real-time user feedback scores for the target task model and calculating the n-gram overlap between erroneous and standard texts using BLEU scores, a quantitative quality assessment of the output generated by each model can be provided. BLEU scores, as an important tool for measuring the similarity between generated and standard texts, can accurately capture the degree of word overlap in the model output. Through n-gram overlap analysis of the erroneous text output, we can identify in which aspects the model is biased and which specific words or phrases caused the model's error. By using user attributes for group label identification, the task model can optimize the output content according to the needs of different groups, especially for task execution for diverse user groups. By calculating the frequency of words related to group labels in the target task model output text and detecting whether there is a bias in text generation based on the deviation changes of continuous windows, we can effectively identify whether the model has output bias or unfairness in certain groups.
[0088] S4 generates log records for periodic analysis, encrypts and stores encrypted data items, and backs up data through secure transmission;
[0089] Preferably, log records are generated for periodic analysis, including:
[0090] Determine the user query text data for deviation warning and the text output data of the target model, and record the timestamp, model version and alarm information. Use a log generation tool to generate log files, use ELK Stack to store the logs and perform regular analysis on the log files.
[0091] By implementing a deviation warning mechanism and recording user query text data and target model output data, the time of each warning and its related information can be accurately tracked. Detailed logs such as timestamps, model versions, and alarm information can be recorded. This not only ensures the accurate location of each deviation warning but also provides important reference data for model performance monitoring and subsequent problem investigation.
[0092] Furthermore, the encrypted data items are stored in encrypted form, including:
[0093] Identify the encrypted data items, including log files, model inputs and outputs, and any related user information, and securely encrypt the data items using the AES-256 encryption algorithm;
[0094] Role-based access control (RBAC) mechanisms are used to set up audit logs that record the user, time, and operation type of each operation, ensuring the traceability of the data access process and performing data anonymization and desensitization on user personal data in input and output.
[0095] By encrypting log files, model inputs and outputs, and related user information using AES-256, the security and confidentiality of this sensitive data can be effectively ensured, preventing unauthorized access or leakage. The AES-256 algorithm provides robust encryption protection, ensuring data security during storage and transmission. Using a role-based access control (RBAC) mechanism ensures that only authorized users can access sensitive data, assigning appropriate permissions according to roles, thereby minimizing data access and reducing potential security vulnerabilities.
[0096] Furthermore, data backup is performed through secure transmission, including,
[0097] Regularly back up encrypted data items, audit logs, and log files, use TLS / SSL protocols for secure transmission, and perform offline environment backups.
[0098] By regularly backing up encrypted data items, audit logs, and log files, it is possible to effectively ensure the rapid recovery of important data in the event of system failure or data loss, thus guaranteeing high system availability. By backing up encrypted data, it is ensured that even if the backup data is stolen, the data remains in an encrypted and protected state, avoiding the risk of leaking sensitive information.
[0099] This embodiment also provides a system for evaluating the performance of AI models based on big data analysis, including:
[0100] The data segmentation module segments the text data of the target task into words, converts the text into vector representations using an embedding model, reduces the dimensionality of the task vectors using the UMAP algorithm, and clusters the data to calculate task clusters.
[0101] The task graph construction module calculates the cosine similarity between task text vectors, constructs the task graph, and further extracts the risk score of the task.
[0102] The prompt template optimization module generates different optimized prompt templates through an adversarial network, and learns and optimizes the target task model based on the optimized prompt templates;
[0103] The model output comparison module calculates the similarity between the target task model output and the standard output and scores them.
[0104] The text quality assessment module compares the quality dataset of the generated text.
[0105] The clustering divergence warning module calculates clustering divergence for different time windows based on BERT encoding and provides divergence warnings.
[0106] The user feedback module calculates the n-gram overlap between the erroneous text output and the standard text, analyzes the sliding deviation of word distribution, and provides score warnings and deviation warnings.
[0107] The log backup module generates log records for periodic analysis and encrypts and stores encrypted data items, backing up data through secure transmission.
[0108] This embodiment also provides a computer device applicable to the AI model performance evaluation method based on big data analysis, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the AI model performance evaluation method based on big data analysis as proposed in the above embodiment.
[0109] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.
[0110] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the AI model performance evaluation method based on big data analysis as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0111] In summary, this disclosure enhances risk assessment for each task by leveraging the hierarchy of graph convolution, making risk scoring more accurate and reliable. This provides a priority reference for subsequent task management and optimization. The RL algorithm continuously optimizes the generation strategy based on a reward mechanism, enabling the prompt templates to better align with task objectives and improve task execution quality. Multiple high-quality optimized prompt templates are generated through generative adversarial networks to ensure execution quality in different task scenarios. Multi-level quality assessment of the model output text is performed using methods such as Jaccard similarity, Cosine similarity, BERTScore, and ROUGE-L. This not only accurately measures the similarity, completeness, and semantic matching between the model output and the standard output but also ensures that the optimized prompt templates meet accuracy and content completeness requirements during generation. By monitoring real-time user feedback scores for the target task model and calculating the n-gram overlap between erroneous and standard texts using BLEU scores, a quantitative quality assessment of the output generated by each model can be provided. By calculating the frequency of words related to group labels in the target task model output text and detecting deviations in text generation based on continuous window changes, it is possible to effectively identify whether the model exhibits output bias or unfairness in certain groups.
[0112] It should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and are not intended to limit it. Although this disclosure has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this disclosure without departing from the spirit and scope of the technical solutions of this disclosure, and all such modifications and substitutions should be covered within the scope of the claims of this disclosure.
Claims
1. A method for evaluating AI model performance based on big data analysis, wherein, include: The task text data is obtained for the target AI model. The word segmentation is performed by the embedding model to output text vectors. The UMAP algorithm is used to reduce the dimensionality of the task vectors and perform clustering to obtain task clusters. The cosine similarity between task text vectors is calculated to construct a task graph. The task risk score is extracted by graph neural network. Prompt templates are created for learning and optimization. Different optimized prompt templates are generated by adversarial network. Based on the optimized prompt template, calculate the cosine similarity and Jaccard similarity between the target task model output and the standard output, filter the optimized prompt template, perform BERT encoding on the target task model output text and the standard output text respectively, calculate the semantic matching degree and quantify the score, perform LCS comparison on the model output text and the standard text, generate the text quality dataset, and perform clustering divergence calculation on different time windows based on BERT encoding to provide divergence warning. Obtain user feedback records, calculate the n-gram overlap between the erroneous text output and the standard text, and calculate the sliding deviation of the word distribution based on the frequency of occurrence of tag words, and issue score warnings and deviation warnings respectively; Generate log records for regular analysis, encrypt and store encrypted data items, and back up data through secure transmission; The generation of different optimization prompt templates includes, Based on the query task text dataset of the target AI model, data cleaning was performed and word segmentation was carried out using the NLTK library; The text for each task is dimensionality-reduced using the pre-trained embedding model BERT to embed the text into text vectors, and the UMAP algorithm is used to reduce the dimensionality of the task vectors. The K-means clustering algorithm is used to cluster the dimensionality-reduced vectors to obtain task clusters. Each cluster contains a set of semantically similar task text vectors. The cosine similarity between task text vectors is calculated as a measure of the similarity between tasks. Based on the similarity between tasks and task clusters, a task graph is constructed using the graph database Neo4j. Each task in a task cluster is treated as a node, and the edge weights between nodes are the similarity between tasks. A graph neural network (GNN) is used to pass information between nodes through graph convolution operations to update the feature representation of each node. The node representation is updated layer by layer through multi-layer convolution. The risk score of each task is output through a fully connected layer. The tasks are prioritized in descending order based on the risk score. The task objectives of the task clusters are clarified, and prompt templates are created, including task objectives, input examples, and output formats. Using the reinforcement learning (RL) algorithm, the prompts for the task cluster are initialized with a standard prompt template. The intelligent agent generates model output based on the initial prompts and interacts with the input and output requirements of the task cluster as the environment. The accuracy and completeness of the output text compared with the standard output are calculated. The prompt template is adjusted according to the reward function. The intelligent agent iterates through multiple interactions with the environment until the changes in accuracy and completeness are no longer significant. Then the iteration stops and an optimized prompt template is generated. Using the Generative Adversarial Network (GAN) algorithm, adversarial training is performed between the generator and the discriminator to generate multiple versions of optimized prompt templates.
2. The AI model performance evaluation method based on big data analysis according to claim 1, wherein, The quality dataset of the generated text is used for divergence warning, including: Based on the target task model input optimization prompt template, the text similarity value between the output text and the standard output is calculated using the Jaccard similarity algorithm. The completeness score between the output text and the standard output is calculated using cosine similarity. The weights of the text similarity value and the completeness score are adaptively adjusted using historical task data, and the quality score is calculated by weighted summation. The quality score threshold is calculated based on the sum of the historical mean and twice the standard deviation of the quality score. If the quality score is less than or equal to the quality score threshold, the corresponding optimization prompt template is removed. Using a pre-trained BERT Score model, the text output by the target task model and the standard output text are encoded using BERT. The similarity between the encoded text vectors is calculated, and each word is compared to obtain word-by-word similarity. The word-by-word similarity is weighted and averaged to obtain the overall output similarity score. The semantic matching degree between the output text and the standard text is quantitatively scored. Using the ROUGE-L tool, the LCS of the model output text and the standard text is compared to find the longest common subsequence. Based on the length and matching degree of the LCS, the recall, precision and F1 score are calculated to obtain a dataset that comprehensively evaluates the quality of the generated text. The target task model is labeled by generating an evaluation signature vector based on the quality dataset and quantitative scores; Based on BERT encoding, the input BERT encoding within a time window is clustered to obtain the probability distribution of each input belonging to each cluster center. For the cluster distribution of two time windows, the change is calculated using KL divergence. A trigger threshold is set based on historical experience. If the divergence between two time windows is greater than or equal to the trigger threshold, re-verification is performed and the evaluation signature vector is updated. 3.The AI model performance evaluation method based on big data analysis of claim 2, wherein, The aforementioned score warning and deviation warning include, Implement real-time feedback ratings from users who are monitoring target task models, identify user feedback ratings based on preset standards, identify user-marked error text output content, and calculate the n-gram overlap between error text output content and standard text; Group label recognition is performed based on user attributes. The frequency of occurrence of different label words in the output text of the target task model is calculated, and the deviation change of the continuous window is calculated as the sliding deviation of the word distribution. The score threshold and deviation threshold are respectively based on the sum of the historical mean and standard deviation of the BLEU score and the sliding deviation. If the BLEU score is greater than or equal to the score threshold and the sliding deviation is greater than or equal to the deviation threshold, a deviation warning is triggered.
4. The AI model performance evaluation method based on big data analysis according to claim 3, wherein, The generated log records are analyzed periodically, including: Determine the user query text data for deviation warning and the text output data of the target model, and record the timestamp, model version and alarm information. Use a log generation tool to generate log files, use ELK Stack to store the logs and perform regular analysis on the log files. 5.The AI model performance evaluation method based on big data analysis of claim 4, wherein, The encrypted storage of the encrypted data items includes, Identify the encrypted data items, including log files, model inputs and outputs, and any related user information, and securely encrypt the data items using the AES-256 encryption algorithm; Role-based access control (RBAC) mechanisms are used to set up audit logs that record the user, time, and operation type of each operation, ensuring the traceability of the data access process and performing data anonymization and desensitization on user personal data in input and output.
6. The AI model performance evaluation method based on big data analysis according to claim 5, wherein, The data backup is performed via secure transmission. include, Regularly back up encrypted data items, audit logs, and log files, use TLS / SSL protocols for secure transmission, and perform offline environment backups.
7. A system for AI model performance evaluation based on big data analysis, based on the AI model performance evaluation method based on big data analysis according to any one of claims 1 to 6, wherein, The system includes: The data segmentation module segments the text data of the target task into words, converts the text into vector representations using an embedding model, reduces the dimensionality of the task vectors using the UMAP algorithm, and clusters the data to calculate task clusters. The task graph construction module calculates the cosine similarity between task text vectors, constructs the task graph, and further extracts the risk score of the task. The prompt template optimization module generates different optimized prompt templates through an adversarial network, and learns and optimizes the target task model based on the optimized prompt templates; The model output comparison module calculates the similarity between the target task model output and the standard output and scores them. The text quality assessment module compares the quality dataset of the generated text. The clustering divergence warning module calculates clustering divergence for different time windows based on BERT encoding and provides divergence warnings. The user feedback module calculates the n-gram overlap between the erroneous text output and the standard text, analyzes the sliding deviation of word distribution, and provides score warnings and deviation warnings. The log backup module generates log records for periodic analysis and encrypts and stores encrypted data items, backing up data through secure transmission.
8. A computer device comprising a memory and a processor, the memory storing a computer program, wherein, When the processor executes the computer program, it implements the steps of the AI model performance evaluation method based on big data analysis as described in any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed by the processor, it implements the steps of the AI model performance evaluation method based on big data analysis as described in any one of claims 1 to 6.
