Large language model-based data processing method and apparatus
By combining a large language model and a comprehensive scoring model, keyword extraction, pseudo-question generation, and evaluation prompt statement generation are constructed, solving the problem of inaccurate scoring in existing technologies and achieving a more efficient automatic scoring effect.
Patent Information
- Application Number
- PCT/CN2026/078306
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-02-10
- Publication Date
- 2026-11-12
AI Technical Summary
In enterprise compliance certification and knowledge examinations, existing technologies have limitations. Rule-based scoring methods are ineffective at handling complex questions and are costly, while scoring methods based on large language models lack in-depth understanding of specific domains, resulting in inaccurate scores.
By combining a large language model and a comprehensive scoring model, we construct a system for extracting keywords, generating pseudo-questions, and generating prompts for evaluation. We leverage the generalization and generation capabilities of the large language model and the vertical scenario learning capabilities of the comprehensive scoring model to determine the text similarity of the respondent's answers and input them into the comprehensive scoring model for scoring.
This improved the accuracy of the scoring by leveraging both the generalization and generation capabilities of the large language model and the vertical scenario learning capabilities of the comprehensive scoring model, resulting in more accurate automatic scoring.
Smart Images

Figure CN2026078306_12112026_PF_FP_ABST
Abstract
Description
A data processing method and apparatus based on a large language model
[0001] This application claims priority to Chinese Patent Application No. 202510602281.6, filed on May 9, 2025, entitled "A Data Processing Method and Apparatus Based on a Large Language Model", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This invention relates to the field of data processing, and more specifically, to a data processing method and apparatus based on a large language model. Background Technology
[0003] In today's business environment, compliance certification and knowledge examinations are crucial for the operations and management of medium and large enterprises. With the advent of the information age, post-certification or examination scoring has shifted from manual to online assessment. Online scoring methods mainly fall into two categories: rule-based scoring and large language model-based scoring. Rule-based scoring is efficient and maintains consistent evaluation criteria, but it struggles with complex questions and incurs high costs for rule construction and maintenance. Large language model-based scoring leverages the powerful semantic understanding capabilities of large language models, but it lacks in-depth knowledge of specific domains and therefore fails to provide a deep understanding of the enterprise's specific domain.
[0004] Therefore, how to achieve more accurate automatic scoring is an urgent problem to be solved. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a data processing method and apparatus based on a large language model, which combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy. This approach leverages the generalization and generation capabilities of the large language model while fully utilizing the vertical scenario learning capabilities of the comprehensive scoring model.
[0006] Firstly, a data processing method based on a large language model is provided, the method comprising:
[0007] Obtain the question set to be scored, which includes the question, the answerer's answer, and the reference answer;
[0008] Based on the question set to be scored and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, respectively construct the corresponding keyword extraction prompt statement, pseudo-question generation prompt statement and evaluation generation prompt statement;
[0009] The keyword extraction prompts, the pseudo-question generation prompts, and the evaluation generation prompts are respectively input into the large language model to obtain the corresponding keyword groups, question groups, and global evaluations;
[0010] Determine the text similarity between the respondent's answer and the reference answer;
[0011] The keyword group, the question group, the global evaluation, and the text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer.
[0012] Secondly, a data processing device based on a large language model is provided, the device comprising:
[0013] The acquisition module is used to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers.
[0014] The prompting module is used to extract prompt templates, generate pseudo-question prompt templates, and generate evaluation prompt templates based on the question group to be scored and preset keywords, and to construct corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements respectively;
[0015] The model processing module is used to input the keyword extraction prompts, the pseudo-question generation prompts, and the evaluation generation prompts into the large language model to obtain the corresponding keyword groups, question groups, and global evaluations.
[0016] The determination module is used to determine the text similarity between the respondent's answer and the reference answer;
[0017] The scoring module is used to input the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the answerer's answer.
[0018] Thirdly, a data processing system based on a large language model is provided, the system comprising:
[0019] The data acquisition unit is configured to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers.
[0020] The keyword extraction unit is configured to construct a corresponding keyword extraction prompt statement based on the question group to be scored and a preset keyword extraction prompt template, and input the keyword extraction prompt statement into the large language model to obtain the corresponding keyword group;
[0021] The pseudo-question generation unit is configured to construct a corresponding pseudo-question generation prompt statement based on the question group to be scored and the preset pseudo-question generation prompt template, and input the pseudo-question generation prompt statement into the large language model to obtain the corresponding question group;
[0022] The evaluation generation unit is configured to construct a corresponding evaluation generation prompt statement based on the question group to be rated and a preset evaluation generation prompt template, and input the evaluation generation prompt statement into the large language model to obtain the corresponding global evaluation.
[0023] A similarity determination unit is configured to determine the text similarity between the respondent's answer and the reference answer;
[0024] The comprehensive scoring unit is configured to input the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer.
[0025] Fourthly, a data processing architecture based on a large language model is provided, the architecture comprising:
[0026] A data communication component is configured to acquire a set of questions to be scored, send the set of questions to be scored to a first processing core and a third processing core, and output the score corresponding to the answer of the respondent. The set of questions to be scored includes the question, the answer of the respondent, and the reference answer.
[0027] The first processing core is configured to construct and send corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements to the second processing core based on the question group to be scored and preset keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, and to determine the text similarity between the answerer's answer and the reference answer.
[0028] The second processing core, equipped with a large language model, is configured to extract prompt statements based on the keywords, generate prompt statements based on the pseudo-questions, and generate prompt statements based on the evaluation, thereby generating corresponding keyword groups, question groups, and global evaluations.
[0029] The third processing core, equipped with a comprehensive scoring model, is configured to determine the score corresponding to the answerer's answer based on the keyword group, the question group, the global evaluation, and the text similarity.
[0030] Fifthly, an electronic device is provided, including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect above.
[0031] A sixth aspect provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0032] In a seventh aspect, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the method described in the first aspect above.
[0033] The technical solution of this invention involves obtaining a set of questions to be scored, which includes the questions, the answer from the respondent, and the reference answer. Based on the set of questions to be scored and keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. These constructed statements are then input into a large language model to obtain corresponding keyword groups, question groups, and global evaluations. The text similarity between the respondent's answer and the reference answer is determined. Finally, the keyword groups, question groups, global evaluations, and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer. This technical solution combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy, leveraging both the generalization and generation capabilities of the large language model and the vertical scenario learning capabilities of the comprehensive scoring model. Attached Figure Description
[0034] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:
[0035] Figure 1 is a schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention;
[0036] Figure 2 is a flowchart of the data processing method based on a large language model according to an embodiment of the present invention;
[0037] Figure 3 is a flowchart of the matching method using a comprehensive scoring model according to an embodiment of the present invention;
[0038] Figure 4 is a schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention;
[0039] Figure 5 is a data flow diagram within the data processing system of this embodiment of the invention;
[0040] Figure 6 is a schematic diagram of the data processing architecture based on a large language model according to an embodiment of the present invention;
[0041] Figure 7 is a schematic diagram of a data processing device based on a large language model according to an embodiment of the present invention;
[0042] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0043] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.
[0044] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.
[0045] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".
[0046] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0047] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.
[0048] Figure 1 is a schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention. As shown in Figure 1, the data processing system based on a large language model includes a first user terminal 11, a second user terminal 12, and a server 13.
[0049] Both the first user terminal 11 and the second user terminal 12 are general-purpose terminals that run the examination platform's application or mini-program. These terminals can be smartphones, tablets, in-vehicle computers, smart wearable devices, laptops, or other types of mobile communication terminals. In some possible implementations, the first user terminal 11 and the second user terminal 12 can also be dedicated terminals with corresponding applications embedded in an application-specific integrated circuit (ASIC). The open platform provides third-party developers with an Application Programming Interface (API), enabling them to develop their own mini-programs based on the platform. Specifically, these mini-programs can be programs developed based on applications within the platform and used to perform corresponding operations. The first user terminal 11 is the terminal held by the question setter, and the second user terminal 12 is the terminal held by the question taker.
[0050] Server 13 refers to a general-purpose data processing device that can provide computing or application services to a charging service platform. It can be a single computer, a cluster of multiple computers, or a cloud server that can flexibly adjust computing resources through cloud technology.
[0051] Server 13 interacts with the first user terminal 11 and the second user terminal 12 via the network to obtain the question sets to be scored. Then, the obtained question sets are processed to obtain the corresponding scores.
[0052] Specifically, the first user terminal 11 obtains the questions and reference answers set by the question setter through human-computer interaction, and then sends an examination task release instruction carrying the questions and reference answers to the server 13. After receiving the examination task release instruction, the server 13 releases the examination task to the second user terminal 12. The second user terminal 12 obtains and sends the answer from the respondent to the server 13 through human-computer interaction. After receiving the answer, the server 13 scores it based on the questions and reference answers. The scoring process of the server 13 is as follows: based on the question group to be scored and the keyword extraction prompt template, the pseudo-question generation prompt template, and the evaluation generation prompt template, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. The constructed statements are input into the large language model to obtain the corresponding keyword group, question group, and global evaluation. The text similarity between the respondent's answer and the reference answer is determined. The keyword group, question group, global evaluation, and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer.
[0053] In one possible implementation, the first user terminal 11 and the second user terminal 12 can be the same device, and the question setter and the answerer can log in to the examination platform using different accounts to complete the above-mentioned interaction with the server 13.
[0054] In one possible implementation, the process of scoring the answer based on the question and the reference answer can be completed on the user's end. For example, the examination platform on the user's end can operate offline, in which case a large language model and a comprehensive scoring model deployed on the user's end can be used to implement the scoring process.
[0055] For example, after the first user terminal 11 obtains the question and reference answer set by the question setter, it sends the question and reference answer to the second user terminal 12. Then, after the second user terminal 12 obtains the answer from the respondent, it scores the respondent's answer locally based on the combination of the question and reference answer.
[0056] The method for sending the questions and reference answers to the second user terminal 12 can be either to send them to the server 13 via a network, and then the server 13 sends them to the second user terminal 12. Alternatively, they can be sent directly to the second user terminal 12 via a network or a non-networked method. Non-networked methods can include data transmission via data cable, mobile storage device, Bluetooth, local area network, crossover cable, or third-party transmission software.
[0057] Optionally, the first user terminal 11 and the second user terminal 12 can be the same device, i.e., the user terminal. After the user terminal obtains the question and reference answer set by the question setter, it stores them in the storage area with restricted permissions, then obtains the answer from the respondent, and then completes the scoring locally.
[0058] The system of this embodiment is used to obtain a set of questions to be scored, which includes questions, answerers' answers, and reference answers. Based on the set of questions to be scored and keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. These constructed statements are input into a large language model to obtain corresponding keyword groups, question groups, and global evaluations. The text similarity between the answerer's answer and the reference answer is determined. The keyword groups, question groups, global evaluations, and text similarity are then input into a pre-trained comprehensive scoring model to obtain the score corresponding to the answerer's answer. This embodiment's system combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy, leveraging both the generalization generation capability of the large language model and the vertical scene learning capability of the comprehensive scoring model.
[0059] Figure 2 is a flowchart of the data processing method based on a large language model according to an embodiment of the present invention. As shown in Figure 2, the data processing method based on a large language model includes the following steps:
[0060] Step S201: Obtain the question set to be scored.
[0061] The question set to be scored includes the question, the answerer's answer, and the reference answer.
[0062] In one possible implementation, a complete set of test paper data is obtained, which includes multiple questions and their corresponding answers and reference answers. At this point, it is necessary to first divide the data according to the questions to divide the complete set of test paper data into multiple groups of questions to be scored.
[0063] In one possible implementation, if the question and the respondent's answer are obtained first, but no corresponding reference answer is found, then the reference answer needs to be searched based on the question, or based on the question and the respondent's answer. The reference answer can be retrieved from a database or from the network. The database can be an open database, a dedicated database of the examination platform, or an authorized database of the user; the user is the enterprise, group, or individual using the examination platform. It should be understood that this embodiment does not limit the method of obtaining the reference answer.
[0064] Step S202: Based on the question group to be scored and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, construct the corresponding keyword extraction prompt statement, pseudo-question generation prompt statement and evaluation generation prompt statement respectively.
[0065] Step S203: Input the keyword extraction prompt, the pseudo-question generation prompt, and the evaluation generation prompt into the large language model to obtain the corresponding keyword group, question group, and global evaluation.
[0066] The following explanation will be provided in conjunction with steps S202 and S203.
[0067] The keyword extraction prompts are used to guide the large language model in extracting keywords from the question set to be scored. Keyword extraction requires at least extracting keywords from the respondent's answer. In actual scoring, keyword extraction helps to extract key information from redundant information; the closer the key information is to the reference answer, or the higher its relevance to the question, the higher the corresponding score.
[0068] In one possible implementation, there are multiple keyword extraction suggestion templates, and correspondingly, there are also multiple keyword extraction suggestion statements built based on the keyword extraction templates, each adapted to different keyword extraction needs.
[0069] Optionally, keywords can be directly extracted from the respondent's answer. The keyword extraction prompt template could be "Extract keywords from [respondent's answer to be entered], extraction requirements: xxx". The keyword extraction prompt statement built based on this template is used to prompt the large language model to extract keywords from the respondent's answer. In this case, the keyword group output by the large language model contains multiple keywords corresponding to the respondent's answer.
[0070] Optionally, keywords can be extracted from the respondent's answer based on the question and / or the reference answer. The keyword extraction prompt template could be: "Please extract keywords from [respondent's answer to be entered] based on [question to be entered and / or reference answer]. Extraction requirements: xxx." The keyword extraction prompt statement built upon this template is used to instruct the large language model to extract keywords from the respondent's answer based on the question and / or the reference answer. Using the question and / or reference answer for keyword extraction from the respondent's answer makes the extracted keywords more accurate, effective, and relevant to the professional knowledge of the question set being scored. In this case, the keyword group output by the large language model contains multiple keywords corresponding to the respondent's answer.
[0071] Optionally, in addition to extracting keywords from the respondent's answer, keywords from the question and / or the reference answer can also be extracted. The keyword extraction prompt template could be: "Compare and analyze the [question to be entered and / or the reference answer to be entered] with the [respondent's answer to be entered] to extract keywords from both." The keyword extraction prompt statement built based on this template is used to prompt the large language model to extract keywords from the respondent's answer and the question and / or the reference answer respectively. The keyword groups extracted by this method contain multiple keywords corresponding to the respondent's answer and multiple keywords corresponding to the question, or vice versa. The keyword groups extracted by this method contain more information, and scoring based on these keyword groups will make the scoring more accurate.
[0072] Here's an example of keyword extraction prompts for a scenario where keywords are extracted from the answerer's response based on the question and the reference answer.
[0073] You are an intelligent exam assistant, tasked with helping examiners extract key knowledge points from test-takers' answers. Here are the specific requirements:
[0074] 1. The answer provided may contain redundant information or overly detailed descriptions. Please focus only on the key knowledge points.
[0075] 2. Based on the question, the reference answer, and the scoring criteria, extract the core knowledge points from the answerer's answer.
[0076] 3. Each knowledge point should be kept as concise as possible, preferably within 25 Chinese characters, to avoid being too long.
[0077] 4. The extracted knowledge points should directly reflect the core content of the exam questions, avoiding redundant descriptions.
[0078] 5. Each knowledge point should be standardized as much as possible to ensure it is concise and clear, and to avoid irrelevant or repetitive information as much as possible.
[0079] 6. Extract a maximum of 2-3 knowledge points. If there are fewer, please ignore them.
[0080] Specific operations:
[0081] 1. We provide exam questions and reference answers. Please extract the key knowledge points from the answers provided.
[0082] 2. The scoring points should be listed in concise Chinese and arranged in the order of appearance.
[0083] 3. The output format should follow standard requirements and avoid any irrelevant content.
[0084] 4. If the answer is short enough to be used directly as a knowledge point, then simply output the original answer.
[0085] Output format requirements:
[0086] 1. Separate each knowledge point with a semicolon and arrange them in order.
[0087] The questions are as follows:
[0088] {title}
[0089] The suggested answer is as follows:
[0090] {analysis}
[0091] The answer from the respondent is as follows:
[0092] {answer}
[0093] The pseudo-question generation prompt is used to prompt the large language model to generate pseudo-questions corresponding to the respondent's answer in the question set to be scored. A pseudo-question is a question that is more closely related to the respondent's answer. By comparing the similarity between the pseudo-question and the original question, the accuracy of the respondent's answer can be assessed, and situations where the answer is very similar but completely wrong can be avoided. The higher the similarity between the pseudo-question and the original question, the higher the score for the respondent's answer.
[0094] Similarly, there are multiple pseudo-question generation prompt templates, and correspondingly, there are various pseudo-question generation prompt statements built based on these templates, each adapting to different pseudo-question generation needs. Specifically, pseudo-questions can be generated directly based on the respondent's answer, or they can be generated based on the respondent's answer, referencing the question and / or the reference answer. The process is similar to the keyword extraction described above and will not be elaborated further here. The question group includes the pseudo-questions corresponding to the question and the respondent's answer.
[0095] In one possible implementation, pseudo-questions for the question set to be scored can be generated by leveraging questions and reference answers from other similar question sets. Specifically, similar question sets are searched in a question bank (database or network), and pseudo-questions are constructed by combining similar questions and reference answers from those sets to generate prompts. This allows the large language model to mimic similar question sets and generate pseudo-questions that elicit the answerer's response.
[0096] For example, the pseudo-question generation prompt statement can be:
[0097] You are an intelligent assistant responsible for evaluating the answers given by respondents. Here are your job requirements:
[0098] Based on similar questions and similar reference answers, we analyze pseudo-questions that are closer to the answers given by the respondents.
[0099] Specific operation and output format requirements:
[0100] 1. Determine the question-and-answer logic between similar questions and similar reference answers, and use this as a sample to identify the pseudo-questions corresponding to the answerer's answer.
[0101] 2. Keep questions concise, preferably within 25 Chinese characters, and avoid making them too long.
[0102] Similar questions are as follows:
[0103] {title sample}
[0104] Similar reference answers are as follows:
[0105] {analysis sample}
[0106] The answer from the respondent is as follows:
[0107] {answer}
[0108] The evaluation prompt statement is used to guide the large language model to analyze the respondent's answer from multiple dimensions to generate a corresponding global evaluation. Dimensions may include logical coherence, accuracy, key points, etc. The evaluation prompt statement contains evaluation statements within a predetermined word count threshold.
[0109] Similarly, there are multiple evaluation prompt templates, and correspondingly, various evaluation prompt statements built based on these templates, each adapting to different evaluation generation needs. Specifically, a global evaluation can be generated directly from the respondent's answer, or it can be generated based on the respondent's answer, referencing the question and / or the answer key. The process is similar to the keyword extraction described above and will not be elaborated further. The evaluation prompt statements include evaluation statements within a predetermined character threshold. The predetermined character threshold is set or updated according to actual circumstances; for example, the predetermined character threshold can be 100, 200, or 100 characters.
[0110] For example, the evaluation prompt statement can be:
[0111] You are an intelligent assistant responsible for evaluating the answers given by respondents. Here are your job requirements:
[0112] 1. Based on the question and the reference answer, analyze the overall performance of the answerer's answer and give a clear evaluation (such as "good", "average", "bad").
[0113] 2. In the evaluation, point out the strengths (correct points) and weaknesses (incorrect points) of the answer to help the examiner understand the key issues of the answer.
[0114] Specific operation and output format requirements:
[0115] 1. Provide the questions, scoring criteria, reference answers, and answers from the respondent.
[0116] 2. Extract the correct and incorrect points from the answer based on the scoring criteria, and give an overall evaluation based on these points.
[0117] 3. You can consider whether the answer is reasonable from the perspectives of logic, fluency of language, and rationality of content.
[0118] 4. Output format: A brief description, preferably no more than 200 Chinese characters. No bullet points are needed.
[0119] 5. Output example as follows:
[0120] The overall rating of the answer from the respondent is "Good". Strengths: The answer details the specific violations via SMS, such as misleading red envelope messages and inappropriate use of brand terms, which comply with the violation regulations mentioned in the scoring criteria. Furthermore, the respondent mentioned that penalties would be imposed according to the violation regulations and suggested that the respondent conduct a self-examination, demonstrating the platform's rigor and fairness in handling issues. Weaknesses: The answer did not mention the caution in imposing penalties or the multiple review process, and the explanation for not being able to provide personal evidence was insufficient, which may affect the respondent's understanding of the reasonableness of the penalties.
[0121] The questions are as follows:
[0122] {title}
[0123] The scoring criteria are as follows:
[0124] {analysis}
[0125] The answer from the respondent is as follows:
[0126] {answer}
[0127] Step S204: Determine the text similarity between the respondent's answer and the reference answer.
[0128] In one possible implementation, the textual similarity between the respondent's answer and the reference answer is determined directly.
[0129] In one possible implementation, the respondent's answer and the reference answer can be truncated and modified separately, and then the text similarity between the modified respondent's answer and the reference answer can be determined. Since the respondent's answer and the reference answer can differ significantly in length, and some reference answers and / or respondent answers are complex and redundant, and may contain garbled characters and extra characters, before performing plain text matching, possible errors in symbols and excessively long sentences can be appropriately truncated and modified to preserve the original meaning while minimizing noise affecting the sentence's content.
[0130] Step S205: Input the keyword group, the question group, the global evaluation, and the text similarity into the pre-trained comprehensive scoring model to obtain the corresponding score of the respondent's answer.
[0131] The comprehensive scoring model includes a general coding layer, a transformer encoder, and a multilayer perceptron layer. The comprehensive scoring model is trained based on sample data from the domain to which the question set to be scored belongs.
[0132] In the field of Natural Language Processing (NLP), a general encoding layer refers to a pre-trained language model designed based on the Transformer architecture. These models possess powerful semantic representation capabilities and broad applicability. Examples include BERT, RoBERTa, ALBERT, and ELECTRA. As a general encoding layer, it encodes information obtained from the different processing branches, including keyword groups, question groups, global evaluations, and text similarity, to provide a unified feature representation method for subsequent feature fusion.
[0133] The Transformer Encoder, also known as the Cross-Attention Transformer, is a crucial component of the Transformer architecture. The Transformer Encoder, constructed by stacking multiple layers of self-attention mechanisms and feedforward neural networks, captures long-range dependencies in the input sequence, enhancing the model's semantic understanding capabilities. Cross-Attention, by calculating the attention weights between the decoder's current position and all positions in the encoder, dynamically weights and aggregates the encoder's output, thereby generating an output sequence relevant to the input sequence.
[0134] A multilayer perceptron (MLP) layer transforms input data into a higher or lower dimensional space to better capture nonlinear relationships within the data. In multimodal learning or tasks that fuse multiple features, the MLP layer can integrate information from different sources to generate a more comprehensive and accurate representation.
[0135] In summary, the comprehensive scoring model can fuse features from multiple sources for comprehensive analysis, thereby making the output score more accurate.
[0136] The scale of the comprehensive scoring model is relatively small compared to large language models. Different domains can correspond to different comprehensive scoring models. That is, the comprehensive scoring model is trained using sample data from different domains, so that the trained comprehensive scoring model has a deep understanding of that domain, thereby making the scoring more accurate.
[0137] Figure 3 is a flowchart of the matching method using a comprehensive scoring model according to an embodiment of the present invention. As shown in Figure 3, the matching method using a comprehensive scoring model includes the following steps:
[0138] Step S301: Determine the domain to which the question group to be scored belongs.
[0139] The fields can include education, healthcare, architecture, and services. The level of detail in the field division can be determined based on actual needs. For example, if education requires a higher level of expertise, it can be further subdivided into subjects such as mathematics, Chinese, and English.
[0140] Step S302: Determine the corresponding comprehensive scoring model based on the field.
[0141] Step S303: Input the keyword group, the question group, the global evaluation, and the text similarity into the comprehensive scoring model to obtain the score corresponding to the answerer's answer.
[0142] The method of this invention involves obtaining a set of questions to be scored, which includes the questions, the answer from the respondent, and the reference answer. Based on the set of questions to be scored and keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. These constructed statements are then input into a large language model to obtain the corresponding keyword group, question group, and global evaluation. The text similarity between the respondent's answer and the reference answer is determined. Finally, the keyword group, question group, global evaluation, and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer. This method combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy, leveraging both the generalization generation capability of the large language model and the vertical scene learning capability of the comprehensive scoring model.
[0143] Figure 4 is a schematic diagram of a data processing system based on a large language model according to an embodiment of the present invention. As shown in Figure 4, the data processing system based on a large language model includes:
[0144] The data acquisition unit 401 is configured to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers.
[0145] The keyword extraction unit 402 is configured to construct a corresponding keyword extraction prompt statement based on the question group to be scored and a preset keyword extraction prompt template, and input the keyword extraction prompt statement into the large language model to obtain the corresponding keyword group.
[0146] The pseudo-question generation unit 403 is configured to construct a corresponding pseudo-question generation prompt statement based on the question group to be scored and a preset pseudo-question generation prompt template, and input the pseudo-question generation prompt statement into the large language model to obtain the corresponding question group.
[0147] The evaluation generation unit 404 is configured to construct a corresponding evaluation generation prompt statement based on the question group to be rated and a preset evaluation generation prompt template, and input the evaluation generation prompt statement into the large language model to obtain the corresponding global evaluation.
[0148] The similarity determination unit 405 is configured to determine the text similarity between the respondent's answer and the reference answer.
[0149] The comprehensive scoring unit 406 is configured to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the answerer's answer.
[0150] The specific functions of each unit are as described in the above embodiments and will not be repeated here.
[0151] Figure 5 is a data flow diagram within the data processing system of this embodiment of the invention. The following explanation is based on Figures 4 and 5. The following description uses the example of referencing the question and answer to illustrate how the answer is processed accordingly when constructing the prompt statement.
[0152] The data acquisition unit 401 acquires the question set to be scored, where Q represents the question, A represents the answerer's answer, and R represents the reference answer in Figure 5. Then, the data acquisition unit 401 inputs the acquired question set to be scored into the keyword extraction unit 402, the pseudo-question generation unit 403, the evaluation generation unit 404, and the similarity determination unit 405, respectively.
[0153] Within the keyword extraction unit 402, a keyword extraction prompt statement is constructed based on the question, the answerer's answer, and the reference answer, namely Prompt1(Q, A, R) shown in the figure. Then, it is input into the large language model, and the large language model outputs keyword groups based on Prompt1.
[0154] Within the pseudo-question generation unit 403, a pseudo-question generation prompt statement is constructed based on the question, the answerer's answer, and the reference answer, namely Prompt2(Q, A, R) shown in the figure. Then, it is input into the large language model, and the large language model outputs a question group based on Prompt2.
[0155] Within the evaluation generation unit 404, an evaluation generation prompt statement is constructed based on the question, the answerer's answer, and the reference answer, namely Prompt3(Q, A, R) shown in the figure. Then, it is input into the large language model, and the large language model outputs a global evaluation based on Prompt3.
[0156] Within the similarity determination unit 405, the answer from the respondent and the reference answer are first converted into corresponding vectors, and then the corresponding text similarity is determined based on the vector similarity.
[0157] The output data of the keyword extraction unit 402, pseudo-question generation unit 403, evaluation generation unit 404 and similarity determination unit 405 are input into the comprehensive scoring unit 406. In the comprehensive scoring unit 406, the final score is output based on the sequential processing of the general coding layer, the transformer encoder and the multilayer perceptron layer.
[0158] The data processing system based on a large language model in this invention combines the advantages of large language models and comprehensive scoring models to improve scoring accuracy. It leverages the generalization and generation capabilities of the large language model while fully utilizing the vertical scenario learning capabilities of the comprehensive scoring model.
[0159] Figure 6 is a schematic diagram of the data processing architecture based on a large language model according to an embodiment of the present invention. As shown in Figure 6, the data processing architecture based on a large language model includes:
[0160] The data communication component 601 is configured to acquire a set of questions to be scored, send the set of questions to be scored to a first processing core and a third processing core, and output the score corresponding to the answer of the respondent. The set of questions to be scored includes the question, the respondent's answer and the reference answer.
[0161] The first processing core 602 is configured to construct and send corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements to the second processing core based on the question group to be scored and preset keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, and to determine the text similarity between the answer of the respondent and the reference answer.
[0162] The second processing core 603, equipped with a large language model, is configured to extract prompt statements based on the keywords, generate prompt statements based on the pseudo-questions, and generate prompt statements based on the evaluation, thereby generating corresponding keyword groups, question groups, and global evaluations.
[0163] The third processing core 604, equipped with a comprehensive scoring model, is configured to determine the score corresponding to the answerer's answer based on the keyword group, the question group, the global evaluation, and the text similarity.
[0164] The data processing architecture of this invention combines the advantages of large language models and comprehensive scoring models to improve scoring accuracy. It leverages the generalization and generation capabilities of large language models while fully utilizing the vertical scenario learning capabilities of comprehensive scoring models.
[0165] Figure 7 is a schematic diagram of a data processing device based on a large language model according to an embodiment of the present invention. As shown in Figure 7, the data processing device based on a large language model includes:
[0166] The acquisition module 701 is used to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers.
[0167] The prompting module 702 is used to construct corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements based on the question group to be scored and the preset keyword extraction prompt template, pseudo-question generation prompt template, and evaluation generation prompt template.
[0168] The model processing module 703 is used to input the keyword extraction prompt statement, the pseudo-question generation prompt statement, and the evaluation generation prompt statement into the large language model to obtain the corresponding keyword group, question group, and global evaluation.
[0169] The determination module 704 is used to determine the text similarity between the answerer's answer and the reference answer.
[0170] The scoring module 705 is used to input the keyword group, the question group, the global evaluation and the text similarity into a pre-trained comprehensive scoring model to obtain the score corresponding to the answerer's answer.
[0171] The specific functions of each module are as described in the above embodiments and will not be repeated here.
[0172] The apparatus of this invention is used to acquire a set of questions to be scored, which includes questions, answerers' answers, and reference answers. Based on the set of questions to be scored and keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. These constructed statements are input into a large language model to obtain corresponding keyword groups, question groups, and global evaluations. The text similarity between the answerer's answer and the reference answer is determined. The keyword groups, question groups, global evaluations, and text similarity are then input into a pre-trained comprehensive scoring model to obtain the score corresponding to the answerer's answer. This embodiment combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy, leveraging both the generalization generation capability of the large language model and the vertical scene learning capability of the comprehensive scoring model.
[0173] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 80 includes a server, a terminal, etc. As shown in Figure 8, the electronic device 80 includes at least one processor 801; a memory 802 communicatively connected to at least one processor 801; and a communication component 803 communicatively connected to a scanning device. The communication component 803 receives and transmits data under the control of the processor 801. The memory 802 stores instructions executable by at least one processor 801, which are executed by at least one processor 801 to implement the above-described data processing method based on a large language model.
[0174] Specifically, the electronic device includes one or more processors 801 and a memory 802, with one processor 801 as an example in Figure 8. The processor 801 and memory 802 can be connected via a bus or other means; Figure 8 shows a bus connection as an example. The memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in the memory 802, thereby implementing the aforementioned data processing method based on a large language model.
[0175] Memory 802 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 802 may optionally include memory remotely located relative to processor 801, and these remote memories can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0176] One or more modules are stored in memory 802, and when executed by one or more processors 801, they perform the data processing method based on the large language model in any of the above method embodiments.
[0177] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.
[0178] The technical solution of this invention involves obtaining a set of questions to be scored, which includes the questions, the answer from the respondent, and the reference answer. Based on the set of questions to be scored and keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements are constructed respectively. These constructed statements are then input into a large language model to obtain corresponding keyword groups, question groups, and global evaluations. The text similarity between the respondent's answer and the reference answer is determined. Finally, the keyword groups, question groups, global evaluations, and text similarity are input into a pre-trained comprehensive scoring model to obtain the score corresponding to the respondent's answer. This technical solution combines the advantages of a large language model and a comprehensive scoring model to improve scoring accuracy, leveraging both the generalization and generation capabilities of the large language model and the vertical scenario learning capabilities of the comprehensive scoring model.
[0179] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program for a computer to execute some or all of the above-described method embodiments.
[0180] Another embodiment of the present invention relates to a computer program product, including a computer program / instructions that, when executed by a processor, implement some or all of the method embodiments described above.
[0181] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0182] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A data processing method based on a large language model, characterized in that, The method includes: Obtain the question set to be scored, which includes the question, the answerer's answer, and the reference answer; Based on the question set to be scored and the preset keyword extraction prompt template, pseudo-question generation prompt template and evaluation generation prompt template, respectively construct the corresponding keyword extraction prompt statement, pseudo-question generation prompt statement and evaluation generation prompt statement; The keyword extraction prompts, the pseudo-question generation prompts, and the evaluation generation prompts are respectively input into the large language model to obtain the corresponding keyword groups, question groups, and global evaluations; Determine the text similarity between the respondent's answer and the reference answer; The keyword group, the question group, the global evaluation, and the text similarity are input into a pre-trained comprehensive scoring model to obtain the corresponding score for the respondent's answer.
2. The method according to claim 1, characterized in that, The keyword extraction prompt statement is used to prompt the large language model to extract keywords from the answerer's answer based on the question and / or the reference answer; The pseudo-question generation prompt statement is used to prompt the large language model to generate a pseudo-question corresponding to the answerer's answer based on the question, the answerer's answer, and the reference answer; The evaluation prompt statement is used to prompt the large language model to analyze the answer from multiple dimensions to generate a corresponding global evaluation; The keyword group contains multiple keywords corresponding to the answerer's answer; The question group includes pseudo-questions corresponding to the question and the answerer's answer; The evaluation prompt statement includes evaluation statements within a predetermined word count threshold.
3. The method according to claim 1, characterized in that, The keyword extraction prompt statement is used to prompt the large language model to extract keywords from the question, the answerer's answer, and the reference answer, respectively. The keyword group includes multiple keywords corresponding to the answerer's answer and multiple keywords corresponding to the question, or the keyword group includes multiple keywords corresponding to the answerer's answer, multiple keywords corresponding to the question, and multiple keywords corresponding to the reference answer.
4. The method according to claim 1, characterized in that, Determining the text similarity between the respondent's answer and the reference answer includes: The answers provided by the respondents and the reference answers were truncated and rewritten respectively. Determine the text similarity between the rewritten answer and the reference answer.
5. The method according to claim 1, characterized in that, The step of inputting the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score for the respondent's answer includes: Determine the domain to which the question set to be scored belongs; Determine the corresponding comprehensive scoring model based on the stated field; The keyword group, the question group, the global evaluation, and the text similarity are input into the comprehensive scoring model to obtain the corresponding score for the respondent's answer.
6. The method according to claim 1 or 5, characterized in that, The comprehensive scoring model comprises a general coding layer, a transformer encoder, and a multilayer perceptron layer. The comprehensive scoring model is trained based on sample data from the domain to which the question set to be scored belongs.
7. A data processing device based on a large language model, characterized in that, The device includes: The acquisition module is used to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers. The prompting module is used to extract prompt templates, generate pseudo-question prompt templates, and generate evaluation prompt templates based on the question group to be scored and preset keywords, and to construct corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements respectively; The model processing module is used to input the keyword extraction prompts, the pseudo-question generation prompts, and the evaluation generation prompts into the large language model to obtain the corresponding keyword groups, question groups, and global evaluations. The determination module is used to determine the text similarity between the respondent's answer and the reference answer; The scoring module is used to input the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score of the respondent's answer.
8. A data processing system based on a large language model, characterized in that, The system includes: The data acquisition unit is configured to acquire a set of questions to be scored, which includes questions, answers from respondents, and reference answers. The keyword extraction unit is configured to construct a corresponding keyword extraction prompt statement based on the question group to be scored and a preset keyword extraction prompt template, and input the keyword extraction prompt statement into the large language model to obtain the corresponding keyword group; The pseudo-question generation unit is configured to construct a corresponding pseudo-question generation prompt statement based on the question group to be scored and the preset pseudo-question generation prompt template, and input the pseudo-question generation prompt statement into the large language model to obtain the corresponding question group; The evaluation generation unit is configured to construct a corresponding evaluation generation prompt statement based on the question group to be rated and a preset evaluation generation prompt template, and input the evaluation generation prompt statement into the large language model to obtain the corresponding global evaluation. A similarity determination unit is configured to determine the text similarity between the respondent's answer and the reference answer; The comprehensive scoring unit is configured to input the keyword group, the question group, the global evaluation, and the text similarity into a pre-trained comprehensive scoring model to obtain the corresponding score of the respondent's answer.
9. A data processing architecture based on a large language model, characterized in that, The architecture includes: A data communication component is configured to acquire a set of questions to be scored, send the set of questions to be scored to a first processing core and a third processing core, and output the corresponding score of the answerer's answer. The set of questions to be scored includes the question, the answerer's answer, and the reference answer. The first processing core is configured to construct and send corresponding keyword extraction prompt statements, pseudo-question generation prompt statements, and evaluation generation prompt statements to the second processing core based on the question group to be scored and preset keyword extraction prompt templates, pseudo-question generation prompt templates, and evaluation generation prompt templates, and to determine the text similarity between the answerer's answer and the reference answer. The second processing core, equipped with a large language model, is configured to extract prompt statements based on the keywords, generate prompt statements based on the pseudo-questions, and generate prompt statements based on the evaluation, thereby generating corresponding keyword groups, question groups, and global evaluations. The third processing core, equipped with a comprehensive scoring model, is configured to determine the corresponding score of the respondent's answer based on the keyword group, the question group, the global evaluation, and the text similarity.
10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-6.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.
12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-6.
