Model training method, data processing method, electronic device, and storage medium

By constructing a multi-preference alignment model for training targets with inverted constraint preferences, the problems of high model training cost and low preference alignment accuracy in scenarios with concurrent multiple preferences and coexistence of multiple constraints are solved, achieving the effect of efficiently generating high-quality and diversified responses.

WO2026237502A1PCT designated stage Publication Date: 2026-11-19ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/082933
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-12
Filing Date
2026-03-11
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing technologies suffer from high training costs and low preference alignment accuracy in complex scenarios with multiple concurrent preferences and multiple constraints, making it difficult to generate high-quality and diverse responses.

Method used

By acquiring multi-turn dialogue data, we construct inverted constraint preference pairs, train a target multi-preference alignment model, and generate a model capable of performing demand analysis in multi-preference scenarios.

Benefits of technology

It significantly reduces model training costs, improves preference alignment accuracy, generates high-quality and diverse responses, and enhances the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026082933_19112026_PF_FP_ABST
    Figure CN2026082933_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to the fields of large model technology and data processing technology, and relates to a model training method, a data processing method, an electronic device, and a storage medium. The method comprises: acquiring multi-turn dialogue data, wherein the multi-turn dialogue data comprises a plurality of responses, which are generated by means of multi-turn dialogue under multi-preference scenarios; utilizing the multi-turn dialogue data to construct reversed-constraint preference pairs; and on the basis of the reversed-constraint preference pairs, training an initial multi-preference alignment model to generate a target multi-preference alignment model, which is configured to perform requirement analysis on multi-preference interaction requirements to obtain a target response. The present disclosure solves the technical problems in the related art of high model training cost and low preference alignment accuracy when faced with complex scenarios involving concurrent multiple preferences and coexisting multiple constraints.
Need to check novelty before this filing date? Find Prior Art

Description

Model training methods, data processing methods, electronic devices and storage media Technical Field

[0001] This disclosure relates to the fields of large model technology and data processing technology, and more specifically, to a model training method, a data processing method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale models are increasingly widely used in fields such as intelligent customer service and role-playing. The core challenge lies in how to simultaneously align multiple user preferences and generate high-quality, diverse responses. To enable large-scale models to respond accurately and meet the diverse preferences and constraints involved in user needs, related technologies generally adopt methods based on single-preference or simple preference alignment. However, this approach lacks an effective mechanism to eliminate noise between samples and relies excessively on hard-to-obtain perfect positive examples. Consequently, it is difficult to efficiently handle complex scenarios with concurrent multiple preferences and coexisting constraints, which not only limits the performance of large-scale models in multi-preference alignment but also increases the model training cost.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This disclosure provides a model training method, a data processing method, an electronic device, and a storage medium to at least solve the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with multiple concurrent preferences and multiple constraints.

[0005] According to one aspect of the present disclosure, a model training method is provided, comprising: acquiring multi-turn dialogue data, wherein the multi-turn dialogue data includes: multiple responses, the multiple responses being generated through a multi-turn dialogue method in a multi-preference scenario; constructing inverted constraint preference pairs using the multi-turn dialogue data; and training an initial multi-preference alignment model based on the inverted constraint preference pairs to generate a target multi-preference alignment model, wherein the target multi-preference alignment model is used to perform demand analysis on multi-preference interaction needs to obtain a target response.

[0006] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, including: obtaining multi-preference interaction requirements; performing requirement analysis on the multi-preference interaction requirements using a target multi-preference alignment model to obtain a target response; wherein the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0007] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, including: obtaining multi-preference virtual character creation requirements; performing requirement analysis on the multi-preference virtual character creation requirements using a target multi-preference alignment model, and generating target virtual characters; wherein the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0008] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, including: obtaining multi-preference customer service solution requirements; performing requirement analysis on the multi-preference customer service solution requirements using a target multi-preference alignment model to generate a target customer service solution; wherein the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0009] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a multi-preference interaction requirement; and returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target response, wherein the target response is obtained by performing a requirement analysis on the multi-preference interaction requirement using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0010] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: acquiring a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multi-preference interaction requirements; responding to the data processing dialogue request and returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a target response, the target response being obtained by performing requirement analysis on the multi-preference interaction requirements using a target multi-preference alignment model, the target multi-preference alignment model being generated according to any model training method in the embodiments of this disclosure; and displaying the target response in a graphical user interface.

[0011] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: displaying multi-preference interaction requirements on the operation interface in response to an input instruction applied to the operation interface; and displaying a target response on the operation interface in response to a processing instruction applied to the operation interface; wherein the target response is obtained by performing a requirement analysis on the multi-preference interaction requirements using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0012] According to another aspect of the embodiments of this disclosure, a data processing system is also provided, including: a client for sending multi-preference interaction requests; a server connected to the client for performing requirement analysis on the multi-preference interaction requests using a target multi-preference alignment model to obtain a target response; the client is also used to output the target response; wherein the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0013] According to another aspect of the present disclosure, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.

[0014] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of the present disclosure during runtime.

[0015] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.

[0016] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0017] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0018] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.

[0019] In this embodiment, multi-turn dialogue data generated in multi-turn dialogue scenarios is acquired. Then, inverted constraint preference pairs are constructed using this data. Finally, based on these inverted constraint preference pairs, an initial multi-preference alignment model is trained to generate a target multi-preference alignment model. This model is then used for demand analysis of multi-preference interaction needs to obtain target responses. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect examples under the new constraint list, thus avoiding the high cost of sampling perfect examples. Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained, which avoids the over-reliance on high-quality perfect positive examples, further accelerates the model training process, and enhances the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexistence of multiple constraints.

[0020] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0022] Figure 1 is a schematic diagram of an application scenario of a model training method according to an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present disclosure;

[0025] Figure 4 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0026] Figure 5 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0027] Figure 6 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0028] Figure 7 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0029] Figure 8 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0030] Figure 9 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0031] Figure 10 is a structural block diagram of a model training device according to an embodiment of the present disclosure;

[0032] Figure 11 is a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 12 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0034] Figure 13 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0035] Figure 14 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0036] Figure 15 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0037] Figure 16 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0038] Figure 17 is a structural block diagram of a computing device according to an embodiment of the present disclosure;

[0039] Figure 18 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0043] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely applied in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, data processing using a target multi-preference alignment model in a multi-preference scenario is used as an example for explanation.

[0044] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0045] Preference Alignment: In the task of generating large models, the model generation results are adjusted to meet the needs of different users or multiple preferences of the same user.

[0046] Constraints: Restrictions or requirements set on the generated content, such as "the number of words must not exceed 200" or "it must contain a certain keyword".

[0047] Reversed Constraint: When a constraint cannot be satisfied, it is reversed into a new constraint that is "opposite to the original constraint". For example, "no less than 200 words" and "less than 200 words" are opposing constraints.

[0048] Domination: If response A is no worse than response B in any of the constrained dimensions (or is better than B in some dimensions and at least equal in the others), then A is said to dominate B. For multi-preference alignment, domination can be used to determine the relative merits of positive and negative examples.

[0049] Perfect Positive: A response that fully satisfies all constraints given a set of constraints.

[0050] Non-dominated: If response A and response B are each better than the other in some dimensions, and an absolute dominance relationship cannot be determined, then they are said to be in a non-dominated relationship.

[0051] Multi-Preference Scenario: User needs include multiple different attributes or constraints, requiring the generated result to satisfy or weigh multiple aspects simultaneously.

[0052] Reversed Constraint Pair (RPO Pair): The available preference pair is obtained by reversing the constraints that are not satisfied in each of two responses that differ in their satisfaction.

[0053] As large-scale models are increasingly applied in various fields such as dialogue, creation, and question answering, the complexity of user commands is also constantly increasing. In many real-world scenarios, users simultaneously impose multiple independent or conflicting constraints. For example, in customer service scenarios, users are required to provide appropriately detailed responses while resolving issues in the shortest possible time; in role-playing scenarios, users need to maintain consistency in their persona while also meeting various specific plot points and text length limitations.

[0054] Multi-preference alignment methods in related technologies typically face the following difficulties: (1) Difficulty in obtaining perfect positive examples: "Perfect positive examples" that fully satisfy all constraints are very scarce, leading to reliance on repeated sampling or combination with other high-cost methods. (2) Noise impact: If the two responses do not form a dominant relationship, i.e., each is dominant on different preferences, then learning preference pairs through simple rating differences can easily introduce noise, causing the large model to mistakenly believe that some constraints do not need to be satisfied. (3) Low multi-preference fusion: Many methods only target a single or a small number of preferences and cannot perform overall optimization on a set of complex preferences.

[0055] Currently, dialogue generation techniques for handling scenarios with multiple concurrent preferences and multiple constraints mainly include three methods, but all have varying degrees of shortcomings, affecting the training efficiency and accuracy of preference alignment. First, while the Direct Preference Optimization (DPO) strategy based on rating differences is simple and intuitive, it struggles to handle the varying strengths and weaknesses of imperfect positive and negative examples in different aspects. This leads to noise introduced by simple rating differences, and it cannot effectively learn the constraint requirement of "positive examples not satisfying while negative examples satisfying". In addition, it suffers from insufficient coverage and low sampling efficiency in multi-preference scenarios.

[0056] Secondly, the Knowledge Transfer Optimization (KTO) method attempts to assist in multi-preference alignment by transferring pre-trained knowledge, but it is helpless in handling the imperfect state of positive examples, and the cost of building the required high-quality knowledge graph or rule system is high, and updating and maintenance are also inconvenient. These limitations weaken its feasibility and effectiveness in practical applications.

[0057] Furthermore, while self-supervised data augmentation methods that incorporate self-evolutionary characteristics aim to strengthen constraint coverage through data rewriting and expansion, the augmented data cannot guarantee that every constraint is perfectly satisfied. In addition, the uncertainty of multi-preference fusion under different expansion strategies can exacerbate learning noise and affect the stability and reliability of model performance.

[0058] In summary, whether it is direct preference alignment, knowledge transfer optimization, or self-supervised data augmentation, the relevant technologies all reveal key problems such as high cost, low efficiency, and preference alignment bias when dealing with complex and ever-changing preferences and constraints, which further affect the model's generalization ability and response quality.

[0059] According to embodiments of this disclosure, a data processing method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0060] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), Internet, or other types of data networks. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0061] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client sends a multi-preference interaction request; after receiving the multi-preference interaction request, the server uses a target multi-preference alignment model to analyze the request and obtain a target response. The target response is then output by the client. During the training of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. Inverted constraint preference pairs are then constructed using this data. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model, which is used to analyze multi-preference interaction requests to obtain a target response. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving the user experience.

[0062] It should be noted that with the rapid development of high-performance computing units, the methods provided in this disclosure can also be applied to integrated model machines in other application scenarios. In one optional embodiment, the integrated model machine has multiple built-in models. Users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the integrated model machine can then directly call the adjusted model to execute the methods provided in this disclosure. In another optional embodiment, the large integrated model machine has a pre-trained model built-in. Therefore, the high-performance computing unit built into the integrated model machine can directly call this model to execute the methods provided in this disclosure.

[0063] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0064] Under the above operating environment, this disclosure provides a model training method as shown in Figure 2. Figure 2 is a flowchart of a model training method according to an embodiment of this disclosure. As shown in Figure 2, the method may include the following steps:

[0065] Step S21: Obtain multi-turn dialogue data, which includes multiple responses generated through multi-turn dialogue in a multi-preference scenario.

[0066] Step S22: Construct inverted constraint preference pairs using multi-turn dialogue data;

[0067] Step S23: Based on the inverted constraint preference pair, train the initial multi-preference alignment model to generate the target multi-preference alignment model, wherein the target multi-preference alignment model is used to perform demand analysis on multi-preference interaction requirements to obtain the target response.

[0068] The aforementioned multi-turn dialogue data represents the dialogue records generated during continuous, multi-turn interactions between the user and the system. This data includes not only the initial user request but also the system's response, user feedback, subsequent supplementary explanations, or new requests. In multi-preference scenarios, each round of dialogue can contain different preferences and constraints, such as tone requirements, information content preferences, and keyword usage restrictions. These preferences and constraints can change or be added to or removed during the dialogue.

[0069] For example, multi-turn dialogue data can be obtained from, but is not limited to, the following scenarios or channels: intelligent customer service systems, social media platforms, user feedback logs, role-playing dialogue systems, educational consulting platforms, and multimodal interaction systems. Specifically, in an enterprise or organization's intelligent customer service system, users may raise a series of questions or requests, including product inquiries, problem-solving, and complaint handling. Multi-turn dialogue records contain user preferences and constraints at different stages, making it a typical scenario for obtaining multi-turn dialogue data. In social media dialogues, multi-turn exchanges between users or between users and the platform often involve various constraints such as emotional inclination, language style, and content preferences. This dialogue data is also rich and diverse and can be used to construct multi-turn dialogue datasets. In various online services, collected user feedback logs, such as user satisfaction feedback on intelligent assistant responses and modification requests, can reveal users' specific requirements under different preferences, providing a basis for constructing multi-turn dialogue data. In virtual role-playing or game dialogue scenarios, user interactions with the system may involve preferences in areas such as role setting, emotional expression, and situational matching. Interaction records during this process can serve as a source for constructing multi-turn dialogue data. In educational consulting platforms, dialogues within educational scenarios, such as online course consultations and customized learning plans, often require meeting preferences regarding content depth, knowledge accuracy, and a gentle tone. The dialogue records generated during these processes are valuable resources for constructing multi-turn dialogue datasets. In interactive systems that incorporate multimodal information such as text, voice, and images, users can simultaneously express multiple preferences, such as speech rate, audio quality, and image clarity. The interaction data generated during these processes can also be used to construct multi-turn dialogue data.

[0070] The user needs in the aforementioned multi-preference scenarios include various attributes or constraints, requiring the generated results to simultaneously satisfy or balance multiple aspects. For example, in an intelligent customer service scenario, users expect responses that are polite, include specific operational instructions, and are of moderate length, encompassing tone preferences, information preferences, and length preferences. In role-playing dialogue generation, responses need to be generated based on the character's personality, background, the context of the dialogue, and possible plot or emotional developments, including character preferences, coherence preferences, and plot preferences.

[0071] In multi-preference scenarios, users often make multiple consecutive requests, each containing multiple constraints. To maintain data diversity, a role-driven self-play model can be adopted. By setting a certain set of background information and constraints on the system side, such as the system's role, language style, and specific keywords that are prohibited or required, the system can generate multiple different personas and needs on the user side based on several personalized descriptions. Finally, the system can interact with the user in a multi-turn dialogue format of up to five rounds, attempting to satisfy the user's multiple preference needs and recording responses. This allows for the rapid construction of rich multi-turn dialogue data.

[0072] After acquiring multi-turn dialogue data, inverted constraint preference pairs are constructed using this data. These pairs are used to improve the model's response generation capability in multi-preference scenarios. In related preference pair learning, perfect positive examples often need to be perfectly satisfied under all constraints, but perfect positive examples are difficult to obtain in complex scenarios and are prone to introducing noise. By inverting unsatisfied constraints into opposing constraints, imperfect responses become so-called "perfect positive examples" under new constraint scenarios, effectively constructing preference pairs that can be used for model training and achieving parallel learning and optimization of multiple preferences.

[0073] For example, suppose there are two responses in an intelligent customer service scenario: Response A: "Dear customer, your order has been processed, please wait patiently," with 30 characters. Response B: "Dear customer, your order will be delivered soon. Please note that our online service hours are from 9:00 AM to 5:00 PM," with 50 characters and containing the keyword "service hours." The original constraints are: Constraint 1: The response should be less than 40 characters; Constraint 2: The response should contain the keyword "service hours." According to the above constraints, Response A satisfies Constraint 1 but not Constraint 2; Response B, on the other hand, satisfies Constraint 2 but not Constraint 1. The two cannot directly form a valid preference pair because Response A and Response B each perform better on different constraints.

[0074] When constructing the inverted constraint preference pair, the constraints are first inverted. Constraint 1 is inverted to: the response should be more than 300 words, and Constraint 2 is inverted to: the response should not contain the keyword "service time". Since response A was originally less than 300 words, it now becomes a perfect positive example under inverted constraint 1. Since response B originally did not contain the keyword "service time", it now becomes a perfect positive example under inverted constraint 2. In the inverted constraint scenario, response A becomes a perfect positive example under the word count requirement, while response B becomes a negative example under the word count requirement. Under the keyword usage requirement, response B is a perfect positive example, and response A is a negative example. Through the above construction process, even though response A and response B each have their shortcomings in the original scenario, under the inverted constraint scenario, they can become positive examples that fully meet the conditions in one aspect. This provides clear preference guidance for the initial multi-preference alignment model, helping it to more effectively learn how to generate high-quality responses that satisfy multiple constraints in multi-preference scenarios.

[0075] After obtaining the inverted constraint preference pairs, the initial multi-preference alignment model is trained based on these pairs to generate a target multi-preference alignment model. This target model is used to perform demand analysis on multi-preference interaction needs to obtain the target response. The aforementioned multi-preference interaction needs refer to multiple different preferences or requirements that participants expect to receive in interactive scenarios (such as dialogue, customer service, role-playing, etc.) that need to be satisfied simultaneously. Multi-preference interaction needs can be, but are not limited to, diverse needs regarding content quality, sentiment, information completeness, response speed, and language style.

[0076] The target response described above is a high-quality response that satisfies all preference requirements as much as possible in a given multi-preference scenario. A target response is not simply an optimal solution for a single preference, but rather seeks a balance among multiple preferences, ultimately presenting a response that achieves a high level of satisfaction across all preferences. It is also the object learned and generated by the target multi-preference alignment model. For example, in the role-playing scenario mentioned above, a response that demonstrates creativity while remaining faithful to the character setting, and also meets the requirements in terms of word count and coherence, is the target response.

[0077] The process of training an initial multi-preference alignment model based on inverted constraint preference pairs to generate a target multi-preference alignment model can improve the model's ability to generate high-quality, multi-constraint-satisfied responses in complex multi-preference scenarios through contrastive learning. The training process can fully utilize the clear preference signals provided by the inverted constraint preference pairs, enabling the initial multi-preference alignment model to learn how to generate responses that satisfy user preferences in multi-constraint scenarios.

[0078] Based on steps S21 to S23 above, multi-turn dialogue data generated in multi-turn dialogue scenarios is acquired. Then, inverted constraint preference pairs are constructed using this data. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model. This model is used for demand analysis of multi-preference interaction needs to obtain target responses. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect examples under the new constraint list, thus avoiding the high cost of sampling perfect examples. Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained, which avoids the over-reliance on high-quality perfect positive examples, further accelerates the model training process, and enhances the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexistence of multiple constraints.

[0079] The model training method in the embodiments of this disclosure will be further described below.

[0080] In an optional embodiment, in step S22, constructing the inverted constraint preference pair using multi-turn dialogue data includes:

[0081] Step S221: Perform constraint annotation on multiple responses to obtain annotation results. The annotation results are used to determine the constraint compliance status of multiple responses. The constraint compliance status is used to indicate whether multiple responses comply with multiple constraints specified in the corresponding round of dialogue.

[0082] Step S222: Construct inverted constraint preference pairs based on the annotation results.

[0083] Specifically, during constraint annotation, natural language processing techniques, such as text parsing, keyword extraction, sentiment analysis, and grammar checking, can be used to automatically analyze the characteristics of each response, thus enabling automated annotation. If certain constraints are difficult to detect automatically, such as subtle differences in sentiment, manual review of responses according to annotation guidelines is required to determine whether they meet the constraints. For each constraint, it is necessary to check or determine whether the response meets the constraint. For example, for a character count constraint, the number of characters in the response is directly counted; for a keyword inclusion constraint, it is checked whether the response contains the specified keyword. During constraint annotation, Boolean values ​​can be used to represent the compliance status of each constraint, forming a constraint compliance vector.

[0084] In multi-turn dialogues, each turn may have different constraints or preferences. When annotating constraints, the constraints for each turn must be determined based on the background information and user instructions. The compliance of responses in each turn is then assessed, and the compliance of each response under each constraint is recorded to obtain the annotation results.

[0085] For example, after constraint annotation is completed, sampling checks can be performed by comparing the annotation results to further ensure the accuracy of the annotation results. If automated annotation is used in the constraint annotation process, the model output results can be verified to ensure that it correctly identifies the constraint compliance status, and annotation errors can be flexibly corrected based on the sampling check results. Constraint annotation provides a clear understanding of the performance of each response under different constraints, providing important information for the subsequent construction of inverted constraint preference pairs.

[0086] Based on the above optional embodiments, by constraining and labeling multiple responses, the labeling results are obtained, and inverted constraint preference pairs are constructed based on the labeling results. This can significantly improve the learning efficiency, constraint satisfaction ability, and quality and diversity of generated responses of the initial multi-preference alignment model in multi-preference scenarios.

[0087] In an optional embodiment, step S222, constructing the inverted constraint preference pair based on the annotation results includes:

[0088] Step S2221: Pair multiple responses to obtain the pairing results;

[0089] Step S2222: Based on the annotation results, perform constraint compliance analysis on the pairing results to obtain the analysis results. The analysis results are used to indicate whether the compliance of the first and second responses in the same constraint dimension is the same.

[0090] Step S2223: Construct inverted constraint preference pairs based on the analysis results.

[0091] Specifically, when matching multiple responses, the multiple responses contained in the multi-round dialogue data are first matched according to a certain logic. Random matching or matching according to a preset strategy can be used, such as matching the two most recent round responses or matching the longest and shortest responses, to obtain matching results for subsequent analysis.

[0092] A detailed comparative analysis of each pair of responses is performed using the annotation results to determine whether the first and second responses in each pair adhere to the same constraints across all dimensions. If the first and second responses in a pair differ in their adherence to one or more constraint dimensions, they can be considered as potential candidates for inverted constraint preference pairs. The analysis results can include information on which constraints each pair of responses exhibits differences, and the nature of these differences; for example, the first response adheres to one constraint while the second does not, or there is a difference in the degree of adherence between the first and second responses.

[0093] For pairings that differ in their compliance with constraints, a constraint reversal approach can be used to construct constraint reversal preference pairs. Specifically, if a response is "not compliant" with a certain constraint, the constraint is reversed to the opposite requirement, and the reversed constraint is applied to the response. The response that was originally not compliant with the constraint can then become a "perfect positive example" in the constraint reversal scenario.

[0094] The purpose of constructing inverted constraint preference pairs is to create a clear learning signal instructing the initial multi-preference alignment model to prioritize generating responses that conform to the constraints. By inverting constraints, even responses that initially appear undesirable or do not fully conform to the constraints can be transformed into positive examples during training, thus providing more comprehensive and in-depth guidance in model training.

[0095] Based on the above optional embodiments, by pairing multiple responses to obtain pairing results, and then performing constraint compliance analysis on the pairing results based on the annotation results to obtain analysis results, and finally constructing inverse constraint preference pairs based on the analysis results, more diverse and accurate learning signals can be introduced during the training process, effectively improving the performance of the initial multi-preference alignment model in the face of multi-preference and multi-constraint scenarios, while reducing training costs and noise interference, and further improving training efficiency.

[0096] In an optional embodiment, step S2223, constructing the inverted constraint preference pair based on the analysis results includes:

[0097] In response to the analysis results showing that the first and second responses have different compliance with the same constraint dimension, a first part of the constraints is determined from the first constraint set corresponding to the first response, and a second part of the constraints is determined from the second constraint set corresponding to the second response. The first part of the constraints consists of the constraints that were not satisfied in the first response, and the second part of the constraints consists of the constraints that were not satisfied in the second response.

[0098] The first set of constraints is obtained by replacing the first set of constraints with their corresponding opposite constraints, and the second set of constraints is obtained by replacing the second set of constraints with their corresponding opposite constraints.

[0099] Based on the first response under the third constraint set and the second response under the fourth constraint set, construct the inverted constraint preference pair.

[0100] Specifically, when the analysis results show that the first response and the second response have different compliance on the same constraint dimension, the constraints that the first response did not satisfy are found from the first constraint set corresponding to the first response, that is, the constraints that the first response violated or did not meet, to obtain the first part of the constraints. Similarly, the constraints that the second response did not satisfy are found from the second constraint set corresponding to the second response to obtain the second part of the constraints.

[0101] The first set of constraints that were not satisfied in the first response is reversed into opposing constraints, i.e., constraints opposite to the original constraints, thus generating a new set of third constraints. Under the constraints of the third set of constraints, the first response is considered to satisfy the requirements. Similarly, the second set of constraints that were not satisfied in the second response is reversed into opposing constraints, generating another new set of fourth constraints. Under the constraints of the fourth set of constraints, the second response becomes satisfied.

[0102] Based on the third and fourth constraint sets, the first and second responses become "perfect positive examples" and their corresponding "negative examples" under the inverted constraint scenario, respectively. This constructs a pair of inverted constraint preference pairs, in which one response performs well under specific constraint conditions, while the other is relatively inferior.

[0103] Constructing inverted constraint preference pairs effectively reduces reliance on perfect positive examples, eliminating the need to depend on scarce positive examples that perform perfectly under all constraints. By inverting constraints, responses that are ordinary or not fully satisfied can also be transformed into effective positive examples for training, expanding the resources available for model training. Simultaneously, utilizing inverted constraint preference pairs allows the initial multi-preference alignment model to learn from clear contrastive signals, avoiding the confusion and noise that non-dominated samples might introduce, thus accelerating model convergence and improving training efficiency and final accuracy. Since the initial multi-preference alignment model can learn in a wider range of scenarios and constraint combinations, the generated responses will be more diverse and flexible, better adapting to different user preferences and needs. Constructing inverted constraint preference pairs reduces the need for manual positive example sampling and labeling, helping to lower overall data preparation costs and making the training process more economical and efficient.

[0104] Based on the above optional embodiments, in response to the analysis results showing that the first and second responses have different compliance with the same constraint dimension, a first part of the constraints is determined from the first constraint set corresponding to the first response, and a second part of the constraints is determined from the second constraint set corresponding to the second response. Then, the first part of the constraints is replaced with the opposite constraints corresponding to the first part of the constraints to obtain a third constraint set, and the second part of the constraints is replaced with the opposite constraints corresponding to the second part of the constraints to obtain a fourth constraint set. Finally, based on the first response under the third constraint set and the second response under the fourth constraint set, inverted constraint preference pairs are constructed. This provides more learning opportunities for the initial multi-preference alignment model using inverted constraint preference pairs, enabling it to extract valuable information from responses that originally seemed to not satisfy the constraints, optimize the generation strategy, better adapt to the multi-constraint environment, and further improve the model training efficiency.

[0105] In one alternative embodiment, both the first response under the third constraint set and the second response under the fourth constraint set are perfect positives.

[0106] Specifically, when the analysis results indicate that the first response and the second response do not conform to the same constraint in at least one constraint dimension, the unsatisfied constraints are determined from the first constraint set corresponding to the first response, denoted as the first part of the constraints, and the unsatisfied constraints are determined from the second constraint set corresponding to the second response, denoted as the second part of the constraints. Subsequently, these unsatisfied constraints are reversed, becoming their opposite constraints, forming a third constraint set and a fourth constraint set. Both the first response under the third constraint set and the second response under the fourth constraint set are perfect positive examples.

[0107] Based on the above optional embodiments, by inverting constraints, responses that originally only partially satisfied the constraints can be transformed into perfect positive examples in the new scenario, thereby providing more diverse and comprehensive learning signals for the initial multi-preference alignment model. Since both the first and second responses become perfect positive examples under the inverted constraint set, the complex positive and negative example selection can be avoided during training, simplifying the training process and improving the model's learning quality and speed.

[0108] In one optional embodiment, the first response under the third constraint set has a dominant relationship with responses that do not satisfy the third constraint set, and the second response under the fourth constraint set has a dominant relationship with responses that do not satisfy the fourth constraint set.

[0109] A dominance relationship refers to a situation where, between two responses, the former is considered to dominate the latter if one response is no worse than the latter in all constraints and better than the latter in at least one constraint. The third set of constraints is formed by reversing the constraints that the first response did not satisfy. In the new constraint scenario, the first response is designed to be a perfect positive example that satisfies all constraints. Therefore, the first response has a dominance relationship with any other response that does not satisfy the third set of constraints. Because the first response performs better or at least no worse than the unsatisfied responses in all dimensions under the third set of constraints, it ensures that the first response is the optimal solution in this scenario. The fourth set of constraints is formed by reversing the constraints that the second response did not satisfy. In the new constraint scenario, the second response becomes a perfect positive example, fully complying with all constraints. Therefore, the second response has a dominant position relative to any response that does not satisfy the fourth set of constraints. Since the second response performs better in the current reversed constraint scenario, it can effectively guide the model to learn how to generate responses that better match user preferences.

[0110] Based on the above optional embodiments, since the dominance relationship is clear, the situation of "some positive examples of constraints not being satisfied while negative examples are satisfied" will not occur during the model learning process, thereby effectively eliminating noise and improving the preference alignment accuracy.

[0111] In an optional embodiment, the model training method in this disclosure further includes:

[0112] A difference parameter is used to control the difference between the first response and the second response, wherein the difference parameter is used to adjust the difference between the first response and the second response in terms of constraint compliance.

[0113] The difference parameter g measures the difference in compliance between the first and second responses under their respective third and fourth constraint sets. The difference between the two responses is quantified by calculating the number of inconsistencies in their compliance across multiple constraint dimensions.

[0114] Based on the above optional embodiments, by using a difference parameter to control the difference between the first response and the second response, the introduction of the difference parameter enhances the initial multi-preference alignment model's ability to perceive subtle differences in response quality under multi-preference scenarios, enabling the initial multi-preference alignment model to more accurately match user preferences when generating responses.

[0115] In an optional embodiment, in step S23, training the initial multi-preference alignment model based on the inverted constraint preference pair to generate the target multi-preference alignment model includes:

[0116] Step S231: Under different training strategies, the initial multi-preference alignment model is used to perform preference analysis on the inverted constraint preference pairs to obtain the target loss;

[0117] Step S232: Update the model parameters of the initial multi-preference alignment model based on the target loss to generate the target multi-preference alignment model.

[0118] Specifically, the previously constructed inverted constraint preference pairs are used as input to the initial multi-preference alignment model. Different training strategies, such as contrastive learning and reinforcement learning, are employed, each having a different impact on how the model parameters are updated. The initial multi-preference alignment model analyzes the positive and negative examples in the inverted constraint preference pairs to understand which responses better reflect the user's preferences under various constraints.

[0119] Based on the performance of the initial multi-preference alignment model on inverted constraint preference pairs, a target loss is calculated. The target loss can be, but is not limited to, based on the log-likelihood ratio, contrastive loss, or other metrics that measure the difference between the model-generated responses and user preferences. The calculation of the target loss is central to model training and reflects the effectiveness of the initial multi-preference alignment model in handling inverted constraint preference pairs under the current parameter settings.

[0120] After obtaining the target loss, the model parameters are updated to generate a more efficient and accurate target multi-preference alignment model. The target loss provides quantitative feedback on the model's performance, indicating the accuracy and efficiency of the initial multi-preference alignment model in handling inverted constraint preference pairs. The model parameters are adjusted using gradient descent of the loss function or other algorithms to reduce the target loss, thus improving the initial multi-preference alignment model's performance in multi-preference alignment tasks. After a series of updates and iterations, the model parameters are gradually adjusted to a better state, resulting in the generated target multi-preference alignment model. When handling complex multi-constraint instructions from users, the target multi-preference alignment model will be able to more accurately identify and satisfy user preferences, improving overall satisfaction and efficiency.

[0121] For example, during training, initialization is performed first. This can be based on any pre-trained large model, with model parameters loaded on top of existing instruction-based fine-tuning or supervised fine-tuning (SFT). Further comparative learning is then performed, inputting the constructed positive and negative example preference pairs into the training framework, and updating the model parameters using the target loss. As training iterates, the initial multi-preference alignment model increasingly tends to match high-quality responses that "satisfy all constraints" under the same instruction.

[0122] Based on the above optional embodiments, under different training strategies, the initial multi-preference alignment model is used to perform preference analysis on the inverted constraint preference pairs to obtain the target loss. Then, the model parameters of the initial multi-preference alignment model are updated based on the target loss to generate the target multi-preference alignment model. This enables the initial multi-preference alignment model to learn how to generate high-quality responses under various constraints, effectively reducing noise in the preference alignment process and improving the model's multi-preference fusion capability.

[0123] In an optional embodiment, in step S231, under different training strategies, the initial multi-preference alignment model is used to perform preference analysis on the inverted constraint preference pairs to obtain the target loss, including:

[0124] Step S2311: Under the current model training strategy, the initial multi-preference alignment model is used to predict the preferences of the inverted constraint preference pairs to obtain the first prediction result corresponding to the first response and the second prediction result corresponding to the second response.

[0125] Step S2312: Under the reference model training strategy, the initial multi-preference alignment model is used to predict the preferences of the inverted constraint preference pairs, and the third prediction result corresponding to the first response and the fourth prediction result corresponding to the second response are obtained.

[0126] Step S2313: Based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result, obtain the target loss.

[0127] Specifically, for each response in the inverted constraint preference pair, the initial multi-preference alignment model, based on the parameters under the current training strategy, predicts whether the response conforms to the user's preference under the inverted constraint set, i.e., the degree to which the response follows the constraints, thus obtaining the first prediction result for the first response and the second prediction result for the second response. Subsequently, another reference model training strategy, such as the strategy of a pre-trained model or a known effective preference alignment strategy, is used to make predictions again through the initial multi-preference alignment model, thereby obtaining a benchmark or reference point so that the effectiveness of the model training strategy can be evaluated later.

[0128] Guided by the reference model training strategy, preference prediction for inverted constraint preference pairs is performed, generating two sets of prediction results: a third prediction result corresponding to the first response and a fourth prediction result corresponding to the second response. These third and fourth prediction results will serve as standards for evaluating model performance, comparing the model's performance under the current training strategy to determine its degree of improvement in multi-preference alignment. Furthermore, based on the first, second, third, and fourth prediction results, the target loss is obtained.

[0129] Based on the above optional embodiments, under the current model training strategy, the initial multi-preference alignment model is used to predict preferences for inverted constraint preference pairs, obtaining the first prediction result corresponding to the first response and the second prediction result corresponding to the second response. Then, under the reference model training strategy, the initial multi-preference alignment model is used to predict preferences for inverted constraint preference pairs, obtaining the third prediction result corresponding to the first response and the fourth prediction result corresponding to the second response. Finally, based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result, the target loss is obtained. This allows for a detailed comparison of the effects of different training strategies, ensuring that the initial multi-preference alignment model continues to develop in a more efficient and accurate direction during the iteration process, thereby demonstrating strong adaptability and generative capabilities in the face of complex and diverse user preferences.

[0130] In an optional embodiment, in step S2313, obtaining the target loss based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result includes:

[0131] The log-likelihood ratio of the first prediction result and the third prediction result is calculated to obtain the first calculation result, and the log-likelihood ratio of the second prediction result and the fourth prediction result is calculated to obtain the second calculation result;

[0132] The expected value is calculated using the first hyperparameter, the second hyperparameter, the first calculation result, the second calculation result, and the difference parameter to obtain the target loss.

[0133] Specifically, the target loss L RPO It can be calculated using the following formula:

[0134] Where: σ is the sigmoid function, π θ For the current model training strategy, π ref As a reference model training strategy, β is the first hyperparameter, γ is the second hyperparameter, and g is the first response y. i Second reply y j The parameter representing the degree of difference in constraint compliance. This is the first prediction result. This is the second prediction result. This is the third prediction result. This is the fourth prediction result.

[0135] Based on the above optional embodiments, a first calculation result is obtained by calculating the log-likelihood ratio between the first prediction result and the third prediction result, and a second calculation result is obtained by calculating the log-likelihood ratio between the second prediction result and the fourth prediction result. Then, the expectation is calculated using the first hyperparameter, the second hyperparameter, the first calculation result, the second calculation result, and the difference parameter to obtain the target loss. This allows for a detailed comparison of the effects of different training strategies, ensuring that the initial multi-preference alignment model continues to develop in a more efficient and accurate direction during the iteration process, thereby demonstrating strong adaptability and generative ability in the face of complex and diverse user preferences.

[0136] Figure 3 is a schematic diagram of a model training method according to an embodiment of the present disclosure. As shown in Figure 3, in the training process of the target multi-preference alignment model, data is first constructed and sampled, and then the constraint compliance of the responses is evaluated. If different constraint compliance exists, the constraints are reversed to generate perfect positive examples; otherwise, they are directly used as reference data. Next, reversed constraint preference pairs are constructed, the initial multi-preference alignment model is trained, and finally the target multi-preference alignment model is obtained.

[0137] This disclosure proposes an efficient multi-preference alignment method based on inverted constraints. By "inverting" imperfect responses, a new constraint list is automatically generated, making the originally imperfect responses perfect positive examples under the new constraint list. This avoids the high cost of sampling perfect positive examples and completely eliminates the noise problem caused by non-dominant relationships. The model training method in this disclosure has application value in multiple scenarios, such as customer service and role-playing scenarios, and has a significant service promotion effect. In customer service scenarios, if the same user message contains multiple constraints, such as message length limits, tone requirements, and prohibition of certain words, the target multi-preference alignment model can efficiently align multiple preferences and accurately generate customer service responses. In role-playing scenarios, users often have different requirements for persona, language style, and dialogue length. The target multi-preference alignment model can generate more realistic and role-fitting dialogue content under multiple role and scenario constraints.

[0138] Figure 4 is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in Figure 4, the method may include the following steps:

[0139] Step S41: Obtain multi-preference interaction requirements;

[0140] Step S42: Use the target multi-preference alignment model to perform demand analysis on the multi-preference interaction requirements and obtain the target response; wherein, the target multi-preference alignment model is generated according to the model training method of any one of the embodiments of this disclosure.

[0141] Based on steps S41 to S42 above, by acquiring multi-preference interaction needs, a target multi-preference alignment model is used to analyze these needs and obtain the target response. During the training of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. This multi-turn dialogue data is then used to construct inverted constraint preference pairs. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate the target multi-preference alignment model, which is then used to analyze multi-preference interaction needs and obtain the target response. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving the user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect examples under the new constraint list, thus avoiding the high cost of sampling perfect examples. Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained, which avoids the over-reliance on high-quality perfect positive examples, further accelerates the model training process, and enhances the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexistence of multiple constraints.

[0142] Figure 5 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 5, the method may include the following steps:

[0143] Step S51: Obtain the multi-preference virtual character creation requirements;

[0144] Step S52: The target multi-preference alignment model is used to analyze the requirements for creating multi-preference virtual characters and generate target virtual characters; wherein, the target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0145] Based on steps S51 and S52 above, by acquiring the multi-preference virtual character creation requirements, a target multi-preference alignment model is used to analyze these requirements and generate target virtual characters. During the training of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. This multi-turn dialogue data is then used to construct inverted constraint preference pairs. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model. This model is then used to analyze multi-preference interaction requirements to obtain target responses. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect examples under the new constraint list, thus avoiding the high cost of sampling perfect examples. Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained, which avoids the over-reliance on high-quality perfect positive examples, further accelerates the model training process, and enhances the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexistence of multiple constraints.

[0146] Figure 6 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 6, the method may include the following steps:

[0147] Step S61: Obtain multi-preference customer service solution requirements;

[0148] Step S62: Use the target multi-preference alignment model to perform requirement analysis on the multi-preference customer service solution requirements and generate the target customer service solution; wherein, the target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0149] Based on steps S61 to S62 above, by acquiring multi-preference customer service solution requirements, a target multi-preference alignment model is used to analyze these requirements and generate a target customer service solution. During the training of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. This multi-turn dialogue data is then used to construct inverted constraint preference pairs. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model. This model is then used to analyze multi-preference interaction requirements to obtain target responses. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect examples under the new constraint list, thus avoiding the high cost of sampling perfect examples. Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained, which avoids the over-reliance on high-quality perfect positive examples, further accelerates the model training process, and enhances the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexistence of multiple constraints.

[0150] Figure 7 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 7, the method may include the following steps:

[0151] Step S71: Obtain a data processing request through the first application programming interface, wherein the request data carried in the data processing request includes: multi-preference interaction requirements;

[0152] Step S72: Return a data processing response through the second application programming interface. The response data carried in the data processing response includes: target response, which is obtained by performing demand analysis on multi-preference interaction requirements using a target multi-preference alignment model. The target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0153] Based on steps S71 to S72 above, a data processing request is obtained through a first application programming interface (API), wherein the request data carried in the data processing request includes: multi-preference interaction requirements. Then, a data processing response is returned through a second API, wherein the response data carried in the data processing response includes: a target response. The target response is obtained after analyzing the multi-preference interaction requirements using a target multi-preference alignment model. During the training process of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. Inverted constraint preference pairs are then constructed using this multi-turn dialogue data. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model, which is used to analyze multi-preference interaction requirements to obtain the target response. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving the user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing users' changing needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, imperfect responses become perfect positive examples under the new constraint list, thus avoiding the high cost of sampling perfect positive examples. Based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained, avoiding over-reliance on high-quality perfect positive examples, further accelerating the model training process, and enhancing the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexisting constraints.

[0154] Figure 8 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 8, the method may include the following steps:

[0155] Step S81: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multi-preference interaction requirements;

[0156] Step S82, in response to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: target response, which is obtained by performing demand analysis on multi-preference interaction requirements using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to any model training method in any of the embodiments of this disclosure;

[0157] Step S83: Display the target response within the graphical user interface.

[0158] Based on steps S81 to S83 above, by acquiring the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multi-preference interaction requirements, and then responding to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: target response, which is obtained by performing requirement analysis on the multi-preference interaction requirements using a target multi-preference alignment model, and finally displayed in the graphical user interface. In the training process of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired, and then inverted constraint preference pairs are constructed using the multi-turn dialogue data. Finally, based on the inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model, which is used to perform requirement analysis on multi-preference interaction requirements to obtain target responses. Thus, in complex and ever-changing multi-preference scenarios, a target multi-preference alignment model capable of generating high-quality and diversified responses can be trained at a lower cost and with higher efficiency, significantly improving the user experience. By constructing multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when faced with changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, previously imperfect responses become perfect positive examples under the new constraint list, thus avoiding the high cost of sampling perfect positive examples. Training the initial multi-preference alignment model based on these inverted constraint preference pairs avoids over-reliance on high-quality perfect positive examples, further accelerating the model training process and enhancing its generalization ability in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios involving concurrent multi-preferences and coexisting constraints.

[0159] Figure 9 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 9, the method may include the following steps:

[0160] Step S91: In response to the input command applied to the operation interface, display the multi-preference interaction requirements on the operation interface.

[0161] Step S92: In response to the processing command applied to the operation interface, the target response is displayed on the operation interface; wherein, the target response is obtained by performing a demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0162] Based on steps S91 to S92 above, by responding to input commands applied to the operation interface, multi-preference interaction needs are displayed on the operation interface. Then, by responding to processing commands applied to the operation interface, the target response is displayed. The target response is obtained after analyzing the multi-preference interaction needs using a target multi-preference alignment model. During the training of the target multi-preference alignment model, multi-turn dialogue data generated in multi-preference scenarios is acquired. Inverted constraint preference pairs are then constructed using this multi-turn dialogue data. Finally, based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model, which is used to analyze multi-preference interaction needs to obtain the target response. This allows for the training of a target multi-preference alignment model capable of generating high-quality, diverse responses in complex and varied multi-preference scenarios with lower cost and higher efficiency, significantly improving the user experience. Through the construction of multi-turn dialogue data, the initial multi-preference alignment model can learn more nuanced preference differences and constraint changes, thereby generating more accurate and comprehensive responses when facing changing user needs. By constructing inverted constraint preference pairs using multi-turn dialogue data, imperfect responses become perfect positive examples under the new constraint list, thus avoiding the high cost of sampling perfect positive examples. Based on these inverted constraint preference pairs, the initial multi-preference alignment model is trained, avoiding over-reliance on high-quality perfect positive examples, further accelerating the model training process, and enhancing the generalization ability of the initial multi-preference alignment model in unseen multi-preference scenarios. This solves the technical problems of high model training cost and low preference alignment accuracy in complex scenarios with concurrent multi-preferences and coexisting constraints.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0164] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0166] According to embodiments of this disclosure, a model training apparatus for implementing the above-described model training method is also provided. FIG10 is a structural block diagram of a model training apparatus according to an embodiment of this disclosure. As shown in FIG10, the apparatus includes:

[0167] The acquisition module 1001 is configured to acquire multi-turn dialogue data, which includes multiple responses generated through multi-turn dialogue in a multi-preference scenario.

[0168] Module 1002 is configured to construct inverted constraint preference pairs using multi-turn dialogue data;

[0169] Training module 1003 is configured to train the initial multi-preference alignment model based on inverted constraint preference pairs to generate a target multi-preference alignment model. The target multi-preference alignment model is used to perform demand analysis on multi-preference interaction needs to obtain the target response.

[0170] Optionally, the construction module 1002 is further configured to: perform constraint annotation on multiple responses to obtain annotation results, wherein the annotation results are used to determine the constraint compliance status of multiple responses, and the constraint compliance status is used to indicate whether multiple responses comply with multiple constraints specified in the corresponding round of dialogue; and construct inverted constraint preference pairs based on the annotation results.

[0171] Optionally, the construction module 1002 is further configured to: pair multiple responses to obtain pairing results; perform constraint compliance analysis on the pairing results based on the annotation results to obtain analysis results, wherein the analysis results are used to indicate whether the compliance of the first and second responses that are paired with each other is the same in the same constraint dimension; and construct inverted constraint preference pairs based on the analysis results.

[0172] Optionally, the construction module 1002 is further configured to: in response to the analysis results indicating that the first response and the second response have different compliance on the same constraint dimension, determine a first part of the constraints from the first constraint set corresponding to the first response, and determine a second part of the constraints from the second constraint set corresponding to the second response, wherein the first part of the constraints are constraints not satisfied by the first response, and the second part of the constraints are constraints not satisfied by the second response; replace the first part of the constraints with the opposite constraints corresponding to the first part of the constraints to obtain a third constraint set, and replace the second part of the constraints with the opposite constraints corresponding to the second part of the constraints to obtain a fourth constraint set; construct a reverse constraint preference pair based on the first response under the third constraint set and the second response under the fourth constraint set.

[0173] Optionally, the first response under the third constraint set and the second response under the fourth constraint set are both perfect positives.

[0174] Optionally, the first response under the third constraint set has a dominance relationship with responses that do not satisfy the third constraint set, and the second response under the fourth constraint set has a dominance relationship with responses that do not satisfy the fourth constraint set.

[0175] Optionally, the model training device further includes a control module 1004, configured to control the difference between the first response and the second response using a difference parameter, wherein the difference parameter is used to adjust the difference between the first response and the second response in terms of constraint compliance.

[0176] Optionally, the training module 1003 is further configured to: perform preference analysis on the inverted constraint preference pairs using the initial multi-preference alignment model under different training strategies to obtain the target loss; and update the model parameters of the initial multi-preference alignment model based on the target loss to generate the target multi-preference alignment model.

[0177] Optionally, the training module 1003 is further configured to, under the current model training strategy, use an initial multi-preference alignment model to predict preferences for inverted constraint preference pairs, obtaining a first prediction result corresponding to the first response and a second prediction result corresponding to the second response; under the reference model training strategy, use an initial multi-preference alignment model to predict preferences for inverted constraint preference pairs, obtaining a third prediction result corresponding to the first response and a fourth prediction result corresponding to the second response; and obtain the target loss based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result.

[0178] Optionally, the training module 1003 is further configured to calculate the log-likelihood ratio between the first prediction result and the third prediction result to obtain the first calculation result, and to calculate the log-likelihood ratio between the second prediction result and the fourth prediction result to obtain the second calculation result; and to calculate the expectation using the first hyperparameter, the second hyperparameter, the first calculation result, the second calculation result, and the difference parameter to obtain the target loss.

[0179] It should be noted that the acquisition module 1001, construction module 1002, and training module 1003 correspond to steps S21 to S23 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0180] Figure 11 is a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 11, the apparatus includes:

[0181] Module 1101 is configured to acquire multi-preference interaction requirements.

[0182] The analysis module 1102 is configured to perform demand analysis on multi-preference interaction requirements using a target multi-preference alignment model to obtain target responses; wherein the target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0183] It should be noted that the acquisition module 1101 and analysis module 1102 mentioned above correspond to steps S41 to S42 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0184] Figure 12 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 12, the apparatus includes:

[0185] Module 1201 is configured to retrieve multi-preference virtual character creation requirements.

[0186] The analysis module 1202 is configured to perform requirement analysis on the creation requirements of multi-preference virtual characters using a target multi-preference alignment model, and generate target virtual characters; wherein, the target multi-preference alignment model is generated according to any model training method in any of the embodiments of this disclosure.

[0187] It should be noted that the acquisition module 1201 and analysis module 1202 correspond to steps S51 to S52 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0188] Figure 13 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 13, the apparatus includes:

[0189] Module 1301 is configured to retrieve multi-preference customer service solution requirements.

[0190] Analysis module 1302 is configured to perform requirement analysis on the multi-preference customer service solution requirements using a target multi-preference alignment model to generate a target customer service solution; wherein, the target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0191] It should be noted that the acquisition module 1201 and analysis module 1202 correspond to steps S51 to S52 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0192] Figure 14 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 14, the apparatus includes:

[0193] The acquisition module 1401 is configured to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multi-preference interaction requirements;

[0194] The return module 1402 is configured to return a data processing response via a second application programming interface. The response data carried in the data processing response includes: a target response, which is obtained by performing a demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model. The target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0195] It should be noted that the acquisition module 1201 and analysis module 1202 correspond to steps S51 to S52 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0196] Figure 15 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 15, the apparatus includes:

[0197] The acquisition module 1501 is configured to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multi-preference interaction requirements;

[0198] Return module 1502 is configured to return a data processing dialogue response in response to a data processing dialogue request. The data processing dialogue response carries information including: a target response, which is obtained by performing a demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model. The target multi-preference alignment model is generated according to any of the model training methods in the embodiments of this disclosure.

[0199] Display module 1503 is configured to display the target response within the graphical user interface.

[0200] It should be noted that the acquisition module 1501, return module 1502, and display module 1503 correspond to steps S81 to S83 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also run as part of the device in the server provided in the above embodiments.

[0201] Figure 16 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 16, the apparatus includes:

[0202] The first display module 1601 is configured to respond to input commands applied to the operation interface and display multi-preference interaction requirements on the operation interface.

[0203] The second display module 1602 is configured to respond to processing instructions applied to the operation interface and display the target response on the operation interface; wherein the target response is obtained by performing demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to any one of the model training methods in the embodiments of this disclosure.

[0204] It should be noted that the first display module 1601 and the second display module 1602 mentioned above correspond to steps S91 to S92 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0205] It should be noted that the preferred embodiments involved in the above embodiments of this disclosure are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0206] Embodiments of this disclosure may provide a data processing system, including: a client for sending multi-preference interaction requests; a server connected to the client for performing requirement analysis on the multi-preference interaction requests using a target multi-preference alignment model to obtain a target response; the client is also used to output the target response; wherein the target multi-preference alignment model is generated according to any model training method in any of the embodiments of this disclosure.

[0207] Embodiments of this disclosure may provide a computing device. FIG17 is a structural block diagram of a computing device according to an embodiment of the present disclosure. As shown in FIG17, the computing device may include: one or more (only one is shown in the figure) processors 172, memory 174, memory controller, and peripheral interfaces.

[0208] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this disclosure pre-installed.

[0209] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools for application management and monitoring.

[0210] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0211] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0212] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0213] Embodiments of this disclosure can provide an electronic device. FIG18 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG18, the electronic device may include: an input / output device 182; a memory 184; and a processor 186, wherein the processor 186 is connected to the input / output device 182 and the memory 184 via a bus 188.

[0214] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0215] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0216] It will be understood by those skilled in the art that the structure shown in the figure is merely illustrative, and the computing device may also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. This figure does not limit the structure of the aforementioned computing device. For example, the computing device may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the figure, or may have a different configuration than that shown in the figure.

[0217] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0218] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0219] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0220] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0221] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0222] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0223] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0224] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0225] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0226] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0227] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0228] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0229] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Acquire multi-turn dialogue data, wherein the multi-turn dialogue data includes: multiple responses, which are generated through multi-turn dialogue in a multi-preference scenario; Construct inverted constraint preference pairs using the multi-turn dialogue data; Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained to generate a target multi-preference alignment model, wherein the target multi-preference alignment model is used to perform demand analysis on multi-preference interaction needs to obtain the target response. 2.The model training method of claim 1, wherein, Constructing the inverted constraint preference pair using the multi-turn dialogue data includes: Constraint annotations are performed on the multiple responses to obtain annotation results, wherein the annotation results are used to determine the constraint compliance status of the multiple responses, and the constraint compliance status is used to indicate whether the multiple responses comply with multiple constraints specified in the corresponding round of dialogue; The inverted constraint preference pair is constructed based on the annotation results. 3.The model training method of claim 2, wherein, Constructing the inverted constraint preference pair based on the annotation results includes: The multiple responses are paired to obtain the pairing results; Based on the annotation results, the pairing results are subjected to constraint compliance analysis to obtain analysis results, wherein the analysis results are used to indicate whether the first and second responses that are paired with each other have the same compliance on the same constraint dimension; Based on the analysis results, construct the inverted constraint preference pair. 4.The model training method of claim 3, wherein, Constructing the inverted constraint preference pair based on the analysis results includes: In response to the analysis results indicating that the first response and the second response have different compliance with the same constraint dimension, a first set of constraints is determined from the first set of constraints corresponding to the first response, and a second set of constraints is determined from the second set of constraints corresponding to the second response, wherein the first set of constraints are constraints not satisfied by the first response, and the second set of constraints are constraints not satisfied by the second response; The first set of constraints is replaced with the opposite constraints corresponding to the first set of constraints to obtain the third set of constraints, and the second set of constraints is replaced with the opposite constraints corresponding to the second set of constraints to obtain the fourth set of constraints. Based on the first response under the third constraint set and the second response under the fourth constraint set, the inverted constraint preference pair is constructed. 5.The model training method of claim 4, wherein, Both the first response under the third constraint set and the second response under the fourth constraint set are perfect positive examples. 6.The model training method of claim 4, wherein, The first response under the third constraint set has a dominance relationship with responses that do not satisfy the third constraint set, and the second response under the fourth constraint set has a dominance relationship with responses that do not satisfy the fourth constraint set. 7.The method of Claim 3, wherein The model training method also includes: A difference parameter is used to control the difference between the first response and the second response, wherein the difference parameter is used to adjust the difference between the first response and the second response in terms of constraint compliance. 8.The model training method of any one of claims 3-7, wherein, Based on the inverted constraint preference pair, the initial multi-preference alignment model is trained to generate the target multi-preference alignment model, including: Under different training strategies, the initial multi-preference alignment model is used to perform preference analysis on the inverted constraint preference pair to obtain the target loss; The model parameters of the initial multi-preference alignment model are updated based on the target loss to generate the target multi-preference alignment model. 9.The model training method of claim 8, wherein, Under different training strategies, the initial multi-preference alignment model is used to perform preference analysis on the inverted constraint preference pair to obtain the target loss, including: Under the current model training strategy, the initial multi-preference alignment model is used to predict the preferences of the inverted constraint preference pair to obtain the first prediction result corresponding to the first response and the second prediction result corresponding to the second response. Under the reference model training strategy, the initial multi-preference alignment model is used to predict the preferences of the inverted constraint preference pair to obtain the third prediction result corresponding to the first response and the fourth prediction result corresponding to the second response. The target loss is obtained based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result. 10.The model training method of claim 9, wherein, Based on the first prediction result, the second prediction result, the third prediction result, and the fourth prediction result, the target loss is obtained, including: The log-likelihood ratio of the first prediction result and the third prediction result is calculated to obtain a first calculation result, and the log-likelihood ratio of the second prediction result and the fourth prediction result is calculated to obtain a second calculation result; The target loss is obtained by calculating the expectation using the first hyperparameter, the second hyperparameter, the first calculation result, the second calculation result, and the difference parameter. 11.The method of Claim 3, wherein The multiple responses are paired to obtain the pairing results, including: The multiple responses are randomly paired, or the multiple responses are paired according to a preset strategy to obtain the pairing result.

12. A data processing method, comprising: Acquire multi-preference interaction needs; A target multi-preference alignment model is used to analyze the multi-preference interaction requirements and obtain the target response; The target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

13. A data processing method, comprising: Acquire multi-preference virtual character creation requirements; A target virtual character creation requirement analysis was performed using a target multi-preference alignment model to generate target virtual characters. The target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

14. A data processing method, comprising: Obtain multi-preference customer service solution requirements; A target multi-preference alignment model is used to analyze the requirements of the multi-preference customer service solution and generate a target customer service solution. The target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

15. A data processing method, comprising: A data processing request is obtained through a first application programming interface, wherein the request data carried in the data processing request includes: multi-preference interaction requirements; The data processing response is returned through the second application programming interface, wherein the response data carried in the data processing response includes: the target response, which is obtained by performing a demand analysis on the multi-preference interaction demand using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

16. A data processing method, comprising: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multi-preference interaction requirements; In response to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: a target response, which is obtained by performing a demand analysis on the multi-preference interaction demand using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11; The target response is displayed within the graphical user interface.

17. A data processing method, comprising: In response to input commands applied to the user interface, display multi-preference interaction requirements on the user interface; In response to the processing instructions applied to the operation interface, the target response is displayed on the operation interface; The target response is obtained by performing a demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model, and the target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

18. A data processing system, comprising: The client is used to send multi-preference interaction requests; The server, connected to the client, is used to perform demand analysis on the multi-preference interaction requirements using a target multi-preference alignment model to obtain the target response; The client is also used to output the target response; The target multi-preference alignment model is generated according to the model training method described in any one of claims 1 to 11.

19. An electronic device comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the model training method according to any one of claims 1 to 11 or the data processing method according to any one of claims 12 to 17.

20. A computer readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the model training method according to any one of claims 1 to 11 or the data processing method according to any one of claims 12 to 17.