204 results about "Reward value" patented technology

The invention provides a multi-modal large model task processing method, device and equipment based on reinforcement learning, and the method comprises the steps: employing a multi-modal large model, generating G groups of responses for multi-modal visual task data, carrying out the scoring of each group of responses, obtaining a reward value, carrying out the standardization of the reward value, obtaining a dominance score, building a strategy updating gradient through the dominance score, and carrying out the calculation of the strategy updating gradient. A final result is obtained after expectation reward maximization, model parameter adjustment and multi-round iteration, and an expectation reward objective function comprises an expectation calculation item and KL divergence constraint, so that a current strategy can be updated by comparing the performance of candidate strategies through group relative strategy optimization, and local optimum can be helped to be jumped out; the strategy updating amplitude is limited by the KL divergence constraint, and the model is prevented from violently changing in the optimization process, so that the training stability is improved; the dynamic strategy iteration allows the model to adjust the balance between exploration and utilization according to the learning progress on the basis of keeping the stability, thereby further ensuring the effectiveness and stability of strategy optimization.

The invention relates to the technical field of robot training. The robot self-adaptive training method based on reinforcement learning comprises the steps that task sub-target information is generated through a high-level strategy network, the task sub-target information is input into a low-level execution network, an action control instruction is generated according to the task sub-target information, interaction feedback information is collected in the execution process, and the action control instruction is sent to a robot through a robot. Calculating a reward value according to the interaction feedback information, carrying out association processing on the reward value and the scene complexity parameter, executing a dynamic reward shaping operation, generating an adjusted reward signal, generating a strategy model optimized by meta-learning based on the adjusted reward signal, loading the strategy model in a simulation environment, and carrying out dynamic reward shaping. A target strategy model optimized through simulation training is generated, the target strategy model is loaded to the robot, and the robot is controlled to execute task operation in the actual interaction scene. The method has the effect of realizing adaptive task learning of the robot in a multi-interaction scene.

The invention provides a multi-round automatic machine learning agent system based on reinforcement learning optimization. Comprising a task analysis module used for generating an initial prompt for an MLE agent to call; the MLE agent module is used for generating an executable code; the code executor is used for generating an execution result; the evaluator is used for outputting a normalized value of each index and a code correctness identifier; the reward construction module is used for generating a reward value; the reinforcement learning optimizer is used for calculating group average return and candidate advantages and updating strategy parameters of the MLE intelligent agent module based on the candidate advantages; and the multi-round interaction control module is used for feeding back the execution result of the previous round and the reward value to the MLE agent module in the multi-round interaction process, and controlling code generation of the next round until a preset termination condition is met. According to the invention, strategy adaptive evolution, reinforcement learning optimization of fine-grained credit distribution and multi-round closed-loop automatic process improvement can be realized.

The invention relates to the technical field of artificial intelligence, and provides a large language model optimization method and device. According to the method, a candidate answer is obtained by inputting a question in a pre-trained large language model, a first reward value, a second reward value and a third reward value of the candidate answer are calculated, the first reward value is used for representing a basic reward of the candidate answer, the second reward value is used for representing the integrating degree of the candidate answer and the question, and the third reward value is used for representing the integrating degree of the candidate answer and the question. The third reward value is used for representing the alignment degree of the candidate answers and the user preferences, then performing multi-head fusion on the first reward value, the second reward value and the third reward value to obtain a target reward value, and finally training a pre-trained large language model based on the target reward value to obtain an optimized large language model. The importance coefficients of different levels of reward mechanisms of the large language model are adaptively learned in a multi-head layering manner, so that the model training efficiency is improved, and the generation quality of the trained model is improved.

The invention discloses a large language model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and reinforcement learning. The method comprises the steps of obtaining an initial prediction answer statement corresponding to a question statement; semantic expansion retrieval is carried out on the initial prediction answer statement, and a retrieval answer statement corresponding to the initial prediction answer statement is determined; scoring the retrieval answer statement, and determining a target reward value of the initial prediction answer statement based on a scoring result of the retrieval answer statement; and if the target reward value is smaller than a preset reward threshold value, adjusting the pre-trained large language model based on the target reward value, and inputting the question statement into the adjusted model again until the finally obtained target reward value is greater than or equal to the preset reward threshold value, thereby completing training of the pre-trained large language model. The optimization training efficiency of the large language model and the accuracy of model output can be improved, and the stability of large language model output is improved.

The invention discloses an automatic leveling method for intelligently controlling an underwater leveling machine. The method comprises the following steps: acquiring real-time attitude data and working surface flatness requirements of a plurality of execution mechanisms of the underwater leveling machine, and constructing state and operation action information characteristics; an execution mechanism is mapped into an agent with a body, information is shared through an underwater acoustic communication network, and each agent selects and executes a leveling action based on self and neighborhood interaction data; after execution, obtaining a reward value, storing the reward value in an experience pool, sampling and updating the state-action function, and performing iterative training to obtain a multi-body agent reinforcement learning optimization model; continuously monitoring the state of a mechanism when the model runs, dynamically updating the features of an affected area when a fault or hydrodynamic abnormality is identified, and recalculating a leveling strategy to obtain a dynamic re-planning scheme; the scheme is combined with model output to generate a real-time leveling instruction sequence, and the real-time leveling instruction sequence is decomposed and then issued to each execution mechanism, so that self-adaptive accurate leveling control of the underwater leveling machine is realized, and the working efficiency and reliability of a complex underwater environment are effectively improved.

Implementations are directed to generating synthetic labeled / preference data by extracting preference pairs from sets of N outputs to a given unlabeled input to a generative model. A plurality of generative outputs are generated by a generative model from a set of input data. A reward model is used to determine a plurality of reward values for the plurality of generative outputs. Based on the reward values, a pair of generative outputs from the plurality of generative outputs is selected for inclusion in a training example. The pair of outputs include a positive training example and a negative training example, where the reward values indicate that the positive training example is preferred over the negative training example. The process can be repeated for a plurality of sets of input data to generate a plurality of training examples for inclusion in a training dataset, which can be used to update reward model(s).

The invention discloses a reinforcement learning training method and system for relieving illusion of a multi-modal large model, and belongs to the field of reinforcement learning training of a multi-modal large language model. Firstly, a planning and visual description generation step is introduced in an early stage to guide a model to perform structured reasoning, then a grouping relative strategy optimization algorithm is used, reward values are calculated for multiple candidate responses generated by the model after cold start, and particularly, a visual perception reward mechanism is set. The reward mechanism evaluates the consistency of the generated text description and the visual information by using an external large language model. Then, based on a vision description attention score advantage distribution method, learning of the model on key vision signals is dynamically enhanced, and the perception ability of the model on the vision signals is improved; and finally, the perception and reasoning performance of the model is further improved by adopting multiple rounds of rejection sampling and supervised fine tuning. The scheme does not depend on a model architecture, the extra overhead is small, the illusion problem caused by early image-text inconsistency is effectively solved, and the accuracy and the reliability are improved.

The invention relates to the technical field of artificial intelligence, and provides a method and device for generating a task execution result, equipment and a medium. According to the method, a reward evaluation agent set is determined based on task description, and a weight is determined for each reward evaluation agent in the set; and obtaining at least one sampling result based on the task description by using a pre-training strategy model, evaluating each sampling result by using each reward evaluation agent, determining a strategy model reward value based on the weight of each reward evaluation agent and the evaluation result, training the strategy model based on the reward value, and obtaining the reward value of the reward evaluation agent. And finally, determining a task execution result by using the trained strategy model, so that a plurality of appropriate reward evaluation agents are automatically and accurately matched for a non-deterministic task, and the plurality of reward evaluation agents are used for jointly training the strategy model to improve the evaluation accuracy of the strategy model. And thus, a task execution result more conforming to the task description is obtained, and the user experience is improved.

In various embodiments, systems and methods for generating interfaces including interface elements representative of next-best actions are disclosed. A request for an interface including a set of features representative of a user associated with the request is received and a user state representation including an implicit user state representation and an explicit user state representation is generated based on the set of features and session data for at least one session associated with the user. An action reward value for each of a plurality of candidate actions is generated based on the user state representation and an interface including at least one interface element representative of a candidate action having a highest action reward value is generated.

The invention provides a reward model training method and device, a strategy model training method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a preference data pair which comprises a preferred response and a non-preferred response generated for the same prompt word, and each of the preferred response and the non-preferred response is composed of a plurality of text unit sequences; inputting each text unit sequence into a to-be-trained reward model to obtain a predicted reward value; and calculating the total training loss according to the predicted reward value, and updating the model parameters of the to-be-trained reward model. According to the method, the response text is subjected to serialized splitting, and the preference data composed of the preferred response and the non-preferred response is introduced for comparative learning, so that the target of model training is no longer to evaluate the absolute quality of a single response, but to identify a key text unit which causes one response to be superior to the other response; the fine-grained evaluation of the response content is realized, and the evaluation accuracy of the reward model and the identification capability of complex user preferences are effectively improved.

The invention discloses an internet integral dynamic incentive mechanism design method based on reinforcement learning. The method comprises the following steps: S1, constructing a user behavior data set; s2, generating a situation label by adopting a natural language processing technology; s3, calculating a reward value by using a context-adaptive reward algorithm; s4, dynamically adjusting the reward strategy through the reward value and the user behavior feedback in combination with the task participation condition of the user and a feedback loop; s5, predicting a future behavior trend of the user by using the gating cycle unit model, and generating a behavior prediction result of the user; s6, dynamically optimizing the reward strategy in combination with a chaos search strategy and a wolf pack optimization algorithm to obtain an optimized reward strategy; and S7, evaluating effectiveness of rewards and user interest changes, obtaining a final global reward strategy based on an A3C algorithm, and realizing real-time dynamic adjustment of an internet integral incentive mechanism. According to the method, a situation self-adaptive reward algorithm, an optimization technology and the like are utilized, and the dynamic adjustment of an internet integral incentive strategy is realized.

The invention provides a large language model training method and device and electronic equipment. The method comprises the steps that control information is set for a large language model through system cue words of the large language model; inputting training data to the large language model, obtaining output data of the large language model, and extracting N digits from the output data; obtaining a first reward value according to the number M of the numbers carrying the preset number label and the number source in the N numbers; in M numbers carrying preset number labels and number sources, standard values corresponding to the numbers are determined according to the number sources, and second reward values are obtained according to comparison results of the standard values and the numbers; and performing reinforcement learning training on the large language model by adopting a near-end strategy optimization mode or a group relative strategy optimization mode, and forming a training reward value in the near-end strategy optimization mode or the group relative strategy optimization mode according to the first reward value and the second reward value. According to the embodiment of the invention, the accuracy of digits generated by the large language model can be improved.

The invention relates to the technical field of artificial intelligence, in particular to a method, a system and equipment for constructing a primary and middle school education tutoring large model and a medium. The method comprises the following steps: firstly, taking Qwen2.5-3B-Instruct as a basic model, and constructing a training data set according to an obtained teaching data set; then, according to the training data set, constructing a reward function to obtain a reward value, and calling a reinforcement learning method to adjust model parameters to obtain a reinforced basic model; and finally, according to the constructed teacher style data set and the set personalized cue words, the strengthened basic model is finely adjusted in a supervision instruction fine adjustment mode, a middle and primary school education tutoring large model is obtained, through two-stage fine adjustment, the reasoning ability of the model is improved, and the calculation efficiency is improved while the low calculation resource requirement is kept. The processing capability of the old people model for complex problems is improved; and a clear teacher style is endowed to the old people model.

A method and apparatus for preference-training a language model are provided. The method according to some embodiments may include obtaining a dataset including a plurality of pieces of response data, wherein each of the plurality of pieces of response data includes multiple responses generated by the language model for a query and user preference information corresponding to each of the multiple responses, filtering out some of the plurality of pieces of response data included in the dataset using reward values for each of the plurality of pieces of response data, output from a proxy model that receives the dataset as input and training the language model using other pieces of response data that have not been filtered out.

The embodiment of the invention provides a reinforcement learning training method and device for a large language model, and the method comprises the steps: inputting a first input text into the large language model, enabling the large language model to process the first input text based on a current strategy, and obtaining G output texts; respectively combining the G output texts with the first input text, and respectively inputting combination results into a trained reward model to obtain G reward values corresponding to the G output texts; the current strategy is updated based on a target function, the target function comprises a global penalty term, the global penalty term represents that a penalty value is applied only when the maximum value in the G reward values does not exceed a preset threshold value, and the penalty value is determined according to respective first output probabilities of the G output texts under the current strategy; therefore, the training of the large language model under the reinforcement learning framework is realized.

In some aspects, the techniques described herein relate to a method including: generating, by a reinforcement learning engine, a plurality of user states within a simulated 3-dimensional environment; generating, by the reinforcement learning engine and for each of the plurality of user states, a plurality of movement actions, wherein each of the plurality of movement actions place a virtual content object in a corresponding position within the simulated 3-dimensional environment; generating, by the reinforcement learning engine and for each corresponding position of the virtual content object, a reward value, wherein the reward value reflects a relative utility of the corresponding position of the virtual content object; and updating, by the reinforcement learning engine, a placement policy based on the reward value for each corresponding position of the virtual content object.

According to the model training method and device, the task execution method and device, the electronic equipment and the storage medium, in the method, the prompt content, the reply content output by the target model for the prompt content and the process data for generating the reply content can be firstly obtained, and then the prompt content, the reply content and the process data are input into the reward system; therefore, the reward value of each reasoning step in the process data is obtained, and finally, the target model is iteratively trained based on the reward value of each reasoning step. The reward system of the method no longer generates the reward value for the token of the sample, but generates the reward value for each reasoning step in the process data, so that the target model can pay attention to the integrity and logicality of the reply content in the training process, and then the performance and stability of the target model in a complex task and the robustness of the model are improved.

A reinforcement learning-based system for adaptively controlling computing performance is provided. The system includes an environment module and an agent module. The environment module is configured to collect environment information from the application environment. Based on the collected environment information, the environment module calculates a reward value using a reward function and then outputs the state data and the reward value to the agent module. The agent module receives the output from the environment module, including the reward value and the state data. Based on the received reward value and state data, the agent module determines a performance adjustment action to take, which is then fed back to the application environment. Upon receiving the performance adjustment action, the application environment executes the performance adjustment operation in response, causing the environment module to collect the updated environment information as a result of the performance adjustment operation.

The invention provides a reinforcement learning training method and system based on a multi-target structured reward function, and the method comprises the steps: an environment simulation layer comprises a physical engine and a blood vessel environment simulator, and is used for simulating the physical movement and interaction of a catheter in a blood vessel, and providing an environment state for a reward calculation layer; the reward calculation layer performs calculation through a plurality of independent calculation modules and finally generates a comprehensive reward value through a composite reward aggregator; the reinforcement learning layer learns and optimizes a navigation strategy according to the comprehensive reward value; and the monitoring evaluation layer monitors each performance index in the training process and adaptively adjusts the weight of each reward component in the reward calculation layer according to the index change. According to the invention, a major breakthrough of reinforcement learning in medical robot training is realized. Compared with a traditional single reward function method, the intelligent agent which is really thought by experts can be trained, and the method not only pursues to quickly reach a target, but also knows how to safely and stably complete operation.

There is described methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The reinforcement learning system comprises an agent configured to perform actions based upon a policy and an intrinsic reward system configured to generate intrinsic reward values for the agent based upon the actions taken by the agent. The method comprises training the reinforcement learning system based upon a plurality of tasks. The training comprises updating the agent's policy based upon the intrinsic reward values generated by the intrinsic reward system and updating the intrinsic reward system based upon an extrinsic reward value obtained based upon the task being performed by the agent. The training further comprises re-initializing the agent's policy when an expiration criterion associated with the agent is met.

The invention discloses an opportunity fair federal learning excitation method and system based on the game theory, and the method comprises the steps: a model owner determines a reward value of a current round, determines the contribution rate of participating in the training of the current round according to each data node of the reward value of the current round, and carries out the local model training corresponding to each data node; the model owner aggregates the local models of the data nodes according to the reputation values of the data nodes to obtain a global model; constructing a first utility function according to the overhead of each data point in the current round, and constructing a second utility function according to the earnings of the model owner; and calculating the utility of a model owner and each data node according to the first utility function and the second utility function, and adjusting the contribution rate of the next round of training and the reward value of the next round according to the utility result. The optimal balance between the power data node retraining contribution rate and the training cost is realized, and the training effect and the duration of federal learning are ensured.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, performing reward training of a diffusion neural network.

Implementations are directed to generating synthetic labeled / preference data by extracting preference pairs from sets of N outputs to a given unlabeled input to a generative model. A plurality of generative outputs are generated by a generative model from a set of input data. A reward model is used to determine a plurality of reward values for the plurality of generative outputs. Based on the reward values, a pair of generative outputs from the plurality of generative outputs is selected for inclusion in a training example. The pair of outputs include a positive training example and a negative training example, where the reward values indicate that the positive training example is preferred over the negative training example. The process can be repeated for a plurality of sets of input data to generate a plurality of training examples for inclusion in a training dataset, which can be used to update reward model(s).

The invention belongs to the technical field of intelligent task-based dialogue, and particularly relates to a task-based dialogue strategy learning method and system based on reinforcement learning, which adopts a soft actor commentator algorithm to be combined with behavior cloning and maximum entropy inverse reinforcement learning to relieve a cold start problem. According to the method, maximum entropy inverse reinforcement learning is utilized, a reward value is calculated, and a user target in a dialogue is accurately deduced according to maximum log likelihood estimation of a human dialogue and a simulated dialogue; a potential reward function is extracted from a successful track through inverse reinforcement learning, manual reward design is replaced, accumulated reward maximization is pursued during strategy optimization, diversity and exploratory performance of the strategy are encouraged, and the dialogue strategy is prevented from being converged to a single mode too early.

The invention belongs to the technical field of network technology and security correlation, and discloses a data enhancement method and system for Android malicious software family classification, and the method comprises the steps: generating an adversarial sample based on a target sample, and carrying out the data enhancement of a target small family; wherein the reinforcement learning strategy network is constructed to generate a selection probability according to an original sample; establishing a reward function, wherein the reward value is higher when the classification result of the disturbed sample is closer to the target small family more easily; training the strategy network according to the reward value and selecting a target sample; and performing family tree analysis on samples in the target small family, and generating an evolution sample according to a family evolution path to perform data enhancement. According to the invention, the discrimination capability of the model near the boundary is improved by using the adversarial sample; the evolution sample improves the distribution density and the coverage range of the small categories in the space, the adversarial sample and the evolution sample are organically combined, and a new technical approach is provided for solving the family classification problem under long-tail distribution.

The invention relates to the technical field of reinforcement learning, in particular to a reinforcement learning training method and system for a small-scale large language model, and the method comprises the steps: obtaining user query and an external tool set, and constructing a tool integration reasoning training task; driving the large language model to generate thinking content and a tool calling instruction, and forming an interaction track in combination with an external tool execution result; respectively calculating a format reward, a tool calling correctness reward, a dynamic length reward and a distillation learning reward according to the interaction track, and fusing each reward item to obtain a total reward value; performing trajectory multiplexing and trajectory data augmentation processing on the interaction trajectory to generate an extended training data set; and based on the extended training data set and the total reward value, performing parameter iterative optimization on the to-be-trained small-scale large-language model by adopting a reinforcement learning strategy. According to the method, the reasoning capability and the structured tool calling capability of a small-scale large-language model in a complex tool integrated reasoning task can be stably improved.

The invention provides a large model fine tuning method and device, electronic equipment and a computer storage medium, and the method comprises the steps: generating a candidate answer sequence according to question information through employing a strategy model after the question information is received; afterwards, generating an evaluation value sequence by using the reference model, and generating a reward value sequence under different reward functions by using a reward model comprising the multi-dimensional reward functions; performing group advantage evaluation according to the reward value sequence under all reward functions to obtain an advantage value sequence; performing difference analysis on the dominant value sequence by using the evaluation value sequence to obtain a difference analysis result; and finally, carrying out fine adjustment on the strategy model based on a difference analysis result to obtain an optimized strategy model. By fusing a multi-dimensional reward mechanism, the accuracy, logicality and robustness of a large model in complex logic and multi-scene tasks are effectively improved.

The invention discloses a malicious software detection model training method and device based on item reward and operation index linkage, and belongs to the technical field of artificial intelligence and malicious software detection. The method comprises the following steps: constructing a reinforcement learning model containing a policy network; the policy network is configured to receive software feature input and output multi-modal actions including classification results, structured interpretation text and detection confidence; based on the training sample, generating a multi-modal action by utilizing a strategy network, and calculating a composite reward value for the generated multi-modal action by utilizing a preset subitem reward; the subitem rewards at least comprise a structure consistency reward, a key element hit reward and a confidence coefficient calibration reward; and calculating advantage estimation based on the composite reward value to construct an objective function of a near-end policy optimization algorithm by using the advantage estimation, and maximizing the objective function by updating parameters of the policy network. The method can improve the explanatory and confidence of model output, and improves the recognition precision.

The invention provides a question and answer model training method and device, electronic equipment and a storage medium. The method comprises the steps that multiple pieces of training data are acquired; in the process of training the question and answer model according to the sample question, performing repeated fragment identification on a reasoning process generated by performing multiple sampling on the sample question to determine a repeated penalty value of a repeated fragment, and performing reward value calculation on an answer generated by performing sampling on the sample question; performing fusion based on the repeated penalty value and the reward value to form a reward signal, performing normalization processing on the reward signal to obtain a dominant value of current sampling, and introducing the dominant value of the current sampling into a grouping strategy optimization algorithm to construct a loss function; and updating model parameters through gradient back propagation to inhibit repeated thinking behaviors of the question and answer model in the reasoning process, and continuing to perform iterative training on the updated question and answer model. More attention is paid to'effective thinking 'in the training process, and the ability of the question and answer model to solve complex problems is improved.