Multi-agent reinforcement learning system, method, and apparatus

Abstract

One embodiment provides a system implemented by a computer, the system comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system to perform operations, wherein the operations include: collecting history data including information on past states and actions of each of a plurality of agents; encoding the history data with temporal information by using a history encoding module; integrating the encoded history data by applying a self-attention mechanism; estimating a joint value function on the basis of the integrated history data; and updating the policy of each of the plurality of agents on the basis of the joint value function.

Description

Multi-agent reinforcement learning system, method, and device

[0001] The present disclosure relates to a system, method, and apparatus for performing multi-agent reinforcement learning.

[0002] In recent years, reinforcement learning has been performed in single-agent environments involving static tasks. However, many real-world problems manifest as multi-agent issues, and in most situations, agents must make independent decisions based on incomplete observations while considering other agents to maximize overall or individual rewards. Multi-agent reinforcement learning serves as a framework to address these problems and can be trained to find effective policies.

[0003] However, these multiple agents perform actions asynchronously with respect to one another, and accordingly, much research is being conducted on considering asynchronous behavior. While such asynchronous multi-agent reinforcement learning has been studied based on deep Q-networks, these methods are unable to support centralized learning and distributed execution due to structural limitations. Therefore, there is a need for research on more efficient asynchronous multi-agent reinforcement learning methods.

[0004] One embodiment of the present disclosure aims to provide a system, method, and apparatus for performing multi-agent reinforcement learning.

[0005] One embodiment of the present disclosure may provide a method and apparatus for deriving an optimized result using reinforcement learning.

[0006] One embodiment of the present disclosure provides a system that is executed by a computer, comprising at least one processor; and at least one memory that stores instructions that cause the system to perform operations when executed by the at least one processor, wherein the operations include: collecting history data including past state and behavior information of each of a plurality of agents; encoding the history data including time point information by a history encoding module; integrating the encoded history data by applying a self-attention mechanism; estimating a joint value function based on the integrated history data; and updating the policy of each of the plurality of agents based on the joint value function.

[0007] In one embodiment, learning can be performed based on the independent history data of each of the plurality of agents without using padding.

[0008] In one embodiment, asynchronous behavior may be reflected by applying an embedding containing time-stamped information to the history data of each of the plurality of agents.

[0009] In one embodiment, the time information may include action duration information for each of the plurality of agents.

[0010] In one embodiment, the history encoding module includes at least one of an RNN (Recurrent Neural Network), an LSTM (Long Short Term Memory), a GRU (Gated Recurrent Unit), and a transformer module, and the self-attention mechanism may include a multi-head attention mechanism.

[0011] In one embodiment, the operation of updating the policy of each of the plurality of agents may include the operation of updating the policy of each of the plurality of agents based on a Proximal Policy Optimization (PPO) algorithm.

[0012] In one embodiment, clipping and entropy normalization can be performed through the PPO algorithm.

[0013] In one embodiment, the operations include an operation to determine a reward based on a Generalized Advantage Estimation (GAE) method that applies a discount rate on a macro action basis, and the macro action can be broken down into a plurality of micro actions and executed.

[0014] In one embodiment, the operations may include an operation to identify whether it is a sparse reward environment; and an operation to apply reward shaping if it is identified as a sparse reward environment.

[0015] In one embodiment, training can be performed using GPU (Graphics Processing Unit) based parallel processing.

[0016] In one embodiment, the operation of estimating the joint value function may include the operation of estimating the joint value function by applying weights that increase the importance of each agent.

[0017] One embodiment of the present disclosure provides a method performed by at least one processor, comprising the steps of: collecting history data including past state and behavior information of each of a plurality of agents; encoding the history data including time point information; integrating the encoded history data by applying a self-attention mechanism; estimating a joint value function based on the integrated history data; and updating the policy of each of the plurality of agents based on the joint value function.

[0018] In one embodiment, learning can be performed without padding by considering the asynchronous nature of the action execution between the plurality of agents.

[0019] In one embodiment, asynchronous behavior may be reflected by applying an embedding containing time-stamped information to the history data of each of the plurality of agents.

[0020] In one embodiment, the step of updating the policy of each of the plurality of agents may include the step of updating the policy of each of the plurality of agents based on a Proximal Policy Optimization (PPO) algorithm.

[0021] In one embodiment, the method may further include a step of determining a reward based on a Generalized Advantage Estimation (GAE) method that applies a discount rate on a macro action basis.

[0022] One embodiment of the present disclosure includes a program stored on a recording medium to execute a method according to one embodiment of the present disclosure on a computer.

[0023] One embodiment of the present disclosure includes a computer-readable recording medium having a program for executing a method according to one embodiment of the present disclosure on a computer.

[0024] One embodiment of the present disclosure includes a computer-readable recording medium that records a database used in one embodiment of the present disclosure.

[0025] According to one embodiment of the present disclosure, more efficient and accurate reinforcement learning training may be possible.

[0026] FIG. 1 is a diagram illustrating training buffers in a MacDec-POMDP (Macro-Action Decentralized Partially Observable Markov Decision Process) according to one embodiment of the present disclosure.

[0027] FIG. 2 is a diagram showing a centralized critique using common history encoders according to one embodiment of the present disclosure.

[0028] FIG. 3 is a diagram illustrating an Agent-Oriented Centralized Critic (AOCC) method according to one embodiment of the present disclosure.

[0029] FIG. 4 is a drawing showing a reinforcement learning system according to one embodiment of the present disclosure.

[0030] FIG. 5 is a diagram showing an agent-centric encoder of an agent-centric centralized critic according to one embodiment of the present disclosure.

[0031] FIG. 6 is a diagram illustrating a method for calculating a value function by integrating agent-centered history according to one embodiment of the present disclosure.

[0032] FIGS. 7a to 7c are drawings illustrating an Agent-Centric Actor-Critic (ACAC) method according to one embodiment of the present disclosure.

[0033] FIG. 8 is a diagram illustrating an Agent-Centric Actor-Critic (ACAC) method according to one embodiment of the present disclosure.

[0034] FIG. 9 is a schematic diagram of a system according to one embodiment of the present disclosure.

[0035] FIG. 10 is a block diagram illustrating the internal structure of an agent-centered history encoding module according to one embodiment of the present disclosure.

[0036] FIG. 11 is a block diagram illustrating the internal structure of a centralized critique module according to one embodiment of the present disclosure.

[0037] FIGS. 12a to 12c are drawings illustrating types of workflows of an agent according to one embodiment of the present disclosure.

[0038] FIG. 13 is a drawing showing a multi-agent discussion type system according to one embodiment of the present disclosure.

[0039] FIG. 14 is a diagram showing a Mixture-of-AI Agents system according to one embodiment of the present disclosure.

[0040] FIG. 15 is a drawing showing a multi-agent workflow system according to one embodiment of the present disclosure.

[0041] FIG. 16 is a drawing showing a Multi-Agent Debate (MAD) system according to one embodiment of the present disclosure.

[0042] FIG. 17 is a diagram showing an A2A (Agent2Agent) protocol system according to one embodiment of the present disclosure.

[0043] To clarify the technical concept of the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, detailed descriptions of related known functions or components will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Components having substantially the same functional configuration among the drawings have been assigned the same reference numerals and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, devices and methods will be described together where necessary. Each operation of the present disclosure does not necessarily have to be performed in the order described and may be performed in parallel, selectively, or individually.

[0044] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account the functions of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0045] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms such as “comprising” or “having” are intended to specify the presence of features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, throughout this disclosure, when a part is described as “comprising” a certain component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0046] Expressions such as "at least one" modify the entire list of components and do not modify the components of the list individually. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0047] Additionally, terms such as “...part,” “...module,” etc., as described in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0048] Throughout the entire disclosure, when a part is described as being “connected” to another part, this includes not only cases where they are “directly connected” but also cases where they are “electrically connected” with other elements interposed between them. Furthermore, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0049] As used throughout this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0050] Throughout the present disclosure, reward shaping refers to a method of inducing learning by providing additional rewards when an agent performs a specific action or reaches a specific state, and may include manual reward shaping, in which a person directly designs a reward function and grants additional rewards to the agent, and automatic reward shaping, in which rewards are automatically formed according to specific rules or algorithms.

[0051] A reinforcement learning system according to one embodiment is a learning method performed by an agent acquiring observations characterizing the current state of an environment and using those observations to select actions to be performed by the agent. In response to the agent performing the selected actions, the environment transitions to a new state and the reinforcement learning system receives a reward. While interacting with the environment, the reinforcement learning system may attempt to maximize the long-term reward received by the agent in response to the actions performed by the agent.

[0052] The MacDec-POMDP (Macro-Action Decentralized Partially Observable Markov Decision Process) can integrate an option framework into the Dec-POMDP by defining a set of macro-actions for each agent. In one embodiment, the MacDec-POMDP is a tuple It can be expressed as. Here, is a set of identifiers for agents, S is a state space, and is a combined primitive behavior space, and is a combined macro behavior space, and is a combined primitive observation space, and is a combined macro-observation space, and is the probability of a state change, and R is a reward shared by all agents, and is a combined observability, may be a combined macro-observability. Throughout this disclosure □ i represents an element or space of agent i, and is for all agents □ i It can mean the combination or multiplication of. Each macro action has a termination condition , initial set , and lower-level policies A tuple consisting of It could be. Here, (or ) is the observation history space of primitive (or macro) behaviors. In one embodiment, the goal of the MacDec-POMDP is a joint higher-level policy that maximizes the expected return from an initial state s0 for given lower-level policies (e.g., macro behaviors). It may be to find. For example, the joint upper-level policy can be determined by the following [Equation 1].

[0053] [Mathematical Formula 1]

[0054]

[0055] FIG. 1 is a diagram illustrating training buffers in MacDec-POMDP according to one embodiment of the present disclosure.

[0056] Referring to FIG. 1, the training buffers of Mac-CERTs (Macro-Action Concurrent Experience Replay Trajectories) (110) and Mac-JERTs (Macro-Action Joint Experience Replay Trajectories) (120) are illustrated. In FIG. 1, z i wa m i can mean the macro observation and macro action of agent i, respectively.

[0057] Referring to the example in FIG. 1, multiple agents may include Agent 1, Agent 2, and Agent 3. Agent 1 may act at time steps 4 and 6, Agent 2 at time steps 1 and 6, and Agent 3 at time step 5, so that a state transition may occur. In one embodiment, Mac-CERTs (110) may create a training buffer by collecting time steps where a state transition occurred for each agent. Accordingly, as in FIG. 1, a buffer containing information on time steps t=0, 4, and 6 may be formed for Agent 1, a buffer containing information on time steps t=0, 1, and 6 may be formed for Agent 2, and a buffer containing information on time steps t=0 and 5 may be formed for Agent 3. Alternatively, Mac-JERTs (120) may create a training buffer by collecting time steps where a state transition occurred for any one of the agents. Accordingly, a buffer containing information for time points t=0, 1, 4, 5, and 6 can be formed. This will be explained in more detail below.

[0058] In Mac-CERTs according to one embodiment, the state transition experience of each agent i is a tuple It can be expressed as. Here, r i is the timestep t i Starting from macro behavior m that persists during a time step i It can mean accumulated rewards for. Accordingly, r i It can be defined as [Equation 2] below.

[0059] [Mathematical Formula 2]

[0060]

[0061] In Mac-JERTs according to one embodiment, the state transition experience is a tuple It can be expressed as. Here, joint macro action With the accumulated rewards, It is possible. Unlike Mac-CERTs, in Mac-JERTs, is the time step when an agent performs the corresponding macro action, and may refer to the end time step when the agent finishes the macro action. In the example of Fig. 1, Agent 2 starts the action at time step 1 and performs a joint macro action This is generated, and the next macro action is initiated by Agent 1 at timestep 4, and the next joint macro action is It could be.

[0062] Multi-Agent Reinforcement Learning is being actively developed and applied in various fields. Using Multi-Agent Reinforcement Learning, distributed agents can effectively collaborate to complete assigned tasks. Currently used Multi-Agent Reinforcement Learning adopts a method where all agents decide on their respective actions simultaneously due to identical durations between actions. However, in real-world scenarios, agents often decide and execute actions at different times, resulting in asynchronous decision-making and execution between agents. An embodiment of the present disclosure aims to provide a framework for modeling multi-agent decision-making where the decision-making and execution of actions between agents occur asynchronously over time using MacDec-POMDP. In one embodiment, in MacDec-POMDP, the start time of each agent's action and the duration of the action may differ.

[0063] FIG. 2 is a diagram showing a centralized critique using joint history encoders according to one embodiment of the present disclosure.

[0064] Referring to FIG. 2, a common history encoder (210) focused on the common history of all agents along the time axis is illustrated. In one embodiment, the training data (220) input to the common history encoder (210) is common macro observation data, which can be generated by collecting observations of time steps where a state transition occurred for any one agent. For example, When represents the k-th macro observation of agent i, as agent 1 undergoes a state transition at time steps t=4 and 6, agent 2 undergoes a state transition at time steps t=2 and 6, and agent 3 undergoes a state transition at time step t=5, training data (220) can be generated to include time steps t=1, 4, 5, and 6 where a state transition occurs in any one of agents 1, 2, and 3. Accordingly, the training data (220) includes the observation of agent 1 at t=0 , observation by Agent 2 , Agent 3's observation It may include, and since Agent 1 has no state transition at t=1, the same observation as at t=0. , observation by Agent 2 , Agent 3 also has no state transition, so the same observation as at t=0. It may include, and at t=4, Agent 1's observation , Agent 2 has no state transition, so the same observation as at t=1 , Agent 3 also has no state transition, so the same observation as at t=0 or 1. It may include, and since Agent 1 has no state transition at t=5, the same observation as at t=4. , Agent 2 has no state transition, so the same observation as at t=1, 4. , Agent 3's observation It may include, and at t=6, the observation at Agent 1 , observation by Agent 2 , Agent 3 has no state transition, so the same observation as at t=5 It may include. This training data (220) can be input into a joint history encoder (210).

[0065] In one embodiment, the joint history encoder (210) may include a first Multi-Layer Perceptron (MLP) (230) and a Gated Recurrent Unit (GRU) (240). The first MLP (230) may include an input layer, one or more hidden layers, and an output layer. The first MLP (230) uses training data (220) The output information is determined by taking the input, and the output information of the first MLP (230) can be input to the GRU (240). The GRU (240) is a model used to process sequential data and can be operated using a reset gate and an update gate. The reset gate determines how much of the previous state should be forgotten, and the update gate determines how much of the previous state information should be retrieved. The output information of the GRU (240) can be input to the second MLP (250).

[0066] A centralized critique focusing on the combination of the histories of all agents along this time axis can use a joint history encoder to abstract the combined macro observations. However, consecutive joint macro observations may contain duplicate observation data at time steps where one agent begins performing a macro action and other agents do not, and consequently, these duplicate observations may risk leading to inaccurate inferences. For example, at t=5 in Fig. 2, duplicate observation data for Agents 1 and 2 is included even though only Agent 3 undergoes a state transition, and such duplicate observation data can lead to inappropriate inference results.

[0067] In one embodiment, in the example of FIG. 2 of MacDec-PDMDP, the combined macro observation history abstracted by the joint history encoder at time step t=6 can be expressed by the following [Equation 3].

[0068] [Mathematical Formula 3]

[0069]

[0070] As shown above, since the shared history encoder uses the same macro observation multiple times (for example, (used three times) can interfere with accurately determining the local history of all agents and reasoning between agents.

[0071] Therefore, because the common centralized critique structure for multi-agent reinforcement learning focuses on histories along a common time axis, redundant macro observations are introduced in the centralized critique for agents capable of non-macro actions at timesteps where one agent begins performing a macro action and other agents are unable to perform actions. However, such redundant macro observations can lead to inaccurate evaluations in the centralized critique. To address the limitations of this approach and enable efficient asynchronous learning, a special architecture is required that can effectively collect individual agent histories and facilitate inference between agents. To this end, one embodiment of the present disclosure aims to provide an agent-oriented centralized critique. This will be described in more detail later with reference to FIG. 3.

[0072] FIG. 3 is a diagram illustrating a centralized critique oriented toward a joint agent according to one embodiment of the present disclosure.

[0073] Referring to FIG. 3, an Agent-Oriented Centralized Critic (AOCC) method for MacDec-POMDP is illustrated. In one embodiment, the agent-oriented reinforcement learning structure may include an agent-oriented history encoder (310, 320, 330) provided for each agent performing positional encoding, and an aggregation module (340) for integrating the histories of all agents. The agent-oriented history encoder (310, 320, 330) provided for each agent may receive the latest local macro observation of the corresponding agent as input and output a local macro observation history. Since this agent-oriented history encoder is not a centralized encoder like FIG. 2, it may not use the same observations redundantly. According to one embodiment, the agent-oriented history encoder may accurately capture local history by focusing only on determining the history of the corresponding agent.

[0074] In one embodiment, a GRU for an agent-oriented history encoder may be used without sharing parameters between agents. That is, a first agent-oriented history encoder (310) corresponding to a first agent may include a first GRU, a second agent-oriented history encoder (320) corresponding to a second agent may include a second GRU, and a third agent-oriented history encoder (330) corresponding to a third agent may include a third GRU.

[0075] In one embodiment, the history of agent i can be expressed as [Equation 4] as follows.

[0076] [Mathematical Formula 4]

[0077]

[0078] Here, Enc iis the agent-oriented history encoder of agent i, and can be the latest local macro observation of agent i.

[0079] In one embodiment, an agent-oriented history encoder may break temporal alignment between agents because it encodes only each agent's local macro-observations rather than joint observations. In one embodiment of the present disclosure, sinusoidal position encoding may be performed to introduce time markers for each agent's macro-observations in order to account for temporal order. The time step of a local macro-observation is a position encoding vector p i It is encoded as, and the encoded result is observed by the local macro z i By connecting with it, the temporal order can also be considered. Accordingly, the history for agent i can be expressed as follows [Equation 5].

[0080] [Mathematical Formula 5]

[0081]

[0082] In one embodiment, location encoding may allow the chronological order and duration of macro actions across all agents to be taken into account.

[0083] In one embodiment, agent-oriented histories derived from an agent-oriented history encoder may be further processed by an aggregation module to approximate a value function. The aggregation module may include attention networks, MLPs followed by connections, summing units, etc. Alternatively, for simplification, only connections with MLP layers may be performed.

[0084] In one embodiment, the input of the aggregation module is a combination of the latest local histories, which can be expressed as [Equation 6] below.

[0085] [Mathematical Formula 6]

[0086]

[0087] The subsequent MLP layers use the shared history as their own values It can be switched to.

[0088] According to one embodiment of the present disclosure, the observation history and time information of each agent can be independently encoded through position encoding and then aggregated to enable efficient central critique learning in an asynchronous setting. That is, the observation history of each agent is encoded with the timestep information at the time when each agent begins to perform a macro action, and the time information is also encoded separately so that it can be explicitly associated for agent-oriented critique learning.

[0089] In addition, according to one embodiment of the present disclosure, reinforcement learning can be performed to derive more accurate results by deleting duplicate macro observations.

[0090] In one embodiment, the system inputs observation data of each of the multiple agents into an encoder module corresponding to each of the multiple agents to generate history information, and can generate output data based on the history information of each of the multiple agents. Here, the encoder module performs position encoding and may be provided for each of the multiple agents.

[0091] FIG. 4 is a drawing showing a reinforcement learning system according to one embodiment of the present disclosure.

[0092] Referring to FIG. 4, a reinforcement learning system performed by three agents, a first agent (410), a second agent (420), and a third agent (430), is illustrated as an example of a multi-agent reinforcement learning system. However, this is merely an example, and unlike FIG. 3, reinforcement learning may be performed by two agents, and it is obvious that reinforcement learning may be performed by more than three agents. For the convenience of explanation, the following will be described using the case of three agents as an example.

[0093] In one embodiment, the environment (405) may be changed by a first action (470) determined by a first agent (410), a second action (480) determined by a second agent (420), and a third action (490) determined by a third agent (430). Accordingly, the first agent (410) may obtain a first observation (440) based on the change in the environment, the second agent (420) may obtain a second observation (450) based on the change in the environment, and the third agent (430) may obtain a third observation (460) based on the change in the environment.

[0094] In one embodiment, a multi-agent reinforcement learning system may include a memory that stores one or more instructions and at least one processor that executes the one or more instructions stored in the memory. By executing one or more instructions, the at least one processor may input observation data corresponding to a first observation (440) of a first agent (410) into a first history encoder for reinforcement learning, thereby generating first history information of the first agent (410). Additionally, the at least one processor may input observation data corresponding to a second observation (450) of a second agent (420) into a second history encoder to generate second history information of the second agent (420). Additionally, the at least one processor may input observation data corresponding to a third observation (460) of a third agent (430) into a third history encoder to generate third history information of the third agent (430). Such history encoders may be provided for each agent of the multi-agents. That is, if there are n agents, n history encoders can be provided.

[0095] In one embodiment, the first history encoder, the second history encoder, and the third history encoder may each perform encoding by including observation data of the corresponding agent and time information related to the observation. If time information is not included, a separate history encoder is provided for each agent, and inaccurate information may be generated because state transition times are not reflected; however, according to one embodiment of the present disclosure, accurate training can be enabled by encoding by matching time information to the observation data of each agent. In addition, the first history encoder, the second history encoder, and the third history encoder may each include a Multi-Layer Perceptron (MLP) and a Gated Recurrent Unit (GRU).

[0096] In one embodiment, a multi-agent reinforcement learning system may include an aggregation module that receives and processes one or more history information corresponding to the output of a history encoder provided for each agent of the multi-agents. That is, a single aggregation module that processes multiple history information may be provided. Additionally, the output value of the aggregation module may be processed by an MLP.

[0097] In one embodiment, the reward (400) may be determined by changing the environment (405) according to the first action (470) determined by the first agent (410), the second action (480) determined by the second agent (420), and the third action (490) determined by the third agent (430). Each agent may cooperate with one another to maximize the value of the reward.

[0098] Multi-agent reinforcement learning may face problems in sparse reward environments where agents struggle to learn effective strategies. Macro-behaviors, which are a series of actions executed by a single decision, facilitate long-term planning, but introduce asynchronous behavior due to varying durations among agents, which can complicate Centralized Training with Decentralized Execution (CTDE).

[0099] In the CTED framework, centralized critiques can generally be trained using the combined histories of all agents. The observation history of the i-th agent at each time step t can be expressed as follows.

[0100] [Mathematical Formula 7]

[0101]

[0102] Here, represents the sequence of all observations by Agent i up to time step t. The critique takes the set of all agents' histories as input and the value at time step t It can predict.

[0103] In an asynchronous setup, all agents observe the macro at every timestep t Since it is not possible to obtain, missing macro observations are the most recent valid macro observations when generating history It can be replaced with.

[0104] This mechanism critiques across time steps even when macro observations are missing It can be ensured that the input to is maintained consistently. By utilizing the latest valid macro observations, centralized critiques can effectively adapt to asynchronous environments while maintaining compatibility with the training process.

[0105] In a decentralized execution, each agent independently receives local macro observations and can select a macro action based on the current local macro observation and historical data. Unlike a critic, since each agent operates independently, there may be no structural difference between synchronous and asynchronous configurations. The macro action of the i-th agent at timestep t. Is It can be sampled as follows.

[0106] Currently studied multi-agent reinforcement learning methods utilize padding to handle asynchronous characteristics, but this padding process can misalign asynchronous experiences, leading to incorrect correlations.

[0107] According to one embodiment of the present disclosure, an Agent-Centric Actor-Critic (ACAC) method that resolves asynchronous behavior without relying on padding may be provided.

[0108] In one embodiment, a Proximal Policy Optimization (PPO) method incorporating a Generalized Advantage Estimate (GAE) method suitable for asynchronous settings may be used to train centralized critics and distributed actors. GAE is a technique that balances the variance of empirical returns and the bias of Temporal Difference (TD) estimation in estimating the advantage function using a hyperparameter λ. When λ=0, reliance is placed on a single-step TD, and when λ=1, the empirical returns over the entire interval excluding the baseline may be used. When Λ is a value between 0 and 1, a compromise balance between the two extremes may be provided. In asynchronous multi-agent reinforcement learning, the interval between consecutive macro observations can vary significantly depending on the agent, situation, etc. If the existing micro-timestep-based λ discount is applied as is, an excessive discount accumulates on the future reward of the decision as the duration of the macro action increases, which can lead to a problem where the relative importance of the macro decision is underestimated. Consequently, policy updates at the time of selecting a long-lasting macro action may be distorted, and learning stability and final performance may degrade.

[0109] According to one embodiment, a modified GAE method may be used to switch the criterion for the λ discount from a micro time step to a macro time step. A micro time step refers to a raw step of the simulator, and a macro time step may refer to an indicator that counts, in chronological order, a set of points in time when any agent acquires a new macro observation. Accordingly, according to one embodiment, the λ discount may be applied to the TD error based on the number of times a macro decision occurred, rather than the elapsed time (number of micro steps).

[0110] In one embodiment, using an actor-critic algorithm, each Distributed actors parameterized by r and centralized critics parameterized by ψ can be trained. The training procedure can be expressed as follows in [Equation 8].

[0111] [Mathematical Formula 8]

[0112]

[0113] Here, τ is the trajectory sampled using the current actors, and r t represents the reward at time step t, and the advantage function A t It can be defined as follows [Equation 9].

[0114] [Mathematical Formula 9]

[0115]

[0116] Here, t next is the next time step in which a new macro observation for an agent becomes possible.

[0117] Here, t next is the next time step in which a new macro observation for an agent becomes possible.

[0118] In synchronous multi-agent reinforcement learning, it is common practice to achieve a shared history abstraction by collecting shared observations from all agents at every time step. However, in an asynchronous setup, only some agents may provide observations at a given time step, while observations are absent for the remaining agents. Conventionally, the shared history was constructed by filling in missing observations with padding data (e.g., copies of the previous observation); however, this padding causes redundant information and spurious correlations, which undermines the accuracy of the shared history representation.

[0119] FIG. 5 is a diagram showing an agent-centric encoder of an agent-centric centralized critic according to one embodiment of the present disclosure.

[0120] Throughout this disclosure, i represents the agent index and t represents the global time step of the environment. represents a macro-observation or equivalent state feature acquired by agent i at time step t, and represents the time information (time position) associated with the observation. is the hidden history representation of Agent i. The output V of the central critique t is the value function estimate at time step t.

[0121] The history of each agent can be abstracted from macro observations using recurrent networks. However, in asynchronous multi-agent reinforcement learning, it is essential to consider the elapsed time between consecutive macro observations. Without this duration information, the history of each agent may be learned inaccurately due to varying durations, as there is a lack of context regarding how long the interval between consecutive macro observations is—for example, whether it is 1 time step or 10 time steps.

[0122] Referring to FIG. 5, in order to solve these problems, one embodiment of the present disclosure involves macro observation and the corresponding time step An agent-centered encoder (500) integrating may be provided. The agent-centered encoder (500) is a time step It is embedded using sinusoidal position encoding, and this embedded time step can be associated with a macro observation. The combined information is processed through a multilayer perceptron (MLP) and a gated recurrence unit (GRU) to effectively abstract the history of each agent. The recurrence module can be implemented as an LSTM or a Transformer Encoder (including a time mask) instead of a GRU.

[0123] In one embodiment, the agent-centered encoder (500) may include an input unit (510), a time embedding unit (520), a combination node (530), a multi-layer perceptron (MLP) module (540), a GRU (550), etc.

[0124] In one embodiment, the input unit (510) and Can receive. may be a compressed vector of a vectorized observation or feature map. The time embedding unit (520) is It can be converted into sine / cosine-based position embeddings, learning-based embeddings, or log scale / clipping, etc. The resulting vector is TE( It can be expressed as ). In one embodiment, the joining node (530) is wa TE( ) can be connected. If necessary, additive combinations may be used. The MLP (multi-layer perceptron) module (540) can perform non-linear transformations by mapping the combination vectors to multiple perceptrons. Activation functions may include ReLU, GELU, etc. The GRU (Gated Recurrent Unit, 550) is the previous time step hidden state With the output of the MLP module (540) as input, the current hidden state It can be calculated. In FIG. 5, a GRU is shown as a recurrent module, but this is merely an example, and LTSM, RNN, etc. may be used. In one embodiment, the output unit (560) is the calculated It can be provided to Agent i's actors and centralized critique. The hidden state is at the start of the episode. Or initialized with a learning vector, optionally reset at macro-episode boundaries (e.g., completion of a specific process step), or damping coefficient Multiply by Memory reduction can be applied.

[0125] According to one embodiment, the combined information is processed by the MLP module (540) and the GRU (550) so that the history of each agent can be effectively abstracted. In addition, according to one embodiment, time step information macro observation By integrating with, the agent-centric encoder (500) can ensure that the history of each agent is learned accurately and consistently even in asynchronous multi-agent reinforcement learning.

[0126] According to one embodiment, the hidden state of the GRU (550) is updated only for agents that have received a new observation, and for agents that have not received an observation, the previous hidden state It can be maintained as is. Therefore, the history of each agent is preserved continuously without the injection of padding data, and redundancy and distortion of the history representation can be suppressed.

[0127] According to one embodiment, since the interval between observations is directly expressed without padding injection, history distortion caused by asynchronous duration inconsistency is reduced, the hidden state is maintained continuously and consistently even with imbalance in learning data between agents, and the bias in value estimation in time-sensitive rewards is lowered so that policy convergence can be stabilized.

[0128] In one embodiment, the parameters of the agent-centered encoder (500) are shared among all agents to increase data efficiency, while additionally connecting agent identification embeddings can be used to correct individual characteristics. In another embodiment, the parameters of the agent-centered encoder (500) may be partially shared only among sets of agents with different roles, or may not be shared at all.

[0129] FIG. 6 is a diagram illustrating a method for calculating a value function by integrating agent-centered history according to one embodiment of the present disclosure.

[0130] Referring to FIG. 6, the agent-centered centralized critique may include agent-centered encoders (500-1, 500-2, ..., 500-N) that abstract the history of each agent and an attention-based aggregation module (640) that combines these abstracted histories. Each of the agent-centered encoders (500-1, 500-2, ..., 500-N) may correspond to the agent-centered encoder (500) described above with reference to FIG. 5.

[0131] In one embodiment, an agent-centric encoder may be provided for each agent. The agent-centric encoder for agent i is and Current hidden state with as input It can output the last valid hidden state, even for agents with no observations at the last time step t. At time step t, the centralized critique can be calculated according to the following [Equation 10].

[0132] [Mathematical Formula 10]

[0133]

[0134] Here, refers to timestep information for agent i's macro observations up to timestep t.

[0135] In one embodiment, the self-attention module (610) is in a hidden state By using as input, a first-order aggregate representation can be produced by weighting the importance of interactions between agents. For example, the self-attention module (610) maps input vectors to queries, keys, and values, and can produce a first-order aggregate representation by weighting the importance of interactions between agents. The self-attention module (610) learnably reflects the relative importance of interactions between agents, enabling more precise co-inference compared to fixed synthesis. Additionally, the hidden state of an unobserved agent is maintained without updating, and since the contribution in attention weights and average pooling becomes zero due to the mask m, duplicate inflow due to padding can be structurally blocked.

[0136] The average pooling module (620) can generate a global integrated representation by performing average pooling on the output of the self-attention module (610). In one embodiment, the output of the self-attention module (610) and the output of the average pooling module (620) may be combined or weighted summed. The MLP module (630) processes the integrated representation corresponding to the output of the average pooling module (620) using multiple perceptrons to perform a high-dimensional non-linear transformation, thereby generating a value function V t It can output. If necessary, layer normalization or scalar scale parameters may be added to stabilize the output scale.

[0137] That is, one embodiment of the present disclosure may organize the history of each agent on an agent-by-agent basis and then integrate these agent-by-agent histories into a common representation. Specifically, an agent-centric history encoder may be used to capture the macroscopic observation history of each agent. By aggregating this agent-centric history using an attention-based module, a more accurate common history representation for value estimation may be possible. By focusing on encoding the histories of each agent before combining them, according to one embodiment, cases where macroscopic observations are lacking can be effectively handled, and the limitations of padding-based methods can be overcome.

[0138] FIGS. 7a to 7c are drawings illustrating an Agent-Centric Actor-Critic (ACAC) method according to one embodiment of the present disclosure.

[0139] Referring to FIGS. 7a through 7c, at time step t=t1, the first agent (710) can acquire a new macro observation and, accordingly, select a new macro action. On the other hand, the second agent (720) and the third agent (730) cannot acquire a new macro observation because the previous macro action is still in progress. In this case, the first agent (710) that has acquired a new macro observation can input the macro observation into the encoder centered on the first agent to generate an updated history. Conversely, the second agent (720) and the third agent (730) that have not acquired a new macro observation can use the existing history as input to the synthesis module. Through this approach, central critics can accurately estimate the value function using the latest history information of each agent, even in an asynchronous multi-agent reinforcement learning setup.

[0140] For example, the first agent (710) obtains an observation at time step t=t1. and time step information indicating the time of acquisition of the observation or the location of the decision It can be input into the first agent-centered history encoder (500-1). The first agent-centered history encoder (500-1) is by the time embedding unit Converts into an embedding vector, and the embedding vector converted by the time embedding unit and Combines them, performs a non-linear transformation by the MLP module, and then the hidden state from all time steps by GRU Update to the current hidden state It can generate. The generated It can be provided to the policy output of the first actor and used to select a new macroscopic action, and at the same time, it can also be used in a centralized aggregation path.

[0141] On the other hand, since there is no observation for the second agent (720) and the third agent (730) at time step t=t1, each history encoder does not update the hidden state, and = , = The existing history can be maintained. The hidden representation maintained in this way can be input directly into the central aggregation path. In this case, padding may not be performed.

[0142] In one embodiment, at time step t=t1, the agent-specific history hiding representation{ , , } can be input to the aggregation module. The aggregation module can generate critique input features by weighting the relative importance of interactions between agents using a self-attention module, generating a global integrated representation using an average pooling module, and performing non-linear transformations using an aggregation MLP. Based on these features, the central critique has a value function V t1 It can generate.

[0143] In one embodiment, the update history of the first agent (710) that has acquired a new observation and the maintenance history of the second agent (720) and the third agent (730) that have no observation are considered simultaneously, so that the contribution of each history can be dynamically reflected in the aggregation module. Accordingly, even in an asynchronous multi-agent environment, the central critic can perform accurate value estimation by utilizing the latest and highly relevant history information.

[0144] FIG. 8 is a diagram illustrating an Agent-Centric Actor-Critic (ACAC) method according to one embodiment of the present disclosure.

[0145] Referring to FIG. 8, the ACAC system may include agent-centered distributed actors (810), agent-centered central critics (820), etc. Each agent may acquire macro observations at different times in an asynchronous environment.

[0146] In one embodiment, a system may be provided that includes an agent-specific history encoder and a central aggregation module to effectively process macroscopic observations acquired at different times by agent in an asynchronous multi-agent reinforcement learning environment. Even if some agents fail to acquire new observations at a specific time, previous history hiding is maintained without padding, so that a value function can be calculated precisely through aggregation based on self-attention and mean pooling.

[0147] In one embodiment, each agent may be equipped with an agent-centric history encoder. That is, a history encoder may be provided for each of the multiple agents. For example, if the multiple agents include a first agent and a second agent, the system may include a first history encoder corresponding to the first agent and a second history encoder corresponding to the second agent.

[0148] In one embodiment, an agent-oriented history encoder embeds temporal information, such as observation time and decision location, using positional encoding, combines the observation and temporal embeddings with an MLP, and updates the previous hidden state with a GRU to obtain the current history hidden representation h. i It can generate. The generated h i It is passed to the upper MLP policy head, and Agent i's policy π(h i A macro action based on ) can be selected. Only agents that have acquired a new observation update their GRU hiding, and the hiding of agents without observations is maintained, so that observation padding is not performed.

[0149] For example, when the first agent receives the first observation data at the first time point, the system can update the first agent's history information by inputting the first observation data and the corresponding time information into the first agent's history encoder. On the other hand, if the second agent's observation data is not acquired at the first time point, padding is not performed, and the previous hiding is maintained as is and used as the second agent's current history. That is, the second agent's history information can be maintained. Here, multiple agents may have different action start and / or duration times and may be asynchronous multi-agents that determine different information at different times.

[0150] In one embodiment, the history information of a plurality of agents is input into a synthesis module, and a value function can be calculated by performing self-attention and average pooling.

[0151] In one embodiment, h calculated by each agent encoder 1 , h 2 , h 3Hidden expressions such as the above can be input into the critique path. The critique may include another agent-centered history encoder and can generate the latest history by reflecting time information in a manner similar to the actor. This set of history can be input into the aggregation module (830).

[0152] In one embodiment, the aggregation module (830) is a module that aggregates the history information of each of a plurality of agents and may include a self-attention module and an average pooling module. The self-attention module may map input history into queries, keys, values, etc., to increase the relative importance of interactions between agents. The average pooling module may generate an aggregated representation by globally aggregating attention outputs. The MLP module may output a value function based on the aggregated representation.

[0153] In one embodiment, the history encoder of each agent may include an MLP module that processes observation data and time information corresponding to the observation data by combining them, and a GRU that processes the hidden state of the previous time point and the hidden state of the current time point in sequence.

[0154] In one embodiment, based on the output of the history encoder of the first agent, the policy of the first agent is determined, and the macro action of the first agent corresponding to the policy of the first agent can be determined.

[0155] In one embodiment, the system defines a set of macro decision time points l(0), l(1), ... and the advantage estimation at macro time point l(0) It can be calculated as. Here λ is the TD error, γ is the depreciation rate, and λ is a value between 0 and 1. The λ discount is accumulated based on the number of macro decision-making cycles, so that even if the duration of a macro action is long, the importance may not decrease excessively.

[0156] In one embodiment, policy updates can be performed using the clipping purpose of Proximal Policy Optimization (PPO). The value function is learned with a standard loss such as mean squared error, and if necessary, normalization (e.g., PopArt) can be applied to stabilize the value scale.

[0157] In one embodiment, an agent-centric actor-critic method can process the trajectory of each agent individually by using an agent-centric history encoder and perform attention-based centralized criticism to integrate this agent-specific information. According to one embodiment, by eliminating the redundancy inherent in padding-based methods, more accurate time abstraction, faster learning, and higher quality policies can be provided.

[0158] According to one embodiment, learning convergence is accelerated under sparse rewards, and higher accuracy can be achieved than with conventional padding-based approaches.

[0159] FIG. 9 is a schematic diagram of a system according to one embodiment of the present disclosure.

[0160] Referring to FIG. 9, the system may include a structure for performing multi-agent reinforcement learning. For example, the system may include a plurality of agents (910a to 910n), an agent-centric history encoder module (920), a centralized critique module (930), an actor-critic learning module (940), etc. The plurality of agents (910a to 910n) are entities that observe a state within an environment and perform actions, and each agent can select and execute a high-level action, i.e., a macro-action. Each macro-action is broken down into a plurality of low-level actions, i.e., micro-actions, and executed, and the execution time may vary from agent to agent. Accordingly, action selection between agents can be performed asynchronously. The system may collect history data including past state, action, and reward information of each agent.

[0161] In one embodiment, the agent-centric history encoder module (920) acquires history data including past state, behavior, and reward information of each agent, and can encode the collected history data including time point information. According to one embodiment, asynchronousness can be reflected by applying time point embeddings to each history data. To this end, the time point information may include behavior duration information, for example, the execution time of a macro behavior. Encoding can be performed using a Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), or Transformer-based network. The agent-centric history encoder module (920) can naturally handle asynchronousness to enable learning without padding. Therefore, by processing the independent history of each agent without using padding, incorrect correlations occurring in the prior art can be prevented.

[0162] In one embodiment, the centralized critique module (930) can integrate history data encoded from multiple agents based on a self-attention mechanism. Through this, the centralized critique module (930) can reflect the importance of each agent as a learnable weight and estimate a co-value function. The self-attention may include multi-head attention, thereby allowing the interaction between agents to be effectively modeled. The centralized critique module (930) can estimate a co-value function by applying learnable weights that reflect the importance of each agent. Through this, the system can perform accurate value estimation even in an asynchronous environment.

[0163] In one embodiment, the actor-critic learning module (940) can update the policy of each agent based on the co-value function estimated by the centralized critique module (930). The policy update is performed based on the Proximal Policy Optimization (PPO) algorithm, and clipping and entropy regularization may be applied. Additionally, the actor-critic learning module (940) can strengthen long-term strategies using a modified Generalized Advantage Estimation (GAE) that applies a discount rate at the macro action level.

[0164] In one embodiment, the system can prevent spurious correlation and improve learning stability and efficiency by performing learning without padding while considering the asynchronous nature between agents.

[0165] In one embodiment, the system identifies whether the environment is a sparse reward environment, and if identified as such, applies reward shaping to improve learning efficiency. Therefore, the system can achieve rapid convergence and high performance even in a sparse reward environment. Additionally, the system can dynamically change the reward shaping strategy according to the characteristics of the environment. For example, in a sparse reward environment, it can provide rewards by setting intermediate goals, while in a dense reward environment, it can maximize learning efficiency by maintaining the existing reward structure. Furthermore, by applying an adaptive algorithm that automatically adjusts reward shaping rules according to changes in the environment, the system can support stable learning in various simulation environments.

[0166] In one embodiment, the system can accelerate large-scale multi-agent learning through GPU-based parallel processing. The model can be lightweighted to enable execution on mobile devices or edge computing environments. To this end, the system can reduce model size and computational load by applying techniques such as network pruning, quantization, and knowledge distillation. This lightweighting can enable real-time learning and inference in environments with limited hardware resources, such as robot control, IoT devices, and autonomous vehicles.

[0167] In one embodiment, the system can operate not only on a single computing node but also in a distributed learning environment where multiple nodes are connected via a network. To this end, the system applies a parameter server structure to centrally aggregate and synchronize model parameters calculated at each node, thereby improving the learning speed in a large-scale multi-agent environment. This distributed processing method can enable efficient learning even in complex simulation environments containing thousands or more agents.

[0168] In one embodiment, the attention mechanism used in the centralized critique module may apply a hierarchical attention structure in addition to basic multi-head attention. For example, by first performing attention at the agent group level and then additionally applying inter-group attention, the system can reduce computational complexity and effectively model interactions even in environments with a very large number of agents. This structure can be particularly useful in large-scale team-based simulations performing collaborative tasks.

[0169] FIG. 10 is a block diagram illustrating the internal structure of an agent-centered history encoding module according to one embodiment of the present disclosure.

[0170] Referring to FIG. 10, the agent-centered history encoding module (920) performs the function of encoding history data collected from multiple agents while reflecting asynchronousness, and may include a history data input unit (1010), a time point embedding processing unit (1020), and an encoding network (1030).

[0171] In one embodiment, the history data input unit (1010) may receive history data including past state, action, and reward information of each agent. The history data is collected independently for each agent and may be recorded in units of macro-actions. Since each macro-action is broken down into multiple micro-actions and executed, the history data may have asynchronous characteristics. In one embodiment, the input may be processed without data loss by taking this asynchronous nature into account.

[0172] In one embodiment, the time embedding processing unit (1020) may apply a time embedding to reflect time information in the input history data. The time embedding processing unit (1020) may model the difference in execution time between agents by adding an embedding that indicates the time at which each state and action occurred. In addition to time information, the time embedding processing unit (1020) may additionally include the duration of the action to reflect the execution length of the macro action. The embedding method may be implemented as a positional embedding or a learnable embedding.

[0173] In one embodiment, the encoding network (1030) can convert history data reflecting time information into a high-dimensional representation. The encoding network (1030) may use a Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or Transformer-based structure. When the encoding network (1030) applies a Transformer-based structure, it can model long-term dependencies through a self-attention mechanism. Additionally, the encoding network (1030) can effectively process asynchronous data sequences and learn agent-specific behavior patterns.

[0174] In one embodiment, history data received from the history data input unit (1010) can be converted into a high-dimensional vector in the encoding network (1030) after asynchronous behavior is reflected through the time point embedding processing unit (1020). The result of this encoding is transmitted to a centralized critique module and can be used to model interactions between multiple agents.

[0175] FIG. 11 is a block diagram illustrating the internal structure of a centralized critique module according to one embodiment of the present disclosure.

[0176] Referring to FIG. 11, the centralized critique module (930) may include an encoding network (1110), a self-attention module (1120), a GAE computation unit (1130), a macro action processing unit (1140), etc. to reflect asynchronousness in multi-agent reinforcement learning and to efficiently perform policy updates.

[0177] In one embodiment, the encoding network (1110) receives a high-dimensional vector transmitted from an agent-centered history encoding module and can convert it into a form usable by a centralized critique module. The encoding network (1110) may apply a Transformer-based structure and may also apply recurrent structures such as RNN, LSTM, and GRU. The encoding network (1110) can normalize the input for modeling interactions between agents and prepare it to be combined with an attention mechanism.

[0178] In one embodiment, the self-attention module (1120) can model interactions between agents by integrating encoded history data. For example, when multi-head attention is used, relationships between agents can be learned from various perspectives using multiple attention heads. The self-attention module (1120) can estimate a co-value function by applying learnable weights that reflect the importance of each agent. Additionally, the self-attention module (1120) can enable efficient computation even in a large-scale agent environment by applying hierarchical attention.

[0179] In one embodiment, the GAE calculation unit (1130) can perform reward calculations suitable for an asynchronous environment based on Generalized Advantage Estimation (GAE). The GAE calculation unit (1130) can support stable learning by adjusting the discount rate according to the execution length of each macro action to strengthen long-term strategies and prevent reward distortion caused by asynchronousness.

[0180] In one embodiment, the macro action processing unit (1140) can break down the macro action selected by each agent into a plurality of micro actions and perform a policy update at the time of completion of execution. Additionally, the macro action processing unit (1140) can adjust the learning schedule based on the execution time (duration) of the macro action.

[0181] In one embodiment, the system processes inputs in an encoding network (1110), models interactions between agents through a self-attention module (1120), calculates rewards in a GAE computer (1130), and performs policy updates in a macro action processor (1140). This process can be combined with a Proximal Policy Optimization (PPO) algorithm to apply clipping and entropy regularization.

[0182] In one embodiment, a new type of encoder combining a position encoding technique may be used in the process of encoding and aggregating observation history collected from multiple agents. This allows time-stamped information to be effectively reflected in the history data of each agent and prevents information loss caused by asynchronous action execution. Additionally, the encoder according to one embodiment is designed so that actors and critics can share the same type of encoding network, thereby maintaining consistency in the network structure and improving learning efficiency.

[0183] In addition, in one embodiment, a self-attention mechanism may be applied during the process of abstracting the joint history. This allows for encoding by considering the interrelationships between the histories of each agent, and enables more sophisticated interaction modeling compared to simple data merging methods. In particular, by applying multi-head attention, interactions between agents can be learned from various perspectives, and if necessary, the hierarchical attention structure can be extended to support efficient computation even in large-scale agent environments.

[0184] In addition, in one embodiment, a new reinforcement learning model and algorithm framework based on an Agent-Centric Actor-Critic structure may be utilized based on encoding and aggregation methods. This structure can independently encode the observation history of each agent and then train a Centralized Critic Network and a Decentralized Actor Network through an aggregation module. Compared to existing simple centralized learning methods, this structure is robust to asynchronous operations and can provide high learning stability in a cooperative multi-agent environment.

[0185] In addition, in one embodiment, a Proximal Policy Optimization (PPO)-based algorithm may be applied instead of the existing Naive Actor-Critic algorithm to learn the agent's policy function. PPO ensures the stability of policy updates through clipping and entropy regularization and can significantly improve sample efficiency. Furthermore, the system can strengthen long-term strategies and prevent reward distortion caused by asynchronous behavior by modifying Generalized Advantage Estimation (GAE) to apply a discount rate at the macro-action level.

[0186] In one embodiment, the agent's workflow may include the types shown in FIGS. 12a through 12c. In one embodiment, most of the structure may consist of a hierarchical structure of "Input → Planning / Evaluation / Branching → Output". Additionally, due to the nature of Agentic, complex problem solving may be possible through patterns such as iteration, parallelism, and cooperation, rather than a single LLM call. Furthermore, since each structure is designed to suit a specific business purpose, selecting the optimal structure according to the purpose is important.

[0187] For example, a workflow like [Table 1] can be recommended depending on the purpose of the work.

[0188] [Table 1]

[0189]

[0190] FIG. 13 is a drawing showing a multi-agent discussion type system according to one embodiment of the present disclosure.

[0191] Referring to Fig. 13, a multi-agent discussion system is a system in which multiple agents present different opinions and select the most appropriate result by voting or evaluating it, thereby deriving the optimal solution based on discussion. That is, multiple agents perform discussion and evaluation based on user input and can output the optimal answer among them. Such a multi-agent discussion system can make the best choice by comparing various perspectives in a competitive structure and can be used in fields such as world simulation.

[0192] FIG. 14 is a diagram showing a Mixture-of-AI Agents system according to one embodiment of the present disclosure.

[0193] Referring to FIG. 14, in a hybrid AI agent system, multiple agents perform parallel processing layer by layer, and an aggregator can integrate the results at the end. For example, a first agent and a second agent may perform a process in parallel at the first layer, and then the first agent and the second agent may perform a process in parallel at the second layer, and an aggregator may synthesize these to generate an output. In one embodiment, the hybrid AI agent system may use a multi-stage approach for complex problems. The hybrid AI agent system has a hierarchical and parallel structure and is characterized by the distribution and combination of expertise, so it can be used for medical research, etc.

[0194] FIG. 15 is a drawing showing a multi-agent workflow system according to one embodiment of the present disclosure.

[0195] Referring to Fig. 15, a multi-agent workflow system can refer to a system in which multiple specialized agents cooperate to perform a single task. For example, upon receiving input from a user such as "Please write a startup market research report," the first agent can collect the latest market trends, the second agent can analyze competitors, and the third agent can summarize investment trends. An aggregator can synthesize the information generated by the first, second, and third agents to generate a report. The multi-agent workflow system can improve accuracy through a cooperative structure and distribute and process complex tasks.

[0196] FIG. 16 is a drawing showing a Multi-Agent Debate (MAD) system according to one embodiment of the present disclosure.

[0197] Referring to FIG. 16, a Multi-Agent Debate (MAD) system may refer to a system in which multiple small language models derive an answer through discussion. An aggregator, such as an aggregator LLM, can receive a query from a user and combine the opinions of multiple small language models to generate a final answer. In one embodiment, when a user inputs a question, the aggregator generates an initial answer, and multiple Small Language Models (SLMs) can present different answers and refute each other. For example, when the first SLM presents an answer such as "This is the answer," the second SLM presents "No, this is the answer. I verified it this way," and the third SLM presents "I think it is almost correct, but there are these points," the aggregator can make an intermediate judgment based on the discussion content. Based on this first discussion and previous judgment, a more refined second discussion can proceed. Some models can verify facts by utilizing tools (search, vector DB, etc.). After repeating these discussions, the aggregator can determine the most appropriate response as the final verdict and deliver it to the user. In one embodiment, various SLMs can participate in discussion and verification with each other.

[0198] FIG. 17 is a diagram showing an A2A (Agent2Agent) protocol system according to one embodiment of the present disclosure.

[0199] Referring to Fig. 17, the A2A (Agent2Agent) protocol enables communication without sharing data with each other, allows for task distribution and negotiation among multiple agents, and enables each agent to maintain shared context and state information. In the example of Fig. 13, the first AI agent (AI Agent 1) can primarily perform local-based file or search tasks and can connect to various MCP servers via the MCP Protocol. In the example of Fig. 13, the second AI agent (AI Agent 2) can primarily handle cloud and communication tasks and can connect to various MCP servers via the MCP Protocol. MCP allows for communication by separating each function (file access, search, cloud, etc.) into separate servers. Furthermore, A2A has the advantage of high security because it enables direct communication between agents without sharing data. Each agent can operate independently using its own language model, framework, and database.

[0200] One embodiment of the present disclosure may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, or program modules and includes any information transmission medium.

[0201] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that modifications can be easily made to other specific forms without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0202] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. As a system performed by a computer, At least one processor; It includes at least one memory that stores instructions that cause the system to perform operations when executed by the above-mentioned at least one processor, and The above operations are, An action of collecting history data including past state and behavior information of each of multiple agents; The operation of encoding the history data including time point information by a history encoding module; An operation to integrate the above-mentioned encoded history data by applying a self-attention mechanism; An operation to estimate a joint value function based on the above integrated history data; and A system comprising an operation to update the policy of each of the plurality of agents based on the above joint value function.

2. In paragraph 1, the above system is, A system characterized by performing learning based on independent history data of each of the plurality of agents without using padding.

3. In paragraph 1, the above system is, A system characterized by reflecting asynchronousness by applying an embedding containing time point information to the history data of each of the plurality of agents.

4. In Paragraph 3, the above-mentioned time point information is, A system including information on the duration of actions of each of multiple agents.

5. In paragraph 1, the history encoding module is, It includes at least one of an RNN (Recurrent Neural Network), an LSTM (Long Short Term Memory), a GRU (Gated Recurrent Unit), and a Transformer module, The above self-attention mechanism is A system including a multi-head attention mechanism.

6. In paragraph 1, the operation of updating the policy of each of the plurality of agents is, A system comprising an operation to update the policy of each of the plurality of agents based on a PPO (Proximal Policy Optimization) algorithm.

7. In paragraph 6, the above PPO algorithm is, A system characterized by applying clipping and entropy normalization.

8. In paragraph 1, the above operations are, It includes an action that determines a reward based on the GAE (Generalized Advantage Estimation) method, which applies a discount rate at the macro action level, and The above macro action is, A system characterized by being broken down into multiple micro-actions and executed.

9. In paragraph 1, the above operations are, An action to identify whether it is a sparsely rewarded environment; and A system comprising an operation to apply reward shaping when identified as the above-mentioned sparse reward environment.

10. In paragraph 1, the above system is, A system characterized by performing learning through GPU (Graphics Processing Unit)-based parallel processing.

11. In paragraph 1, the operation of estimating the joint value function is, A system comprising the operation of estimating the above joint value function by applying weights that increase the importance of each agent.

12. A method performed by at least one processor, A step of collecting history data including past state and behavior information of each of multiple agents; A step of encoding the above history data including time point information; A step of integrating the encoded history data by applying a self-attention mechanism; A step of estimating a joint value function based on the above-mentioned integrated historical data; and A method comprising the step of updating the policy of each of the plurality of agents based on the above joint value function.

13. In Paragraph 12, the above method is, A method characterized by reflecting asynchronousness by applying an embedding containing time point information to the history data of each of the plurality of agents.

14. In Clause 12, the step of updating the policy of each of the plurality of agents is, A step of updating the policy of each of the plurality of agents based on the PPO (Proximal Policy Optimization) algorithm; and A method comprising the step of determining a reward based on a Generalized Advantage Estimation (GAE) method that applies a discount rate at the macro action level.

15. A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 12 through 14 on a computer.

Images