Language model-based multi-agent industrial automation environment simulation method and device for performing same

Abstract

The present disclosure relates to a method for high-speed prototyping and control of an industrial automation environment. When a user places infrastructure components through a visual editor and configures agent tasks, a system serializes corresponding placement information and operating parameters into a structured text format. Subsequently, a hardware specification file is parsed to normalize state variables and actuator spaces into metadata, the normalized metadata and task inputs are fed into a pre-trained artificial intelligence model to automatically generate interface code that mediates data exchange between agents and a simulation environment, and thus a behavior model is linked to the simulation environment on the basis of the generated code, and an interaction result of an agent cluster is visually displayed on a user interface. Through this, it is possible to rapidly establish an environment and intuitively pre-verify a control algorithm without complex hard-coding.

Description

Language model-based multi-agent industrial automation environment simulation method and apparatus for performing the same

[0001] The present disclosure relates to artificial intelligence-based industrial automation control and simulation technology.

[0002] With the recent advancement of the Smart Factory and Smart Logistics industries, the importance of multi-agent systems—where tens to hundreds of autonomous guided vehicles (AGVs), manipulators, and automated equipment collaborate to perform tasks—is rapidly increasing. In these complex industrial environments, it is essential to test Multi-Agent Reinforcement Learning (MARL) algorithms and verify control policies in a simulation environment before deploying actual physical infrastructure. This is necessary to prevent unexpected bottlenecks or conflicts between agents and to optimize overall throughput.

[0003] However, the process of building conventional multi-agent simulation environments had limitations, requiring extreme time consumption and a high level of technical expertise. To implement the layout of a specific logistics warehouse or factory on a simulator, researchers or practitioners had to deeply understand the characteristics of physics engines (e.g., MuJoCo, Isaac Gym, etc.) and hardcode the environment's initial state and boundary conditions using complex scripts.

[0004] In particular, the biggest technical barrier lies in the interface gap between the simulator's physical output and the artificial intelligence control model. To train an agent, it is necessary to parse low-level hardware specification files (e.g., XML, URDF, etc.) that define the physical connectivity and kinematic constraints of environment objects. Subsequently, wrapper code had to be written manually to convert raw data, such as joint angles and sensor outputs, into an observation space recognizable by the reinforcement learning model, and to replace the model's outputs with an action space transmitted to the actuator. This process frequently leads to human errors, such as inconsistencies in tensor shape or index mapping errors.

[0005] Recently, there have been attempts to automatically generate code by utilizing Large Language Models (LLMs) to overcome the difficulties of writing such code. However, if only natural language-based task descriptions are injected into the LLM without the precise context of the hardware specifications of the physics engine, a phenomenon known as hallucination inevitably occurs, in which the LLM arbitrarily references non-existent system variables or confuses indices to generate physically impossible code.

[0006] Furthermore, conventional simulation platforms tend to be designed to be strongly dependent on specific reinforcement learning algorithms or single control paradigms. Consequently, when a user wants to objectively compare and evaluate the performance of heterogeneous behavioral models—such as existing rule-based heuristic algorithms, modern multi-agent reinforcement learning (MARL) policies, and large language model-based autonomous planners (LLM Planners)—within the same industrial environment, there is an inefficiency in having to rebuild the platform itself or modify a significant amount of system code.

[0007] Therefore, there is an urgent need to develop a new and integrated simulation platform technology that enables non-experts to intuitively prototype complex industrial environments through a visual interface, fundamentally blocks the illusions of LLM while automatically generating communication interface code between the simulator and behavioral models, and allows for the immediate plug-and-play integration and verification of any heterogeneous control logic through standardized API specifications.

[0008] The present disclosure is derived to solve the problems of the aforementioned background technology, and the first objective of the present disclosure is to provide a simulation method and apparatus that allows a user to prototype the layout of a multi-agent industrial environment through an intuitive visual editor without complex physics engine-based hardcoding or high technical expertise.

[0009] The second objective of the present disclosure is to provide a method and apparatus that ensure the construction of a complex simulation environment by serializing and storing visually designed environment information as machine-readable structured text.

[0010] The third objective of the present disclosure is to provide a method and apparatus for automatically generating interface code between a simulator and a behavioral model using a language model, wherein normalized metadata containing tensor dimension and index information is extracted from low-level hardware specifications and injected into the language model to eliminate hallucinations regarding system variable references.

[0011] The fourth objective of the present disclosure is to provide a method and apparatus that can dynamically bind heterogeneous behavioral models, such as heuristic algorithms, multi-agent reinforcement learning policy models, and autonomous planners based on large language models, to a runtime environment in a plug-and-play manner without code modification by utilizing generated interface code and standard application programming interface (API) specifications.

[0012] The fifth objective of the present disclosure is to provide a method and apparatus that support intuitive interactive verification by a user by synchronizing and providing an animation view representing the dynamic behavior of an agent swarm performed in a runtime environment and a live chart representing quantitative performance indicators to a dashboard.

[0013] A method for prototyping and controlling an industrial automation environment executed by a computer according to an embodiment of the present disclosure for solving the above-mentioned problem comprises: a step in which at least one processor of the computer visually places a plurality of infrastructure components in a virtual workspace and sets detailed operational parameters through a visual editor provided to a user computing device, and receives user input regarding a task to be performed by an agent cluster; a step in which the at least one processor loads a hardware specification file corresponding to the infrastructure components stored in at least one memory of the computer based on the received input; a step in which the at least one processor combines spatial placement information of the placed components and operational parameter values ​​to generate data in a standardized structural text format having a hierarchical structure; a step in which the at least one processor parses the loaded hardware specification file to extract the state variable space and actuator space of the agent cluster and normalizes them into machine-readable metadata; and a step in which the at least one processor inputs the normalized metadata and the user input regarding the task into at least one pre-trained artificial intelligence model to generate an interface code that mediates data exchange between the agent cluster and a simulation environment. The above at least one processor may include the step of linking at least one behavioral model determined via the generated interface code to a simulation environment, and implementing content representing the interaction result of the agent cluster performed in the simulation environment through the user interface of the user computing device.

[0014] According to the embodiments of the present disclosure, the following technical effects can be obtained.

[0015] First, core infrastructure components of industrial sites can be spatially mapped and environment variables set using only intuitive user input events without writing complex code, enabling the prototyping of simulation environments and lowering construction time and entry barriers.

[0016] Second, by serializing visually designed environment information into a machine-readable hierarchical file (e.g., YAML format) and storing it as an asset, the reproducibility of the experimental environment is ensured, and environment settings can be easily shared among multiple operators.

[0017] Third, by automatically extracting highly refined metadata containing tensor dimension and index mapping information from low-level hardware specifications and injecting it into a large-scale language model as a constraint of system prompts, the illusion that the artificial intelligence model arbitrarily references system variables and malfunctions can be suppressed, thereby enabling the automatic acquisition of physically valid and intact interface wrapper code.

[0018] Fourth, by modularizing environment objects and agent control policies and thoroughly separating them into a multi-agent standard API interface, it provides excellent flexibility and scalability that allows heterogeneous control paradigms, such as heuristic algorithms, reinforcement learning (RL) policies, and LLM autonomous planners, to be immediately replaced and mixed without hardcoding and objectively cross-verified.

[0019] Fifth, by providing a main view that visualizes the qualitative physical behavior of agents and a chart view that displays the system's quantitative performance indicators (e.g., throughput, cumulative rewards) in real-time synchronization, it supports users in intuitively identifying local collisions or bottlenecks between robot swarms and making rapid decisions to deploy the optimal model to actual robots.

[0020] FIG. 1 illustrates an example of a block diagram of a computing system in which one embodiment of the present disclosure can be implemented.

[0021] FIG. 2a illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an example of a computing system in which an embodiment of the present disclosure may be implemented.

[0022] FIG. 2b is a block diagram illustrating the internal architecture and data processing pipeline of an artificial intelligence model according to one embodiment of the present disclosure.

[0023] FIG. 3a illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system performing an industrial automation environment simulation method according to one embodiment of the present disclosure.

[0024] FIG. 3b is a block diagram illustrating the data flow and system interaction of an industrial automation environment simulation service application process according to one embodiment of the present disclosure.

[0025] FIG. 4 is a diagram illustrating the flow of industrial automation environment simulation and optimization steps according to one embodiment of the present disclosure.

[0026] FIG. 5 is a flowchart of an industrial automation environment simulation method according to one embodiment of the present disclosure.

[0027] FIG. 6 is a schematic diagram illustrating a visual editor-based layout design interface according to one embodiment of the present disclosure.

[0028] FIG. 7 is a block diagram illustrating the standardization of designed environment layout data and the multi-agent standard interface (API) mapping process based thereon according to one embodiment of the present disclosure.

[0029] FIG. 8 is a schematic diagram showing a simulation dashboard main interface and a multi-window structure user interface for selecting a behavior model according to one embodiment of the present disclosure.

[0030] FIG. 9 is a flowchart of a method for automatically generating a robot control interface according to one embodiment of the present disclosure.

[0031] FIG. 10 is a diagram illustrating the process of converting a structured prompt group according to one embodiment of the present disclosure into an interface code group generated by inputting it into a large-scale language model.

[0032] FIG. 11 is a diagram showing the modular structure of a large-scale language model entire prompt according to one embodiment of the present disclosure.

[0033] FIG. 12 is a diagram illustrating the process of selecting an optimal candidate code through an optimistic uncertainty-based resource allocation mechanism according to one embodiment of the present disclosure.

[0034] FIG. 13 is a performance evaluation graph comparing the learning efficiency between an interface automatic generation framework according to one embodiment of the present disclosure and a prior art.

[0035] FIG. 14 is a diagram illustrating the process of setting up a specific reinforcement learning experimental environment and algorithm to match a task to be achieved by a user according to one embodiment of the present disclosure.

[0036] FIG. 15a is a drawing illustrating a screen before the start of project learning according to one embodiment of the present disclosure.

[0037] FIG. 15b is a diagram illustrating an experiment generation and dataset selection interface for multi-phase learning according to one embodiment of the present disclosure.

[0038] FIG. 16 is a drawing illustrating a screen during or after learning according to one embodiment of the present disclosure.

[0039] FIG. 17 is a schematic diagram illustrating the process of heterogeneous reinforcement learning framework integration and target algorithm selection according to one embodiment of the present disclosure.

[0040] FIG. 18 is a schematic diagram showing the simulation dashboard operation and real-time verification screen of a multi-agent system according to one embodiment of the present disclosure.

[0041] FIG. 19 is a drawing illustrating a screen for storing learning results according to one embodiment of the present disclosure.

[0042] FIG. 20 is a flowchart illustrating the control result of an artificial intelligence model having a plurality of learning phases according to one embodiment of the present disclosure.

[0043] FIG. 21 is a drawing illustrating a screen that displays on / offline learning performance separately according to one embodiment of the present disclosure.

[0044] FIG. 22 is a drawing illustrating a multi-phase learning status and lineage management interface according to one embodiment of the present disclosure.

[0045] FIG. 23 is a drawing illustrating a user interface for checkpoint-based interactive evaluation and verification according to one embodiment of the present disclosure.

[0046] FIGS. 24 and 25 are drawings illustrating a performance comparison graph before and after the application of a multi-stage learning pipeline according to one embodiment of the present disclosure.

[0047] As the present disclosure is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a limiting sense but for the purpose of distinguishing one component from another. Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as "include" or "have" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, so the present disclosure is not necessarily limited to what is depicted.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0049]

[0050] First, the system configuration for automatically generating a robot control interface is described in detail.

[0051] The aforementioned modules, models, and operations may be implemented through at least one program or machine learning model, stored in at least one memory, and operated by a processor.

[0052] One embodiment of the present disclosure may be an advanced AI agent system that performs tasks through a single machine learning model, as well as autonomously formulates plans, utilizes internal and external tools, learns the user's context through long-term and short-term memory, and performs complex tasks through the cooperation of multiple agents.

[0053] Below, a method for automatically generating such a robot control interface and a configuration for a system executing the same are described.

[0054] FIG. 1 illustrates an example of a block diagram of a computing system in which the present disclosure can be implemented.

[0055] Referring to FIG. 1, a computing system (1000) performing a method for automatically generating a robot control interface according to one embodiment of the present disclosure may include a user computing device (110), a server computing system (130), a training computing system (150), a simulation environment (180), and a robot / physical agent (190), and each of the components may be connected to communicate through a network (170).

[0056] In one embodiment, the computing system (1000) may include a large-scale language model (LLM). The LLM may be implemented as part of a machine learning model (140) installed inside a server computing system (130), or may be provided by communicating with an external LLM providing system (not shown) via a network (170).

[0057] The user computing device (110) receives task descriptions, robot hardware / environment specifications, learning settings, evaluation requests, etc. from the user and transmits them to the server computing system (130), and can display the learning progress status and evaluation results (e.g., graphs / videos / logs) provided by the server (130). In addition, the user computing device (110) can perform some inference locally by having a local machine learning model (120), or call the machine learning model (140) of the server (130) to perform automatic generation of a robot control interface.

[0058] The server computing system (130) can perform backend services for experiment orchestration, prompt generation, LLM-based code generation (e.g., observation model functions and behavior model functions), management of candidate code / models, checkpoint storage, and provision of a user interface. The server (130) can generate code by internally loading an LLM or by interacting with an external LLM providing system, and can control the loading of the generated code and related configuration information into the training computing system (150) and / or simulation environment (180).

[0059] The training computing system (150) can perform multi-stage reinforcement learning including offline and online learning using a model trainer (160) and training data (161), and transmit the learned policy / weight / hyperparameter and performance indicators of the generated code to the server (130). The training computing system (150) can generate rollout data in conjunction with a simulation environment (180) or perform online fine-tuning using sensor data collected from an actual robot / physical agent (190).

[0060] The simulation environment (180) may include a simulation engine (181) and a digital twin (183) to simulate the dynamics and working environment of a robot / physical agent (190), and may generate observation / reward / termination signals to provide to a training computing system (150) and / or a server (130). The robot / physical agent (190) may include a sensor (191), an actuator (192), a local controller (193), a processor (195), a communication interface (196), and a memory (197), and may perform physical actions according to policies and interface codes loaded from the server (130) or the training computing system (150), and provide the results of the execution as feedback through a network (170).

[0061] According to an embodiment of the present disclosure, 1) a user computing device (110) can perform a method for automatically generating a robot control interface by using a local or / and external machine learning model (120) or a machine learning model (140) provided by a server.

[0062] Additionally, according to another embodiment of the present disclosure, 2) a server computing system (130) communicating with a user computing device (110) may provide a robot control interface automatic generation service to the user computing device (110) on an application or / and the web in response to a request from a user through the user computing device (110).

[0063] In addition, according to another embodiment of the present disclosure, 3) a user computing device (110) and a server computing system (130) may perform at least a part of the robot control interface automatic generation method in conjunction with each other to provide a robot control interface automatic generation service to the user.

[0064] Additionally, according to various embodiments of the present disclosure, a user computing device (110) and / or a server computing system (130) can learn a machine learning model (120 / 140) that performs automatic generation of a robot control interface through interaction with a training computing system (150) that is communicated via a network (170).

[0065] Additionally, in embodiments of the present disclosure, machine learning models (120, 140) may include various forms of AI agents or agentic architectures beyond simple prediction models.

[0066] In one embodiment, the machine learning model may be an AI agent having a structure that receives system prompts and user prompts, autonomously plans and executes tasks through core components such as planning, memory, reasoning, and tools, and improves itself through feedback.

[0067] In another embodiment, the machine learning model may include a Search Augmented Generative (RAG) architecture that retrieves relevant information from an external database and generates a response based thereon to provide an accurate answer based on the latest information or expertise.

[0068] In another embodiment, the machine learning model may be a multi-agent system in which multiple AI agents cooperate to achieve a specific goal. The multi-agent system may have a supervisory pattern in which a central supervisor agent distributes tasks to subordinate expert agents and aggregates the results. Alternatively, it may have a hierarchical pattern in which a meta-agent acts as an intermediary manager to control and coordinate subordinate agents. Furthermore, it is possible to include a multi-agent debate pattern in which multiple agents present different opinions and derive an optimal conclusion through discussion and evaluation.

[0069] The server computing system (130) can host AI agents such as those mentioned above, particularly multi-agent systems requiring complex computations or large-scale long-term memory. Additionally, the server computing system (130) includes a Multi-Channel Processing (MCP) server to manage integration with various external tools, and can relay communication with cloud APIs, payment services, search engines, etc.

[0070] The training computing system (150) can generate a ToolFormer model that learns how to use a specific tool, or perform iterative learning that gradually improves the performance of the agent through a self-reflection mechanism in which another LLM evaluates and modifies the results generated by the agent.

[0071] At this time, the training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0072] In some embodiments, the training computing system (150) may be part of the server computing system (130) or part of the user computing device (110). The user computing device (110) may include all other types of computing devices, such as a smartphone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0073] The user computing device (110) includes at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one of a central processing unit (CPU), a Tensor Processing Unit (TPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0074] The memory (112) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs memory storage functions on the internet. This memory (112) may store data and instructions necessary for the at least one processor (111) to perform the operation of an application for performing automatic robot control interface generation.

[0075] In one embodiment, the user computing device (110) may store at least one machine learning model (120). For example, the user computing device (110) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform automatic generation of a robot control interface based on structured / quantitative data, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0076] For example, the prediction model may store linear regression, decision tree, random forest, gradient boosting, or / and deep learning-based robot control interface auto-generation models. The neural network model may include feedforward neural networks, recurrent neural networks (e.g., LSTM), convolutional neural networks, Transformers, or / and combinations thereof.

[0077] In one embodiment, the user computing device (110) can perform automatic robot control interface generation in real time by utilizing a locally stored automatic robot control interface generation model.

[0078] In another embodiment, a user computing device (110) transmits a request for automatic generation of a robot control interface to a server computing system (130), and the server processes time series data based on at least one prediction model and provides the prediction result to the user device.

[0079] The user computing device (110) may include one or more input components (121) that detect user input. For example, the input components (121) may include a touch sensor, an image sensor, a microphone, a button, a mouse, a keyboard, etc., and may include an interface capable of receiving input from an external controller (e.g., a mouse, a keyboard, etc.).

[0080] A server computing system (130) may include at least one processor (131) and memory (132). Here, the processor (131) may be composed of one or more electrical computing devices for performing implicit policy alignment-based language model optimization operations, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), or an FPGA, a microprocessor, etc.

[0081] The memory (132) may include non-transient or transient storage media such as RAM, ROM, flash memory, magnetic disk, etc., and may store instructions or data for the execution and inference of a robot control interface automatic generation model executed by the processor (131).

[0082] The prediction model (140) may include at least one of a feedforward neural network, a recurrent neural network (LSTM), a convolutional neural network (CNN), or a Transformer-based model.

[0083] In one embodiment, the machine learning model(s) (140) may include a large-scale language model (LLM), a reinforcement learning policy model, a value function model, a reward model, or a combination thereof. For example, the LLM may interpret natural language input to generate control interface code such as an observation model function and a behavior model function, and the LLM may be included within a server computing system (130) or provided via API integration with an external LLM providing system through a network (170).

[0084] In one embodiment, the server computing system (130) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0085] The server computing system (130) may further include a data store. This data store may be composed of a relational database, a NoSQL database, a data warehouse, or a local file system, and may be used to store and manage, for example, prompt templates, robot / environment specifications, candidate code, training datasets, training logs, model checkpoints, and evaluation results.

[0086] The training computing system (150) may include at least one processor (151) and memory (152). The memory (152) may store a dataset for model training, a loss function, an optimization algorithm, parameters, etc.

[0087] For example, the training computing system (150) can update the parameters of a model that performs automatic robot control interface generation by executing a learning algorithm including backpropagation. This process may include iterative learning to improve the performance of the model that performs automatic robot control interface generation, and generalization techniques such as dropout and weight reduction may be applied.

[0088] The model trainer (160) is a logical component that executes this learning and can be implemented in the form of hardware, firmware, or software. For example, the model trainer can be executed by a processor after loading a learning program file stored in a storage device into memory.

[0089] The system can be connected via a wired / wireless network (170) for communication between a user computing device (110) and a server computing system (130). This network may include various forms such as the Internet, LAN, wireless LAN, WAN, and mobile communication network, and data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, and SSL.

[0090] This server system (130) may further include a plurality of logically and physically separated specialized engines and repositories to perform automatic generation of a robot control interface.

[0091] In one embodiment, the server system (130) may include at least one of a reasoning engine that processes a user's natural language input or system event to establish an action plan, a membership management engine, and a supervision signal generation engine.

[0092] Here, the term engine may include not only a set of instructions executed by a processor (131) to perform specific logic, but also dedicated hardware circuits to accelerate said logic.

[0093] This robot control interface automatic generation engine can be executed on hardware accelerators optimized to handle the computational load of large-scale generative artificial intelligence models. The hardware accelerator is a processor specialized in matrix operations and vector processing and may include at least one of the aforementioned Tensor Processing Unit (TPU), Graphics Processing Unit (GPU), Field-Programmable Gate Array (FPGA), or Application-Specific Integrated Circuit (ASIC). This hardware accelerator can provide technical improvements that distribute the computational load of large-scale language models (LLM) and / or diffusion models and enable real-time inference.

[0094] Additionally, the data repository (133) may include a structured embedding repository to support search augmentation generation (RAG) of the AI ​​model, rather than a simple set of data. The embedding repository stores high-dimensional vector representations of text, code, or audio data, thereby enabling the engine performing automatic robot control interface generation to perform high-speed search based on semantic similarity. This can serve as a technical means to suppress hallucinations in the model and increase the accuracy of the output.

[0095] Specifically, the memory (132) of the server system (130) may include a structured Knowledge Base Layer to physically support Search Augmentation Generation (RAG). The Knowledge Base Layer may include an Embedding Repository that stores high-dimensional vector representations of unstructured text, code, or multimodal data, and a Policy Document Repository that stores agent behavior constraints and business rules.

[0096] At this time, the robot control interface automatic generation engine can be configured to vectorize a query received from a user computing device (110) and query the embedding repository to retrieve context information with high semantic similarity in real time. This structure can technically suppress the hallucination phenomenon of the artificial intelligence model by allowing the agent to refer to external verified knowledge rather than relying only on intrinsic parameters.

[0097] Additionally, the user computing device (110) may include a trigger event detector that provides an interface for interaction with the agent and initiates the agent's operation. The trigger event detector can detect not only user input but also the arrival of a specific time, a change in the state of an external system, etc., and transmit a processing request to the server system (130).

[0098] Additionally, the model trainer (160) of the training computing system (150) may include a Supervision Signal Engine. The Supervision Signal Engine may compare the output generated by the agent (e.g., Raw Output) with a verified result (e.g., Grounded Output) obtained through an external tool (e.g., search engine, API) to calculate a difference value, and execute a reinforcement learning process to update the Reward Model or fine-tune the agent model based on this.

[0099] Additionally, the simulation environment (180) may include a physics engine-based simulation engine (181) and a digital twin (183) that virtually replicates the structure / dynamics of the actual robot and workplace. The simulation engine (181) receives a robot model (e.g., URDF, MJCF, XML) and environment parameters as input, generates state, observation, reward, and done signals at each simulation step, and these data may be provided to a training computing system (150) and / or a server computing system (130). Additionally, the digital twin (183) may be configured to reflect the kinematics and sensor configuration of the actual robot / workplace to correct parameters (e.g., friction, mass, sensor noise) so that the simulation results can be transferred to the real environment, or to represent the robot state through the same interface.

[0100] The robot / physical agent (190) may include a sensor (191), an actuator (192), a local controller (193), a processor (195), a communication interface (196), and a memory (197). The sensor (191) may generate data such as joint position / velocity, force / torque, and vision sensor data, and the local controller (193) may be configured to preprocess the sensor data or apply safety constraints (e.g., torque limit, collision prevention) and apply control commands calculated by a higher-level policy (e.g., reinforcement learning policy or interface function code) to the actuator (192). Additionally, the communication interface (196) may be configured to enable the robot / physical agent (190) to communicate bidirectionally with a server (130) or a training computing system (150) to perform actions such as loading policies / codes, streaming sensor data, and transmitting evaluation results.

[0101] As such, in one embodiment, the system (1000) of the present disclosure may be implemented not as a simple set of software algorithms, but as a technical system in which specialized hardware accelerators, vectorized data storage, and physical engines controlling them are organically combined.

[0102] FIG. 2a illustrates a block diagram example of a computing device (100) that may be included in a user computing device (110), a server computing system (130), and a training computing system (150), as an embodiment of a computing system (1000) in which the present disclosure can be implemented.

[0103] The computing device (100) may include a plurality of applications (e.g., Application 1 to Application N), and each application may include a machine learning library and a model execution environment for performing machine learning-based time series processing functions.

[0104] For example, the computing device (100) may include a model that performs automatic generation of a robot control interface, and an application that provides related services to the user, namely a specialized application for automatic generation of a robot control interface.

[0105] In one embodiment, the computing device (100) may include a model trainer (160) for pre-training or retraining a model that performs automatic generation of a robot control interface.

[0106] The model trainer (160) can learn or update parameters based on input time series data and save the learned model to be used for real-time or non-real-time prediction tasks.

[0107] Each application included in the computing device (100) can communicate with sensors, device status managers, or external input devices within the device through an API (Application Programming Interface).

[0108] In one embodiment, each application can perform the role of an interface with device components, such as receiving sensor data or status data through a public or private API, or transmitting prediction results to an output device.

[0109] FIG. 2b is a block diagram illustrating the internal architecture and data processing pipeline of an artificial intelligence model according to one embodiment of the present disclosure.

[0110] Referring to FIG. 2b, a machine learning model (120 / 140) of a computing device (100) or a server computing system (130) may have a pipeline structure that receives input data (202) and generates output data (212) through a series of transformation processes to perform automatic generation of a robot control interface. The input data (202) may include, for example, a user task description, robot hardware specifications, environment variable information, learning / evaluation settings, or a combination thereof, and the output data (212) may include, for example, a natural language response, observation / behavior model function code, learning control command, or a result to be displayed on a user interface. The pipeline may be performed through a preprocessing module (204), an encoder / embedding model (206), a neural network layer (208), and a decoder / generation head (210).

[0111] First, the preprocessing module (204) receives input data (202) (e.g., system prompt, user prompt, robot specification metadata, image, or multimodal signal) from a user or system. The preprocessing module (204) performs tokenization and normalization on the input data to generate a sequence of tokens, which are the smallest units that the model can process.

[0112] Next, the encoder / embedding model (206) receives the generated token as input and converts it into a vector / embedding mapped to a number in a high-dimensional vector space. At this stage, the discrete information of the input data is converted into a continuous numeric matrix, making it a form that can be computed by the machine learning model.

[0113] Next, the neural network layer (208) receives the vector / embedding and performs deep computation. The neural network layer (208) may have a structure in which a plurality of sub-layers (e.g., Layer 1 to Layer N) are stacked. Each layer abstracts and refines input features through an attention mechanism or convolution operation, etc.

[0114] In particular, the final output of the neural network layer (208) is defined as a latent representation. This latent representation has a structure different from the original input data (202) and may correspond to an intermediate representation in which the semantic features of the data are highly compressed and abstracted. This implies that it is not a simple transmission of data, but a technical data structure that is valid only within the system.

[0115] Finally, the decoder / generation head (210) receives the latent representation as a conditioning input. The decoder / generation head (210) interprets the compressed latent representation and reconstructs or generates output data (212) in a form recognizable by the user. For example, the output data (212) may include (i) executable control interface code such as observation model functions (compute_obs) and behavior model functions (compute_action), (ii) evaluation results of candidate code, (iii) natural language descriptions to be provided to the user, or (iv) control parameters to be applied to the robot / simulation.

[0116] Such a stepwise data transformation structure (token, vector / embedding, latent representation, and output sequence) can clearly demonstrate operation as a concrete device that technically processes input data to generate executable outputs (e.g., code, control parameters), rather than a simple sequence of operations.

[0117] FIG. 3a illustrates an example of a block diagram in another aspect of a computing device, which is one of the components of a computing system (1000) that performs a method for automatically generating a robot control interface according to an embodiment of the present disclosure.

[0118] The computing device (200) may include a plurality of applications (e.g., Application 1 to Application N), and each application may communicate with a central intelligence layer.

[0119] In one embodiment of the present disclosure, the application may include a robot control interface automatic generation application, an energy management application, a logging and analysis application, etc.

[0120] Each application can interface with shared models within the central intelligence layer through APIs (e.g., common APIs).

[0121] The central intelligence layer includes a machine learning model that performs the automatic generation of one or more robot control interfaces, and can share this among multiple applications or provide it independently to each.

[0122] In one embodiment, the central intelligence layer may be integrated as part of the operating system or implemented as a separate logical layer, and may perform the role of transmitting input time series data to the corresponding model to return a prediction result.

[0123] In addition, the central intelligence layer can communicate with the central device data layer. The central device data layer can integrate and store sensor data, device status information, and external environment information stored within computing devices, and provide this as input data necessary for the automatic generation of robot control interfaces. Each device component (e.g., sensors, state managers, etc.) can communicate with the corresponding data layer via private APIs, etc.

[0124] The technology described in this specification may be composed of a single or multiple computing devices, and a model for performing industrial automation environment simulation may be executed sequentially or in parallel on one component or multiple distributed components.

[0125] Data repositories, predictive models, and applications can operate in a distributed configuration locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0126] FIG. 3b is a block diagram illustrating the data flow and system interaction of a robot control interface automatic generation process according to one embodiment of the present disclosure.

[0127] Referring to FIG. 3b, the computing system may be configured as an organic data pipeline between a trigger event detector (310), an inference engine (330), and a mobile device screen (350).

[0128] First, the trigger event detector (310) is configured to monitor and detect a signal initiating the operation of the system. The trigger event detector (310) receives at least one of (i) a time event indicating the arrival of a specific point in time, (ii) a user action resulting from a user's physical input, or (iii) a system state indicating a change in internal system data, and transmits an activation signal to the inference engine (330). This means that the service can be actively initiated depending on the situation without an explicit request from the user.

[0129] The reasoning engine (330) performs a multi-stage operation process that transforms raw data into a final result in response to the activation signal. This process is implemented as a series of logically connected prompt chains.

[0130] For example, the first prompt, contextualization (332), allows the inference engine (330) to receive unstructured raw data (e.g., user logs, channel metadata), analyze and refine it, and generate compressed summary information.

[0131] And the second prompt, Content Generation (334), can generate multiple candidate results that match the user's intent or situation using a generative AI model based on the summary information generated above.

[0132] In addition, the third prompt, Verification (336), performs filtering and verification by comparing the generated candidate results with a predefined policy (e.g., safety guidelines, format rules) to derive a reliable final result.

[0133] Finally, the final result generated by the inference engine (330) can be transmitted to and implemented on the mobile device screen (350) through an auto pre-filling (354) operation. Specifically, the system changes the state of the interface by directly writing the final result to a memory address of a specific target field (352) (e.g., text input field, setting value) within the mobile device screen (350). Here, the mobile device screen (350) may be an example of a user computing device (110).

[0134] Such a configuration can go beyond the simple display of information and provide specific technical means for data generated by an external trigger to physically control and complete the input interface of the user terminal.

[0135]

[0136] The definitions of key terms used in this specification are as follows. However, the following definitions are intended to aid in understanding the present disclosure and may be interpreted flexibly within the scope of what a person skilled in the art perceives according to the context.

[0137] An Integrated Platform refers to a system that integrates and manages the entire process from user task definition to interface code generation, experimental environment configuration, and the execution and evaluation of multi-stage reinforcement learning, and may encompass a distributed computing environment including user computing devices, servers, and training computing systems. In the following description, unless specifically designated as a subject, a processor may be understood as at least one processor of the aforementioned server computing system.

[0138] Additionally, the interface code is software code that mediates the interaction between a reinforcement learning agent, which is a robot interface learning agent, and a simulation environment, and may include an observation model function that converts raw data into observation vectors and an action model function that converts the output of a policy into actuator commands.

[0139] Additionally, the Re-modeled Environment may refer to an execution environment having a redefined Observation Space and Action Space, in which the interface code is combined in the form of a wrapper with a simulation environment or an actual physical environment.

[0140] Furthermore, multi-phase learning may refer to a strategy of sequentially performing multiple learning phases to complete at least one agent model. This includes an offline phase utilizing a static dataset and an online phase learning through interaction with the environment, and parameter transfer may be performed during transitions between phases.

[0141] In addition, racing-based selection refers to a process of performing parallel learning on multiple interface code candidate sets generated by a language model within limited resources and selecting the optimal candidate based on statistical indicators such as the upper confidence bound (also known as LOAM-Race).

[0142] In addition, paradigm-aware monitoring may refer to a user interface function that identifies the attributes (offline / online) of the learning phase currently in progress and automatically distinguishes and displays the X-axis of the learning curve (e.g., Gradient Step vs. Interaction Step) or the visualization area.

[0143] Hereinafter, the execution process of the integrated platform for robot control will be explained in detail with reference to Fig. 4.

[0144] Referring to FIG. 4, the task definition and environment setting step performed by the processor of the integrated platform system is a process in which the integrated platform receives basic data for a reinforcement learning experiment from a user, obtains hardware information of a robot from a simulation environment, and prepares for the subsequent step of automatic interface generation (S200).

[0145] First, the processor of the user computing device can provide a project creation interface through the user computing device (such as a web browser). Through this interface, the user can select the target environment and robot model that serve as the purpose of the experiment.

[0146] In one embodiment, the interface may automatically play and provide a preview video on one side of the screen that exemplifies a target action (e.g., walking, picking up, pushing, etc.) that an agent must perform in a selected environment, so that the user can intuitively recognize the content of a task that is difficult to understand from text alone.

[0147] The user enters a task description in the form of natural language text into an input field provided by the system. For example, specific goals and constraints such as "control the humanoid robot to run forward at a speed of 5 m / s without falling over" can be entered.

[0148] Next, the processor of the server computing system, having received information regarding task definition and environment settings from the user computing device, can parse hardware specifications and obtain available variable information. When the user confirms the environment and the robot, the processor of the server can access the backend of the corresponding simulation environment (e.g., MuJoCo, Isaac Gym, etc.) to load and parse the robot's hardware specification file (XML, URDF, MJCF, etc.). The processor can extract and normalize the following available variable information from the parsed data.

[0149] Here, State Variables is a list of all observable data provided by the simulator. This includes the robot's joint position (Generalized Position, qpos), joint velocity (Generalized Velocity, qvel), 3D coordinates of the body and end effector (xpos), attitude information (xquat, xmat), and force sensor data (sensordata), etc.

[0150] Additionally, Actuator Info is a list of the robot's actuators and controllable channel information. It includes the ID, physical control range, gear ratio, and index mapping information of the corresponding joint for each actuator.

[0151] In addition, the physical object information (Body / Geom Info) includes information regarding dynamic characteristics such as the robot's link structure, mass, and moment of inertia.

[0152] And the processor of the above server computing system can normalize data and construct prompt-based data. The processor converts the extracted available variable information into a structured text format (e.g., JSON schema or Markdown table) that a large-scale language model (LLM) can understand.

[0153] At this time, for each variable, the data type, tensor dimension (Shape), and physical meaning (Description) are combined in the form of annotations.

[0154] For example, for the data.qpos variable, metadata such as Shape: (batch, 76), Description: Generalized positions including joint angles and base pose is generated and mapped.

[0155] In this way, the natural language task description and available variable information collected and normalized in step S100 are subsequently used as the core material (Context) for configuring the system prompt in step S200, helping the language model to write accurate code based on physical entities.

[0156] Next, the processor of the integrated platform system can automatically generate and optimize interface code. (S200)

[0157] Specifically, this is a process of generating executable interface code from information collected in the S100 stage using a Large Language Model (LLM) and selecting the optimal code from a pool of candidates. This stage can be performed specifically through the following detailed processes.

[0158] The processor can combine normalized available variable information and task description in step S100 to generate a structured system prompt that a language model can understand.

[0159] The above prompt assigns the role of a reinforcement learning specialist robotics engineer to the language model and specifies the objective to design an optimal observation and behavior control space that can efficiently learn a given task.

[0160] In addition, the prompt includes the index map and data type information of the parsed sensors and actuators, which can guide the language model to write code using accurate variable names and shapes without hallucination.

[0161] In addition, the processor can input a configured prompt into a pre-trained artificial intelligence model and generate two core interface codes in response.

[0162] Specifically, the observation model code is a language model that writes logic to select only the features essential for task performance from hundreds of raw data points. For example, it can generate code that improves learning efficiency by including geometric operations, such as converting the robot's global coordinates to the robot's local coordinates or converting a quaternion into a rotation matrix.

[0163] Furthermore, the behavioral model code can be written from the logic in which the language model defines a low-dimensional latent action space to be output by the policy network and expands it into actual robot actuator commands. In particular, during this process, the language model can generate synergy-based control mapping code that considers physical stability to ensure harmonized movements of the upper and lower body.

[0164] In this case, instead of generating only a single code, the processor can generate K candidate code sets with different logic and structures.

[0165] Each candidate set is initialized by mapping it one-to-one with an independent reinforcement learning policy. This is to experimentally verify the optimal design through the competition process described later.

[0166] The above processor can perform a racing algorithm to select the optimal model by performing parallel learning on K generated candidate models within a limited computational resource (Budget).

[0167] At this time, the processor can collect return data by performing a short initial training corresponding to a portion (e.g., 1 to 5%) of the total training steps for each candidate model through an initial evaluation.

[0168] Furthermore, the processor can estimate and select confidence intervals. Specifically, based on the collected data, it calculates the mean and uncertainty (variance) of the expected final performance of each candidate model using statistical techniques such as Bayesian Ridge Regression. Following the principle of Optimism in the Face of Uncertainty, the processor can select the candidate model with the highest Upper Confidence Bound (UCB) and allocate additional training resources to it. Through this iterative evaluation and selection process, the processor confirms a single interface code with the highest potential performance as the final output.

[0169] Next, the processor can perform code verification and register artifacts. Specifically, the finalized code is verified for syntax checking, shape validation, and whether it calls unauthorized external libraries, etc., to ensure execution safety within the platform. The verified code is stored as an artifact along with version information and can be passed to the subsequent step of configuring the experimental environment (S300).

[0170] Next, the processor of the integrated platform system can configure the experimental environment and load code.

[0171] In an embodiment, the processor of the server computing system or / and the processor of the training computing system can load the interface code into the Runtime Environment of the integrated platform to build a Re-modeled Environment for the reinforcement learning agent to learn.

[0172] Specifically, the processor can generate a wrapper and inject code. Specifically, the processor objectifies the raw simulation environment selected in step S100 and generates an environment wrapper class that wraps it.

[0173] The observation model code and behavior model code generated in step S200 are injected into the above wrapper class as member functions or modules.

[0174] Through this, the input / output interfaces of the source environment are abstracted according to the logic designed by the language model. Specifically, the injected functions are configured to automatically execute and perform data conversion whenever the environment's step() method is called.

[0175] And the above processor can redefine the observation space and action space where the reinforcement learning agent interacts based on the injected interface code.

[0176] Specifically, the above processor can redefine the action control space. Specifically, the actions output by the agent's policy network are a high-dimensional space that directly controls all actuators of the robot. A low-dimensional latent space defined by a behavioral model function, rather than this ( It is set to ). When running the simulation, the compute_action function receives the agent's potential action as input, expands it into a physical actuator command (torque, position control signal, etc.), and passes it to the source environment.

[0177] Vast sensor data output from the raw environment ( ) is not passed directly to the agent, but passes through the compute_obs function to become a feature vector essential to the task( It is compressed and converted into ) and transmitted.

[0178] The above processor can transmit and distribute the specifications and interface code of the remodeling environment configured as above to the training computing system (computation node).

[0179] This modeling environment applies equally to all learning phases (offline and online) of the S400 stage described later, as well as to the evaluation simulations and actual robot deployment of the S500 stage.

[0180] This prevents the problem of input data distribution and control signal specifications differing between data-driven learning and interaction-driven learning, and ensures the consistency of the entire pipeline.

[0181] Next, the processor of the training computing system can perform multi-stage reinforcement learning. Specifically, this is a process of progressively enhancing the performance of the agent by sequentially performing data-driven learning and interaction-driven learning on the remodeling environment established in the S300 stage. This stage can be performed specifically through the following detailed processes.

[0182] Specifically, the processor loads an offline dataset containing a large amount of pre-collected process data (or expert demo data).

[0183] The raw state data included in the above dataset passes through an observation model function injected into an environment wrapper at step S300 before being input into the agent. Through this, the raw data is transformed into low-dimensional feature vectors optimized for the task, thereby increasing learning efficiency.

[0184] Then, the processor runs a Phase 1 algorithm (e.g., TD3-BC, CQL, etc.) using the transformed data. Through this, the agent's policy network performs pre-training to acquire initial intelligence without interacting with the simulator.

[0185] Before Phase 1 learning is completed and the system transitions to Phase 2 (e.g., online learning), the system verifies whether the Phase 2 algorithm selected by the user (e.g., SAC, PPO, etc.) is structurally compatible with the Phase 1 model.

[0186] The compatibility checker within the system compares the neural network layer structures of the source model trained in Phase 1 and the target model to be used in Phase 2.

[0187] And the processor automatically determines whether the input dimensions (observation space size) and output dimensions (behavior control space size) of the two models match, and whether there exists a feature extractor layer that can share weights.

[0188] Once compatibility is confirmed, the processor can transfer the knowledge (weights) learned in the first phase to the second phase model.

[0189] Specifically, to address variable naming conventions that differ across reinforcement learning libraries or algorithm implementations, the processor refers to an Algorithm Prefix Map. This converts source model parameter names (e.g., actor.net) to match the target model's conventions (e.g., policy.network).

[0190] In addition, weight values ​​from the source model are injected into the corresponding layers of the target model based on the mapped information. Through this, Phase 2 learning begins with pre-trained intelligence rather than in a random initialization state, which can shorten the learning convergence speed.

[0191] And the above processor allows an agent having the transferred parameters as initial values ​​to interact with the simulator in real time within the remodeling environment and perform second-phase learning.

[0192] Specifically, when the agent outputs a low-dimensional latent action according to the policy, the action model function within the environment wrapper immediately converts it into the robot's physical actuator commands (current, torque, etc.) and transmits them to the simulator.

[0193] The agent then fine-tunes its policy based on feedback (rewards) received from the simulator. This process enhances the agent's adaptability to new situations (out-of-distribution) that were not present in the static dataset, and consequently, provides the effect of reducing the Sim-to-Real Gap.

[0194] Next, the processor of the server computing system can perform learning monitoring and interactive evaluation. (S500) Specifically, this is a process of tracking the progress of the multi-stage learning performed in step S400 in real time, qualitatively verifying the model performance at a specific point in time, and determining whether to transition to the next phase. This step can be performed specifically through the following detailed processes.

[0195] Specifically, the processor visualizes learning log data received from the training computing system on a web interface. At this time, since offline and online learning differ in the nature of the data and the reference of the X-axis, the processor identifies the learning paradigm of the currently executing phase and dynamically adjusts the visualization method.

[0196] Here, since the offline phase is iterative training on the dataset, the graph is rendered by setting the X-axis to Gradient Steps or Epoch.

[0197] In addition, since the online phase is primarily about interaction with the environment, the graph is rendered by setting the X-axis based on the interaction steps.

[0198] In one embodiment, the processor displays the offline phase graph and the online phase graph in different tabs or split views, thereby preventing the user from analyzing disparate indicators with confusion.

[0199] In addition, the processor can visualize the training lineage to understand the overall flow of multi-stage learning.

[0200] This is represented as a hierarchical structure leading to a pre-trained model (parent node), parameter transfer (edge), and fine-tuning model (child node).

[0201] Through this, users can intuitively track which algorithms and datasets a specific model went through to reach its current performance.

[0202] And the processor can perform a checkpoint-based interactive evaluation. When input is received in which a user clicks a specific point (Checkpoint) on the learning curve graph, the processor immediately initiates the interactive evaluation process.

[0203] Here, the processor loads the model weights at the selected time point and simultaneously reloads the interface code (observation and behavior model functions) generated in step S200 to configure the remodeling environment for evaluation. This is to align the input and output specifications of the training and evaluation phases.

[0204] In addition, it can generate simulations and videos. It performs agent simulations in the configured environment, renders the process, and converts it into a video file.

[0205] In addition, the generated video is immediately played as a pop-up on the web interface. This allows users to visually verify specific behavioral patterns of the agent (e.g., gait stability, gripping accuracy, etc.) that are difficult to ascertain from simple numerical values ​​(Return) alone.

[0206] Finally, the processor can determine and deploy a phase transition. Based on the evaluation results, the user decides whether to terminate the current phase and transition to the next phase (e.g., from the simulation phase to the real-world phase) or to modify the hyperparameters and retrain. The finally verified model and interface code are deployed to an actual robot or digital twin system and introduced into the autonomous manufacturing process.

[0207] FIG. 5 is a flowchart of an industrial automation environment simulation method according to one embodiment of the present disclosure.

[0208] More specifically, the detailed process of the industrial automation environment simulation method of S100 is explained with reference to Fig. 5.

[0209] Step S110 may be a step in which the processor of the user computing device provides a visual editor to rapidly prototype the layout of a multi-agent industrial environment in which reinforcement learning agents will interact. The processor of the server computing device can control a series of processes that create a logical workspace on the user interface, spatially map multiple environment component objects within the workspace, and parameterize and store the operational attribute values ​​of each environment component. Through this, the processor can support the user in defining the initial state and boundary conditions of the runtime environment using only intuitive user input events, without the need for complex physics engine-based hardcoding or high-level technical expertise. Furthermore, the system can ensure the reproducibility of environment construction by serializing the defined spatial and attribute data into machine-readable, standardized structured text and storing it as an asset in a database.

[0210] Specifically, the processor of the server computing device can provide a visual editor screen in the form of an independent view through the browser of the user computing device. The user can set the size of the workspace in a grid format on a 2D plane or in 3D space, reflecting the scale of the physical factory or logistics warehouse to be built or tested. This grid can form an absolute and relative coordinate system where components are placed. Once the workspace is created, the processor can provide core infrastructure components for simulating the industrial site in the form of visual rendering objects via menus or palettes. The component objects may include autonomous robots, inventory shelves, robot charging stations, conveyor belts, and work stations. The user can select objects from the menu and then place or remove them at locations on the set grid by performing click or drag-and-drop operations.

[0211] Furthermore, the processor can provide an operational parameter setting interface to control not only the physical layout but also the logical and dynamic characteristics occurring within the environment. Users can set detailed environmental variables, such as the size of the order queue, operating speed, and material input interval, by manipulating the provided numeric input window. This allows users to immediately reflect hypothetical scenarios into the environment. The processor can aggregate the coordinate mapping information of completed components and the set operational parameter values ​​to convert them into a standardized text format. In one embodiment, the data is generated as a format file and saved to the system in an export format, or the layout can be restored on the screen by loading the saved file. This file-based asset management method can guarantee the reproducibility of experiments when evaluating the performance of multi-agent reinforcement learning algorithms. At the same time, it can provide technical benefits to industrial field practitioners by facilitating version management of facility design assets and supporting the sharing of environmental settings among workers.

[0212] In step S110, the structure of the user interface (UI) of the visual editor (Environment Builder) provided by the processor through the user terminal is illustrated as an example.

[0213] The above visual editor interface can be functionally divided into a control panel (800) on the left that performs various settings and controls, and an interactive grid workspace (850) on the right where environment components are actually visualized and mapped.

[0214] The layout setting area (810) located at the top of the control panel (800) provides a slider and numeric input interface that allows the user to set the size of a workspace (850) in the form of a 2D or 3D grid (e.g., the number of rows and columns) reflecting the scale of a physical factory or logistics warehouse.

[0215] The component selection area (820) located in the middle of the control panel (800) provides core infrastructure objects constituting an industrial site, such as autonomous robots, inventory shelves, robot charging stations, conveyor belts, and work stations, in the form of a drop-down menu. Without a separate coding process, the user can select a desired object from the component selection area (820) and intuitively place the object at a specific coordinate on the grid workspace (850) on the right by clicking or dragging and dropping.

[0216] The operating parameter setting area (830) located at the bottom of the control panel (800) provides an interface for manipulating detailed parameters to control dynamic characteristics occurring within the environment, such as the driving speed of the conveyor belt and the feed interval at which materials are fed.

[0217] The grid workspace (850) on the right is dynamically rendered in response to the user's input events and provides visual feedback of the initial state of the runtime environment.

[0218] Furthermore, the processor aggregates all spatial layout information and operational parameter values ​​set through the control panel (800) and the grid workspace (850) and serializes them into a machine-readable, hierarchical, standardized text format (e.g., a YAML file). Through this, the system supports the export or import of complex layout assets designed by the user, thereby ensuring perfect reproducibility of the environment construction.

[0219] Step S120 may be a step in which a processor of a server computing device assigns a logical control objective function and rules for interaction with the environment that a multi-agent system must achieve on the designed physical layout infrastructure. The processor of the server computing device may receive user input selecting one of pre-built operational logic templates or receive a task description sequence entered in the form of natural language text. The processor may parse this input data to set quantitative indicators and state transition constraints to evaluate success when the agent swarm performs the task. This serves as a reference point for designing a reward function to optimize agent behavior during the subsequent reinforcement learning process and may provide a technical means to instantiate the dynamic characteristics of a complex industrial site as a mathematical goal.

[0220] Once the layout design is completed via a visual editor, the processor can provide standard scenario templates simulating the process on the interface. Users can immediately map complex objective functions to the environment through input. For the order picking task, a robot swarm transports stock shelves from the storage area to a station, a worker retrieves the ordered items, and the robot returns the shelves; the optimization goal can be set to minimize the total order completion time. The inventory loading task involves robots taking empty or available shelves to a worker to replenish stock and returning them to the storage area; the optimization goal can be assigned to minimize the loading completion time. The automated production facility task involves robots transporting materials between points between workbenches and conveyor belts; the optimization goal can be set to maximize total throughput.

[0221] Furthermore, the processor can provide complex scenarios involving the combination of conflicting goals. An integrated logistics center task could be a scenario where order picking and inventory loading tasks are performed simultaneously within a single layout. When the two tasks compete for limited robot resources and movement paths, the processor can assign a multi-goal reward function to the agent that requires simultaneous optimization of the goals of minimizing order completion time and loading completion time. This can provide a benchmark environment for evaluating multi-agent coordination algorithms. Additionally, the processor can receive task descriptions in natural language form in a text input field. The processor collects and temporarily stores this natural language input, and the text data can be utilized as constraints and design guidelines injected into the AI ​​model to contribute to the generation of customized interface code.

[0222] Step S130 may be a step of specifying a behavioral model for controlling multiple agents within the constructed physical environment layout and dynamically binding it to an interface specification to enable communication with simulation environment objects. The processor of the server computing device may adopt a modular architecture that separates the control policies of the simulation environment and the agents. The processor can abstract the agent's observation transmission and behavior input processes according to standard specifications. When a user selects a control algorithm or a pre-trained weight model, the system can complete the interaction loop by integrating the model in a plug-and-play manner without additional code-level modifications. This can provide a technical foundation that enables cross-verification of the performance of heterogeneous control logics within the platform under identical conditions.

[0223] The processor can configure wrappers in compliance with standard interface specifications of reinforcement learning environments. This enables the establishment of multiple communication channels through which a robot swarm can exchange state, reward, and termination signals. Models can be immediately integrated as long as the input / output format is satisfied. The processor can receive at least one of multiple heterogeneous behavior models via the user interface and inject it into the environment. It can receive script-based rule-based heuristic algorithms and insert them into action decision modules, which can be utilized when prototyping the effects of conditional control. It can apply neural network-based multi-agent reinforcement learning policies and load algorithms specialized for inter-agent cooperation by integrating with external frameworks. The processor can initiate training from random weight states or perform evaluations by injecting checkpoint weights. Additionally, it can utilize a language model-based autonomous planner. The processor can convert the physical state of the environment into natural language and send it as a prompt to the language model. When the language model generates and returns a high-level plan, the parser can convert it into low-level action commands and apply them to actuators.

[0224] In one embodiment, the processor can mix-assign multiple agents within a single environment to different behavioral model paradigms. By having some robot swarms operate with heuristic algorithms and the remaining robot swarms operate with reinforcement learning policies or language model planners, the interaction and collision avoidance performance between heterogeneous agents can be verified.

[0225] Figure 7 is a block diagram illustrating the standard assetization and standard interface mapping process of designed environment data.

[0226] Referring to FIG. 7, the processor of the system receives spatial layout input data (860) configured by the user through the preceding visual editor (see FIG. 6). The processor collects the coordinate information and operational parameter values ​​of the components included in the layout input data (860) and serializes and stores them as standardized structural text asset data (862) having a hierarchical structure.

[0227] In one embodiment of the present disclosure, the structured text asset data (862) may be instantiated into a file in YAML (YAML Ain't Markup Language) format. This serialization process enables the storage, sharing, and reproduction of complex simulation environment settings in a lightweight, file-based manner, thereby achieving efficiency in environment construction.

[0228] Next, the processor performs a process of mapping a behavior model (see step S130) for actually controlling an agent (robot swarm) in the environment to the runtime environment.

[0229] Specifically, the processor receives, along with the serialized structured text asset data (862), predefined hardware specification data (864) that defines the physical / kinematic constraints of the agent. In one embodiment, the hardware specification data (864) may be a file in the format of XML (Extensible Markup Language) or URDF (Unified Robot Description Format).

[0230] The API mapping and normalization process module (866) within the system parses and normalizes the input asset data (862) and hardware specification data (864) to define the agent's observation space and action space.

[0231] Finally, the processor creates (wraps) a standard API-based runtime environment interface (868) that follows the international standard API specification for multi-agent interaction (e.g., PettingZoo API) based on the normalized data.

[0232] Through the flow illustrated in this block diagram, the present disclosure provides technical means to map input data (860) for a visually designed industrial environment to a machine learning-enabled standardized API environment interface (868) in a dynamic and automated manner without code-level hard coding.

[0233] FIG. 8 is a schematic diagram showing a simulation dashboard main interface and a multi-window structure user interface for selecting a behavior model according to one embodiment of the present disclosure.

[0234] Referring to FIG. 8, a specific interface structure provided by the processor in step S130 to dynamically map heterogeneous control logic to a runtime environment is illustrated. The user interface according to the present disclosure has a multi-window structure in which a main interface (870) responsible for the execution and monitoring of the simulation and a builder popup interface (875) defining the environment and control logic are overlaid.

[0235] First, the builder popup interface (875) is a space where the user sets the details of the simulation environment. The top area (876) within the popup provides a list of templates for predefined standard industry scenarios (e.g., order picking, inventory loading, etc.) described in step S120, and the central area (877) provides a preview of a static grid schematic of the selected layout.

[0236] As a key feature of the present disclosure, a parameter setting section (878) located at the bottom of the builder popup interface (875) includes a 'behavior model selection area (879)' for determining the agent's control policy. As illustrated in FIG. 22, the behavior model selection area (879) is implemented as an expandable dropdown menu UI element. The processor provides the user with heterogeneous behavior model options through the dropdown menu (879), including Python script-based heuristic algorithms, multi-agent reinforcement learning policies (RL) such as MAPPO and QMIX, and autonomous planners (LLM) linked with a massive language model API.

[0237] When a user selects a specific control logic (e.g., RL or LLM) through the behavior model selection area (879), the processor immediately maps (dynamically binds) the model internally within the system to the specifications of a multi-agent standard interface (e.g., PettingZoo API) without additional code-level modification (hard-coding).

[0238] After the selection and mapping of the behavior model is completed, the user can switch to simulation mode by operating the mode selection interface (874) at the bottom right of the main interface (870).

[0239] In simulation mode, the processor activates the integrated control button group (873) of the main interface (870). When the user clicks the Run button within the button group (873), the selected behavior model (879) is applied to the runtime environment, and the agent (robot swarm) begins to interact.

[0240] The results of these interactions are visualized in real-time as 2D or 3D animations representing physical behaviors such as robot movement, shelf transport, and reaching the charging station in the main grid view area (871) of the main interface (870). At the same time, the processor renders quantitative metrics representing the performance of multiple agents, such as throughput, robot status (Idle / Moving, etc.), and episode rewards, in the form of multiple graphs in synchronization through the live chart view panel (872) located at the bottom left of the main interface (870).

[0241] As a result, the integrated interface structure illustrated in FIG. 8 provides a powerful platform environment that allows users to freely replace heterogeneous control logic (Plug-and-play) with only drop-down menu (879) operations without complex coding, and to immediately and intuitively verify the resulting dynamic changes and performance indicators of the system.

[0242] Step S140 may be a data preprocessing step that replaces and normalizes the constructed environment and assigned task information into machine-readable text so that a language model can interpret it. The processor of the server computing device may parse a low-level hardware specification file that defines kinematic constraints. The processor may extract the state variable space and the actuator space, and then generate metadata including the tensor dimension, data type, index, and semantics of each variable. The normalized metadata acts as a constraint to be injected into the artificial intelligence model, which can block hallucinations that cause arbitrary referencing of variables or confusion of indices. Specifically, the processor may load and parse the hardware specification file of the simulator backend. The processor may trace a hierarchical tree of multiple links, joints, sensors, and actuators and identify their relationships.

[0243] The processor can extract and classify the list of data that the simulator can provide. State variables may include joint state information, posture information, and sensor output values. The actuator map may be a table-like organization of actuator identifiers, tolerances, gear ratios, and mapping information. The processor can normalize the extracted variable information into a structured format that a language model can recognize. The processor can combine data types, dimensions, index information, and semantics. Through this, a refined profile can be generated. After normalization is complete, the processor can combine task descriptions and available variable information to complete the basic data package for the prompt. This package can be transferred to a subsequent stage and used as a guideline when the artificial intelligence model writes code.

[0244] FIG. 9 is a flowchart of a method for automatically generating a robot control interface according to one embodiment of the present disclosure.

[0245] Referring to FIG. 9, the interface automatic generation and optimization step of S200 is a process of converting a user's abstract task command into a concrete interface code capable of physical control and determining the optimal design through competition among multiple candidates. This process follows a mechanical flow of code generation, selection, verification, and registration, and can be performed specifically in the following steps.

[0246] Specifically, the processor of a server, user computing device, or training computing system can parse hardware specifications and configure available variable information. (S210)

[0247] In an embodiment, the processor of the training computing system may parse hardware specifications and available variable information of the robot and environment from at least one of a simulation environment, a digital twin environment, or a real environment, and normalize this into metadata that can be interpreted by a large-scale language model. This step may be performed to suppress hallucination errors in the code generation process by configuring available variable information and actuator maps, including variable lists, physical semantics, tensor dimensions, and indices, so that the large-scale language model can accurately reference the state, sensors, and driving information of the robot and environment.

[0248] According to one embodiment, hardware specification parsing and available variable information configuration may be performed by including the following detailed data acquisition, processing, and output processes.

[0249] The processor can load a hardware specification file (e.g., XML, URDF, or MJCF format, etc.) that defines the robot and task environment selected by the user, or access state objects provided by the simulator backend. Through this, the processor can obtain access rights to sensors, state variables, and control interfaces available in the environment to prepare for data extraction.

[0250] The above processor can parse an acquired hardware specification file to identify links, joints, sensors, and actuators constituting the robot, and determine the hierarchical and kinematic connection relationships between them.

[0251] In addition, the processor may extract a list of all raw state variables accessible in a simulator or real environment and classify them in detail according to their physical properties and meanings from a control perspective. The classified state variables may include at least one of joint state information, kinematic information, dynamics, and sensor information.

[0252] Here, the joint state information is generalized position information representing the angle or position of each robot joint, and generalized velocity information representing the rotation or movement speed.

[0253] In addition, kinematic information is position in three-dimensional space, such as base links or end devices, and attitude information expressed as quaternions or rotation matrices.

[0254] In addition, dynamic and sensor information may include forces or torques applied to each joint, external forces, moments of inertia, and output values ​​of contact sensors or inertial measurement devices attached to the soles of the feet, etc.

[0255] Furthermore, the processor can configure an actuator map that serves as a basis for generating a behavioral model in a subsequent step. Specifically, the processor can systematize the identifier of each actuator, the allowable range of controllable physical quantities (e.g., torque and position limit values), the gear ratio, and index mapping information of the joints controlled by the actuator in the form of a map.

[0256] Next, the processor can normalize the extracted and classified variable and driver information into a structured text format (e.g., key-value structure, JSON schema, or Markdown list, etc.) that a large-scale language model can clearly understand. The normalized text data can be structured in the form of comments that go beyond a simple list of variable names and combine the name of each variable, tensor dimension, data type, detailed description of physical meaning, and index mapping information within an array.

[0257] As a result of performing the above S210 step, the processor can finally output available variable information to be inserted into the available variable information item of the system prompt and an actuator map to be referenced when designing a behavior model.

[0258] By accurately providing large-scale language models with the robot's physical constraints and interface structure, such structured metadata can fundamentally suppress illusions where the language model references non-existent variables or maps incorrect indices, thereby inducing the generation of safe code that does not cause runtime errors.

[0259] The processor can generate a system prompt for a robot control interface based on the task description and available variable information. (S220)

[0260] In an embodiment, the processor can combine a task description obtained from a user with available variable information configured in step S210 to generate a structured system prompt that a large-scale language model can interpret and execute.

[0261] Step S220 according to one embodiment may be performed including the following detailed prompt configuration and data injection process.

[0262] The processor can generate the prompt to be generated as a modular structure combining a common area and individual areas. As illustrated in FIG. 10, the processor can generate a structured prompt group (500), which may include a system prompt (510) corresponding to the common area, and observation model design prompts (520) and behavior model design prompts (530) corresponding to individual areas. Additionally, referring to FIG. 11, the entire LLM prompt (550) may be hierarchically divided into sub-modules: a system prompt (560), an observation design (570), and a behavior design (580) area.

[0263] Specifically, the processor may insert text defining the role of a large-scale language model into the introduction of the prompt (e.g., the role and goal block within the system prompt (560) of FIG. 11). For example, the processor may assign the role of a robotics engineer specializing in deep reinforcement learning to the large-scale language model. Additionally, the processor may induce the language model to output an optimized result by setting an explicit goal to design a physically valid interface function with high sample efficiency to perform a given task.

[0264] In addition, the processor can construct background knowledge of the language model by injecting the acquired task description and the normalized available variable information into the body of the prompt.

[0265] Specifically, the processor can insert the task goal, success conditions, and constraints (e.g., running while maintaining a specific target speed without falling) into a prompt in the form of natural language text so that the language model can recognize the physical purpose of the task.

[0266] In addition, the processor can inject available variable information. Specifically, the processor can insert the exact name of the variable accessible in the simulator, the tensor dimension, a description of its physical meaning, and index mapping information according to the actuator map in the form of a list. This allows the language model to fundamentally block hallucinations such as arbitrarily referencing non-existent variables or confusing indices.

[0267] As illustrated in FIG. 10, the processor can add design guidelines and constraints to the system prompt.

[0268] The processor may include specific instructions, reasoning and planning steps, output formats, style guides, and additional constraints within the observation design (570) and behavior design (580) blocks. In particular, physical validity in terms of full-body control may be enforced by specifying in the prompt that maintaining a stable and balanced posture of the robot in all physical tasks is an implicit requirement.

[0269] As a result of performing the above S220 step, the structured prompt group (500) can be input into a large-scale language model and converted into a generated interface code group (500a) shown in FIG. 10.

[0270] The structured prompt group (500) may include physical stability guidelines. Specifically, the processor may specify in the prompt that maintaining a stable and balanced posture of the robot is an implicit requirement in all physical tasks. This compels the language model to write code that takes into account physical validity in terms of whole-body control as well as single-joint movement.

[0271] Additionally, the structured prompt group (500), implementation, and output constraints may be included. The processor may specify that the generated result must be in the form of an executable programming function (e.g., a Python function). Specifically, the processor may instruct to clearly define the required function signature, the tensor dimensions of the input arguments and return values, whether batch processing for parallel learning is supported, and the operational device consistency of the newly generated tensor.

[0272] As a result of performing the above S220 step, the processor can finally output an advanced and structured system prompt for generating observation and behavior models of reinforcement learning. The modular prompt configured in this way can be input into a large-scale language model in a subsequent step and utilized as a core foundation for generating high-quality executable interface code that connects the physics engine and the reinforcement learning algorithm.

[0273] Below, a specific example of the above-mentioned structured system prompt is described. The example below illustrates a part of the observation model design prompt for a humanoid robot's locomotion task.

[0274] Prompt Components (Section) Details and Parameters (Content) Role A robotics engineer specializing in deep reinforcement learning Objective Design the optimal observation model function (compute_obs) to maximize sample efficiency based on robot hardware specifications and tasks Task Description Control the humanoid robot (H1) to run stably forward at a target speed (5 m / s) without falling over Available State Variables 1. data.qpos (Shape: batch, 76): [0:3] Position, [3:7] Quaternion, [7:19] Left Leg, [19:31] Right Leg, [31:43] Left Arm, [43:55] Right Arm, [55:76] Finger Angle 2. data.qvel (Shape: batch, 75): [0:3] Linear Velocity, [3:6] Angular Velocity, [6:18] Left Leg Joint Velocity, etc. 3. data.sensordata (Shape: batch, 8): [0:4] Left foot contact force, [4:8] Right foot contact force Actuator Map (Index / Name / Joint / Range / Gear Ratio) 0 / left_hip_yaw / left_hip_yaw / [-1.57, 1.57] / 100... (omitted) ...20 / left_thumb / left_thumb / [0.0, 1.57] / 20 Design Guidelines 1. Dimensionality Reduction: Exclude unnecessary joint (fingers, etc.) state information 2. Heading Invariance: Convert global coordinate system data to robot base local coordinate system 3. Explicit Features: Include error between target velocity and current velocity 4. Attitude Stability: Mandatory inclusion of contact sensor and gravity orientation attitude data Output Format Function Signature: def compute_obs(data, command) -> torch.Tensor Input / Output: Input data, command (batch, 3) / Output obs (batch, D_obs) Constraints: Preserve batch dimension, use only vectorized tensor operations.

[0275] Referring to Table 1, the system prompt can be configured by combining the role definition of the language model, the physical purpose of the task, the names, dimensions, index mappings, and physical meanings of all accessible state variables, the actuator map, design guidelines, and output format constraints in a modular structure. In particular, by providing available variable information as structured text along with tensor dimensions and index mappings as described above, the language model can suppress hallucinatory errors such as referencing non-existent variables or using incorrect indices. The behavioral model design prompt shares the same common domains (Role, Task Description, Available State Variables, Actuator Map) as the observational model design prompt, but the Objective and Design Guidelines can be configured by replacing them with content specialized for the design of the behavioral control space (synergy mapping, inactive joint fixation, output scaling, etc.). Next, the processor can generate observational model code. (S230) See details. The processor can generate an observation model code (510a) that converts raw observations into observation vectors suitable for reinforcement learning based on the system prompt generated in step S220.

[0276] In this step, the processor uses a large-scale language model to the raw observation space ( ) work-related feature space( A mathematical mapping function that reduces to ). You can generate code that defines .

[0277] [Mathematical Formula 1]

[0278]

[0279] Here, refers to a high-dimensional raw sensory input space (Raw observation space) collected from simulators or sensors, and can refer to a dimensionally reduced and refined observation space (Modeled observation space) to be provided as input to a policy neural network.

[0280] Step S230 according to one embodiment may be performed by including the following detailed engineering operation logic to implement the mapping of Equation 1.

[0281] The above processor can select task-oriented features based on task descriptions and available variable information within the prompt.

[0282] Specifically, the processor, based on the task description and available variable information within the prompt, high-dimensional A compact representation by selecting only the variables significant to task performance and physical stability among the variables It can be derived.

[0283] In other words, the processor can reduce the dimensionality of the observation space by analyzing the semantic association between the task description included in the prompt and available variable information through a large-scale language model, and by selecting only the variables significant to task performance and stability from among hundreds of raw state variables. For example, in the case of a running task, the processor can write code that selects the joint state information of the legs and pelvis, while excluding from the observation vector the state information of finger joints that do not actively participate in the task movement. Through this, the processor can prevent unnecessary dimensionality expansion and significantly increase the learning convergence speed of the agent.

[0284] Next, the processor can perform coordinate system transformation and normalization. Specifically, the processor can generate geometric operation code that transforms the coordinate system so that the agent's policy does not overfit to global position information and has generalized performance for the environment. Specifically, the processor can transform position and velocity data based on the global coordinate system provided by the simulator into a local coordinate system with the robot's base as the origin. To this end, logic that calculates a rotation matrix using a quaternion representing the direction of the base and performs vector rotation operations can be implemented within the function. Additionally, to enable the robot to learn the same behavior pattern regardless of the direction it is facing, operations can be included to ensure heading invariance by aligning the gravity vector or the direction vector of the target point with respect to the robot's direction of travel.

[0285] Additionally, the processor can calculate target information and relative vectors. Specifically, the processor may include logic for calculating a relative vector or error value representing the difference between the target state and the current state so that the goal of a given task can be clearly conveyed to the agent. In the case of a task involving movement to a specific point, the processor may configure the code to calculate the relative distance and direction vector from the current robot position to the target point and return them as observation values, instead of inputting the absolute coordinates of the target point directly. Furthermore, by calculating the error value between the target speed and the current speed as an explicit feature indicator and adding it to the observation vector, the agent may be encouraged to grasp the structure of the reward function more quickly.

[0286] In addition, the processor can integrate physical stability indicators. Specifically, the processor can generate code that integrates indicators representing dynamic stability into observation vectors to support stable control in a physical environment. Specifically, the processor can write code that extracts contact sensor data attached to the robot's feet, or attitude data regarding the height and velocity of the center of mass and the direction of gravity. This enables the agent to perceive the state of contact with the ground and the state of balance of the entire body, thereby contributing to learning a policy that is physically stable and prevents falling from the beginning of learning.

[0287] Furthermore, the processor can optimize parallel processing and multidimensional array operations. Specifically, the processor can implement observation model functions as vectorized multidimensional array operations to efficiently process inputs of batch processing units (batches) occurring from multiple simulation instances simultaneously. By outputting code that can process large amounts of data in parallel using only matrix operations without sequential loops through a large-scale language model, the processor can eliminate computational bottlenecks in large-scale reinforcement learning environments and maximize processing efficiency.

[0288] Below, a specific example of the above observation model code is explained in the form of pseudocode. [Table 2] below illustrates the structure of an observation model function that a large-scale language model can generate for a humanoid robot's movement task.

[0289] Operation Step Processing Purpose and Function Detailed Operation Content (Pseudocode-based) 0. Interface Definition Set Input / Output Tensor Specifications - Input: latent_action (Shape: batch, 14), data - Output: raw_action (Shape: batch, 21) 1. Core Joint Direct Mapping 1:1 Control of Leg Joints - Directly assign the first 12 dimensions ([0:12]) of the latent action to commands for the 6 joints (hip, knee, ankle, etc.) of the left and right legs. 2. Synergy-Based Action Expansion (Contralateral Coordination) Implementation of Natural Arm Swing (Ensuring Gait Stability) - Use the 13th latent action ([12:13]) as the arm swing synergy signal - Contralateral coordination operation: Cross-mapping the left shoulder to right hip flexion and the right shoulder to left hip flexion by multiplying them by a negative weight (-0.5). 3. Synergy-based Action Expansion (Grasp Synchronization) Simplified gripper control - Uses the 14th potential action ([13:14]) as the Grasp signal - Synchronizes multiple finger flexion joints equally with a single signal 4. Locking Inactive Joints Control of unnecessary degrees of freedom - Assigns a zeros array to the remaining joints of the arm not involved in the task (elbow, wrist, etc.) to lock them in a neutral position (0) 5. Command Concatenation Final dimension expansion and combination - Concatenates the command vectors for the leg (12), left arm (4), right arm (4), and Grasp (1) to create a full 21-dimensional actuator command array 6. Output Scaling (Hardware Protection) Applying actuator physical limits - Clips command values ​​based on the lower (actuator_low) and upper (actuator_high) limits of the driving range for each joint defined in the hardware specification (Actuator Map)

[0290] As can be seen in the example above, the behavior model function is a latent behavior control space ( The low-dimensional output of ) ) to the raw actuator command space ( A mathematical mapping that extends to ) is implemented. In particular, in step (2), a single arm swing synergy signal is mapped to the left and right shoulder joints in opposite phase according to the principle of contralateral coordination, and in step 3, a single gripping signal is transmitted to multiple finger flexion joints in synchronization. Through this synergy mapping, the agent only needs to explore a 14-dimensional compressed latent space instead of directly exploring a 21-dimensional raw action control space, so the exploration efficiency can be significantly improved.

[0291] Next, the processor can generate behavioral model code by analyzing system prompts through a large-scale language model. (S240)

[0292] Specifically, the step may be for the processor to analyze a system prompt using a large-scale language model and generate a behavior model code (520a) that converts a low-dimensional potential behavior output by a reinforcement learning policy into a high-dimensional actuator command of an actual robot. The code generated in this process is a mathematical mapping function that expands the optimized low-dimensional control signal of the policy network into a total actuator control signal. It can implement.

[0293] [Mathematical Formula 2]

[0294]

[0295] Here, It refers to a low-dimensional latent action space that minimizes the reinforcement learning search space by including only the joint groups essential for task performance, and can refer to a high-dimensional raw action space that is directly applied to robot hardware.

[0296] Step S240 according to one embodiment may be performed by including the following detailed control logic and processing steps, going beyond simply extending the mapping of Equation 2 one-to-one.

[0297] First, the processor can define and dimensionally reduce the latent action control space using a large-scale language model. Specifically, the processor can use the large-scale language model to identify key joints that play a leading role in task execution among the robot's total degrees of freedom, and generate code that defines the optimal latent action dimension corresponding to them. For example, when a humanoid robot with multiple actuators performs a locomotion task, the processor can define a low-dimensional latent space that includes only the major joints of the legs and arms, instead of setting all actuators as the control space.

[0298] That is, the above processor analyzes the actuator map through a large-scale language model and the optimal potential behavior dimension ( ) can be defined.

[0299] Through this, the processor can significantly reduce the space that the agent needs to explore, thereby accelerating the learning convergence speed.

[0300] Next, the processor can perform synergy-based behavior expansion. Specifically, the processor can implement synergy mapping logic based on biomechanical principles or robotic control theory to control multiple joints associated with a single potential action value simultaneously. To implement contralateral coordination of a bipedal robot, the processor can write code that maps a motion signal of one hip joint to the motion of the opposite shoulder with a negative weight, thereby inducing the agent to perform a natural arm swing motion without a separate learning process. Additionally, for areas that do not require individual and precise control, the processor can generate code that synchronizes a single grasping signal to be transmitted simultaneously to all multiple finger flexion joints.

[0301] The above processor is based on biomechanical principles or robotic control theory, The single action value of Synergy mapping logic can be included to allow simultaneous conversion into multiple joint commands.

[0302] Additionally, the processor may apply fixation and constraints to inactive joints. Specifically, the processor may include logic for applying constraints to joints that are unnecessary or may interfere with task performance. For example, the processor may preemptively prevent unnecessary energy consumption and control noise generation by assigning a neutral position value or zero (0) to joints that do not contribute to movement. Furthermore, the processor may include an output scaling operation that adjusts the output of the policy neural network to the physical limits of each actuator, thereby reducing the possibility of hardware damage caused by control commands exceeding torque or position limit values ​​and ensuring physically stable control.

[0303] In addition, the processor can support multidimensional array operations and batch processing. The processor can implement behavioral model functions as multidimensional array operations considering batch dimensions so that behavioral data generated in parallel in multiple simulation environments can be processed in batches. Furthermore, the processor can prevent runtime errors in advance by including logic to verify the dimensions or shape of the output tensor. This can maximize computational efficiency in large-scale parallel reinforcement learning environments where numerous simulation instances are run simultaneously.

[0304] Next, the processor can select a competition-based candidate from multiple candidate interface code sets. (S250)

[0305] The processor can perform parallel learning or short-term evaluation on a plurality of previously generated candidate interface code sets and select the optimal candidate code through a resource allocation mechanism based on Optimism in the Face of Uncertainty (OFU).

[0306] Specifically, the processor may perform parallel learning or short-term evaluation on a previously generated set of multiple candidate interface codes within a limited computational budget, and select the optimal candidate code based on performance and uncertainty. This step may be optionally performed to overcome the stochastic variability of large-scale language model outputs and to derive the interface design with the highest sample efficiency.

[0307] Step S250 according to one embodiment may be performed including the following detailed competitive evaluation and resource reallocation process.

[0308] The processor can perform initial training and evaluation. Specifically, it can initialize a policy neural network corresponding to each of the generated sets of multiple candidate codes (Model 1, 2, 3) and perform initial training and evaluation in parallel for a predetermined training unit (e.g., a certain number of steps or episodes) in each candidate environment. Through this, the processor can collect data on the performance trajectory during the initial training for each individual candidate code (e.g., the trend of change in reward return values ​​or loss values).

[0309] And the processor can predict performance and estimate uncertainty. Specifically, the processor can statistically estimate the expected final performance of each candidate code and the resulting uncertainty based on collected initial performance data. In a specific embodiment, the processor can calculate the predicted mean value and prediction uncertainty of each candidate code by extrapolating the reward trajectory of the policy using a statistical regression technique such as Bayesian Ridge Regression. Furthermore, the processor can derive an Upper Confidence Bound for the performance of each candidate by combining the calculated mean value and uncertainty.

[0310] Below, specific formula examples for the above performance prediction and uncertainty estimation process are explained.

[0311] The above processor, in the initial learning phase for the k-th candidate code set, Collect performance data at multiple time points, and a feature vector including a time index and the observation reward at that point in time A dataset consisting of pairs It can be composed of.

[0312] Here, feature vector For example, a polynomial basis It can be composed of.

[0313] The above processor is the collected dataset By applying Bayesian Ridge Regression, the distribution of the prediction performance of candidate k at the target time point Ttarget can be calculated as shown in the following Equation 3.

[0314] [Mathematical Formula 3]

[0315]

[0316]

[0317] Here, is the posterior mean vector, and is the posterior covariance matrix, and represents the average predicted performance of candidate k at the target time point, and represents the uncertainty (standard deviation) of the prediction.

[0318] The above processor is a precision parameter (Precision of weighted prior distribution) and noise precision parameter Set the , and calculate the posterior distribution of the weights from the observed data in a closed-form.

[0319] The above posterior covariance matrix can be calculated as shown in the following mathematical formula 3-1.

[0320] [Mathematical Formula 3-1]

[0321]

[0322] [Mathematical Formula 3-2]

[0323]

[0324] Here, is the identity matrix, and Is It is a design matrix formed by stacking feature vectors of several time points into rows, and is the corresponding observation reward vector. is a precision hyperparameter of the weight prior distribution and serves as a regularization that suppresses overfitting.

[0325] Based on the Optimism in the Face of Uncertainty principle, the above processor can derive the upper confidence interval (UCB) of each candidate code as shown in the following mathematical formula 4.

[0326] [Mathematical Formula 4]

[0327]

[0328] Here, is a positive constant (Confidence parameter) that balances exploration and exploitation.

[0329] The above processor for each racing round The candidate with the largest value Select and ( ), additional learning resources for the corresponding candidate (e.g., After proceeding with training by assigning steps, reflect the updated data and Re-estimate.

[0330] This process can be repeated until the total learning budget is exhausted or a preset competition termination condition is met.

[0331] The definitions of the main symbols used in the above mathematical formulas 3, 3-1, 3-2, and 4 are as shown in Table 3 below.

[0332] Symbol definition k Index of the candidate code set T Number of time points of performance data collected during the initial training phase Feature vector at time t (e.g., polynomial basis [1, t, t², ..., t]) The observation reward (Return) value of candidate k at time t Training dataset for candidate k {(φ(t), r(t))} A design matrix formed by stacking feature vectors from T time points into rows Precision hyperparameter of weighted prior distribution (normalization factor) Noise precision parameter (inverse variance of observed noise) weighted posterior covariance matrix of candidate k weighted posterior mean vector of candidate k The target point in time to predict performance (e.g., completion of full training) Average prediction performance of candidate k at the target time point Prediction uncertainty (standard deviation) of candidate k at the target time point Search-Utilization Balance Control Constant (Confidence parameter) Upper Confidence Bound of candidate k Number of additional learning steps allocated in each racing round

[0333] In this way, the processor quantitatively estimates the predictive performance and uncertainty of each candidate code through Bayesian Ridge regression and applies an optimistic selection principle based on an upper confidence interval (Equation 4), thereby enabling the rapid identification of the most promising interface code within a limited computational budget while maintaining an optimal balance between search and utilization. Compared to the equal distribution method that consumes the same learning resources for all candidates or the conventional method of sequentially performing multiple pilot experiments, this can provide technical effects such as the efficient allocation of computing resources and the early discovery of the optimal code. FIG. 12 is a diagram illustrating the process of selecting the optimal candidate code through an optimistic uncertainty-based resource allocation mechanism according to an embodiment of the present disclosure.

[0334] Specifically, referring to FIG. 12, the processor can calculate the predicted mean value (circle symbol) and prediction uncertainty (vertical line) of each candidate code using Bayesian Ridge regression, etc., and combine them to derive an upper confidence interval (UCB, asterisk symbol).

[0335] Additionally, the processor can perform resource reallocation and iterative racing. Specifically, the processor can dynamically reallocate additional training resources to the most promising candidate based on the principle of optimistic selection under uncertainty. More specifically, the processor can select the candidate model with the highest calculated upper confidence interval and allocate a certain number of additional training resources to that policy. The processor can repeatedly perform a racing process, in which it re-estimates the confidence interval and reallocates resources by reflecting the additional training results, until a predetermined competition end point. This can provide the effect of concentrating computing resources on superior models while maintaining a balance between the process of searching for new promising candidates and the process of utilizing already verified candidates.

[0336] As shown in the left graph of Fig. 12, the processor can select the candidate model (e.g., Model 3) with the highest calculated upper confidence interval (asterisk) as a promising model (Step 1). Subsequently, training can be performed by allocating additional training resources to the policy of the selected model as shown by the central arrow (Step 2). After additional training, as shown in the right graph, the processor can re-estimate the performance and confidence interval (Step 3), and this racing process can be repeated.

[0337] Furthermore, the processor can determine the final winner code among the set of candidate interface codes. Specifically, when the processor reaches a predefined competition end point or the point where the total training budget is exhausted, it can determine the single environment model code with the highest upper confidence interval or the best cumulative performance among the evaluated multiple candidates as the final winner. Subsequently, the processor can control the continuation of training for the remaining training budget by utilizing only the policy of the selected winner code. Through this, the processor improves upon the conventional inefficiency of conducting multiple pilot experiments and can rapidly identify the most robust and high-performing optimal interface code within a single experiment loop.

[0338] Next, the processor can verify the determined interface code and execute it in an isolated environment. (S260)

[0339] Specifically, the processor may perform static and dynamic verification so that the selected optimal candidate interface code (or a single generated interface code) can be safely integrated into the reinforcement learning pipeline. This step may be performed to preemptively block syntactic and logical errors in the code generated by the large-scale language model, as well as potential risk factors that could threaten system security.

[0340] Step S260 according to one embodiment may be performed including the following detailed verification and exception handling processes.

[0341] The processor can perform static verification on candidate interface code. Specifically, the processor can perform static verification at the code syntax level before the code is actually executed. Specifically, the processor can check whether the signature of the generated function meets the requirements and whether it references only the available variables allowed via the prompt. Furthermore, by analyzing prohibited system calls, unnecessary external network connections, or attempts to access unauthorized system files, the processor can block malicious or incorrect code in advance that could compromise security and system stability.

[0342] The processor can dynamically verify candidate interface code and execute it in an isolated environment. Specifically, the processor can experimentally execute the generated interface function using arbitrary example input data within an isolated execution environment (e.g., a sandbox environment) that is logically separated from the main learning system. During this process, the processor can check whether the dimensions and shapes of the input and output multidimensional arrays (tensors) exactly match predefined specifications. Additionally, the processor can dynamically verify whether the batch processing of the multidimensional arrays is performed normally, whether the function's execution time does not exceed an allowed computational threshold, and whether unexpected exceptions or memory leak errors occur during execution.

[0343] Additionally, the processor may perform exception handling and recovery in the event of a verification failure. Specifically, if an error is detected during the static or dynamic verification process and verification fails, the processor may activate exception handling logic to prevent the interruption of the entire learning pipeline. In one embodiment, the processor may discard the code that failed verification and retry verification by replacing it with another promising candidate interface code generated in the previous step. In another embodiment, the processor may analyze the cause of the verification failure (e.g., a dimension mismatch error message, etc.), modify the system prompt accordingly, and then request the large-scale language model to regenerate the interface code. In yet another embodiment, if it is determined that automatic recovery is impossible due to continuous generation and verification failures, the processor may be configured to fallback to a predefined default interface (e.g., a manually designed reference model) to maintain system stability.

[0344] Next, the processor can transmit the verified interface code to the code artifact registration and execution environment. (S270)

[0345] More specifically, the above processor may register the finalized interface code, which has passed the verification process, as a permanent code artifact and transfer it to the reconfigured Environment Wrapper of a subsequent learning step (e.g., step S300) for integration. This step may be performed to manage the history of the automatically generated environment interface and to establish a practical interaction loop between the reinforcement learning agent and the physics simulator.

[0346] Step S270 according to one embodiment may be performed including the following detailed registration, management, and distribution processes.

[0347] The processor can register code outputs and store metadata. Specifically, the processor can register the body data of the finalized observation model and behavior model interface code in a repository. At the same time, the processor can merge and store the unique identifier (hash) of the code, the structured system prompt used during code generation, version information of the simulation environment and robot, competition-based candidate selection results, and static and dynamic verification results into a single metadata package. Through this, the processor can ensure high reproducibility by restoring the environment configuration and control interface from the time of training without error when repeating the same experiment or deploying the trained policy model to a physical robot in the future.

[0348] The above processor can manage version and compatibility synchronization. Specifically, even when performing the same task, the structure of the optimized interface code may change due to updates to the simulation environment engine or minor changes to the robot hardware configuration (e.g., changes in sensor attachment positions, replacement of actuators, etc.). Therefore, the above processor can manage the interface code created and registered at a specific point in time and the weights (training checkpoints) of the reinforcement learning policy neural network that has completed training within that interface environment to always match the same set of versions. This can prevent critical control errors that occur when different versions of the observation space or behavior control space are mixed in the runtime environment.

[0349] In addition, the processor can deploy an execution environment and integrate a wrapper. The processor can deploy registered observation model function and behavior model function codes by inserting them into a reconfigured environment wrapper that encloses the input and output of the original simulator. Specifically, the processor can control the system to operate in such a way that the registered code is automatically called at every step within the training and evaluation loop of the reinforcement learning agent. That is, the processor can complete a cyclic structure in which the low-dimensional potential behavior output by the agent is converted into a full driver command through the behavior model function and applied to the simulator, and the raw observation returned from the simulator is converted into a task-oriented low-dimensional observation vector through the observation model function and provided back to the agent.

[0350] FIG. 13 is a performance evaluation graph comparing the learning efficiency (Sample Efficiency) between the interface automatic generation framework (LOAM and LOAM-Race) of the present disclosure according to one embodiment and prior art (FastTD3, LESR, etc.).

[0351] Referring to the learning curves for (a) the locomotion task, (b) the static manipulation task, and (c) the dynamic manipulation task in FIG. 13, it can be seen that the learning curve (solid line and square marker, etc.) applying the framework of the present disclosure (LOAM, LOAM-Race) draws the steepest rising curve compared to the prior art and rapidly converges to a high return value.

[0352] In particular, the embodiments of the present disclosure optimize the search space by removing unnecessary observation variables through a large-scale language model and mathematically designing a synergy-based behavior mapping (Equations 1 and 2). As a result, as shown in FIG. 13, it is possible to provide a technical effect that achieves asymptotic performance equivalent to or greater than that of conventional manual-based design or simple feature addition methods (LESR), while significantly shortening the timestep required to reach the target performance.

[0353] Referring to FIG. 14, step S410 may be an initialization process for setting up a specific reinforcement learning experiment environment and algorithm to match the task the user intends to achieve through the web interface of the integrated platform. The processor of the server computing system may receive an experiment configuration for multi-stage learning from the user terminal.

[0354] First, the processor may provide a hierarchical menu for selecting an environment to the user terminal. Through the hierarchical menu, the user can sequentially select a major category, a detailed category, and a specific task of the simulation environment. In one embodiment, the major category may be provided as robot motion or navigation, and the detailed category may be provided as Muzoco, Antmaze, or Adroite, which are environments compatible with OpenAI Gym. Additionally, when the user selects a specific environment or task, the processor may automatically play and provide a video on one side of the screen that exemplifies the ideal target action that an agent must perform in that environment. For example, the video may include a bipedal robot walking without falling over. Through this, the user can intuitively perceive the dynamic characteristics of complex physical simulation tasks that are difficult to understand from text descriptions alone, and create a project.

[0355] In addition, the processor may provide a pre-built asset library containing verified algorithms and datasets in system memory so that the user can configure experiments without a separate complex coding process. The user can easily configure experiments by selecting desired algorithms, benchmark datasets, and checkpoints, which are pre-trained models, from the library embedded in the platform. The processor may classify and provide the library list according to learning paradigms, and the user can select a reinforcement learning algorithm to use in the experiment from the displayed list. In one embodiment, the learning paradigm may be subdivided into online reinforcement learning, offline reinforcement learning, and offline-to-online reinforcement learning. Furthermore, the learning paradigm may further include a Multi-Agent Reinforcement Learning (MARL) paradigm for cooperation and coordination among multiple robot swarms.

[0356] Specifically, online reinforcement learning algorithms can be expressed as PPO, TD3, SAC, AWAC, IQL, PARS, etc., offline reinforcement learning algorithms can be expressed as BC, TD3-BC, CQL, AWAC, IQL, PARS, etc., and offline-to-online reinforcement learning algorithms can be expressed as AWAC, Off2On, IQL, Cal-QL, SPOT, RLPD, PARS, OPT, etc.

[0357] In addition, the above multi-agent reinforcement learning algorithm can be integrated with external open-source frameworks (e.g., EPyMARL) and expressed as a list of cooperative algorithms such as MAPPO (Multi-Agent PPO) and QMIX, and additional behavioral model options utilizing a large language model (LLM) as an autonomous planner for the agent may be provided.

[0358] When performing offline training, users can select benchmark datasets such as D4RL. Additionally, they can specify a pre-trained model or a specific checkpoint to be used as the initial value for training, and the processor can be configured to selectively load only specific network parts, such as the feature extractor of the selected model, into the system memory. In addition to the checkpoints provided by default, custom models provided by the user can be utilized, and users can flexibly expand their assets by uploading their own custom algorithms, datasets, and environments as needed.

[0359] In this step, the processor can automatically integrate observation and behavior model functions, which are interface codes generated and verified through a large language model in the preceding S200 step, as part of the experiment configuration. When the user completes project creation through the web interface, the processor can automatically set the optimal code selected from the backend as the default environment wrapper for the project. Accordingly, the raw environment selected by the user can be immediately converted into a remodeled environment having an optimized input / output space designed by the large language model without manual coding or configuration intervention by the user, and the processor can deliver the converted experiment configuration to the training computing system to complete preparation for training input.

[0360] FIG. 15A illustrates a screen prior to the start of project learning according to one embodiment of the present disclosure. Referring to FIG. 15A, a processor of a server computing system may provide a learning execution screen (440) through a user terminal (400). Since learning has not yet started, the processor may simplify the rendering area within the learning execution screen (440) to an empty state or a state with no results. Additionally, the processor may clearly display a learning start control unit (442) at the bottom or on one side of the learning execution screen (440) to maintain a waiting state for receiving a learning start command from the user.

[0361] FIG. 15B may be a diagram illustrating an experiment creation and dataset selection interface for multi-phase learning according to one embodiment of the present disclosure. Referring to FIG. 15B, when a user inputs a new experiment creation command on the dashboard screen, etc., a processor of a server computing system may provide an experiment creation screen in the form of a popup overlay. The processor may provide a user interface that sequentially guides the process of setting an experiment name, selecting a dataset, selecting a pre-trained model, and selecting an algorithm through the experiment creation screen.

[0362] For example, in the dataset selection step, the processor may link with a pre-built asset library to provide a menu that allows selecting an option not to use a dataset or a specific benchmark dataset. In one embodiment, when a user selects a specific benchmark dataset, the processor may display a list of detailed dataset types, such as Expert, Medium Expert, Medium, or Random, as a sub-menu, allowing the user to easily configure an experimental environment for offline learning, etc., without complex coding.

[0363] Step S420 may be a process in which the processor of the training computing system performs Phase 1 learning according to the experimental configuration set in the previous step, and generates the resulting model weights and log data.

[0364] In this stage, if the first phase is offline learning, the processor may perform a preprocessing step before injecting the raw dataset into the reinforcement learning model. At this time, the raw state data included in the raw dataset may pass through the observation model function generated in the previous step. Through this, the raw state data can be transformed into a feature vector of the same specifications as the observation space that the agent will experience in the subsequent online phase. This is intended to prevent input space mismatch between offline and online learning and to ensure the continuity of learning.

[0365] Additionally, the processor of the training computing system may run a reinforcement learning algorithm using the preprocessed data or data obtained through interaction with a simulator. For example, when using a dataset-based algorithm as an offline paradigm, the processor may sample batches from the dataset and perform gradient descent on the policy network and value network. In one embodiment, the dataset-based algorithm may be TD3-BC or CQL, etc. On the other hand, when using an interaction-based algorithm as an online paradigm, the processor may roll out agents in a remodeling environment to collect experiences and update the policy based on the collected trajectories. In one embodiment, the interaction-based algorithm may be PPO or SAC, etc.

[0366] In another embodiment, when using a multi-agent reinforcement learning paradigm, the processor can drive simulation training so that a swarm of dozens of robots within a remodeling environment learns cooperation and collision avoidance by exchanging states and rewards through a standard interface of a multi-agent environment (e.g., PettingZoo API).

[0367] During the training process, the processor may generate checkpoints by saving the model's weight parameters in the form of snapshots at set intervals. For example, the set interval may be every 1,000 steps or every epoch. The generated checkpoints may be stored in a database of a server computing system along with a unique identifier, the time of creation, and performance indicators at that time. In one embodiment, the performance indicators may include average reward or loss values, etc. At the same time, the processor may process various metric data generated during the training process by streaming them in real-time to the server computing system so that they are utilized as basic data for configuring the monitoring user interface described later.

[0368] In addition, the processor may continuously monitor whether a set termination condition is satisfied. For example, the termination condition may be reaching the maximum learning step, convergence of target performance, or satisfaction of an early termination condition. If the termination condition is satisfied, the processor may terminate the first phase of learning and proceed to the next step, the transition decision process. After the learning phase is completed, the stored network weights may be used to initialize the next phase.

[0369] Step S430 may be a decision-making process that analyzes the performance of Phase 1 learning to determine whether to terminate the phase and proceed to Phase 2. This step may be performed by an automated policy or through manual user intervention.

[0370] In the case of automatic transition conditions, the processor of the training computing system can monitor in real time whether a preset transition trigger condition is satisfied while the first phase of training is in progress. The processor of the training computing system can automatically stop training and transmit a training termination signal to the server computing system if quantitative criteria or resource limits defined in advance by the user are satisfied. In one embodiment, the quantitative criteria may include conditions such as the average reward reaching a specific score or the validation error failing to improve for a certain number of consecutive times, and the resource limit may include conditions such as termination upon reaching a specific step. Additionally, as a manual transition condition, if a user monitors the training progress through a web user interface and clicks an input command to move to the next step after determining through an interactive evaluation that the agent's behavior is sufficiently mastered, the processor of the server computing system recognizes this as a transition signal and transmits a training termination command to the training computing system.

[0371] When a transition is determined, the processor of the server computing system can select the source checkpoint most suitable for use as the initial value of the second phase from among the multiple checkpoints generated in the first phase. In one embodiment, the model at the very last point in time may be used as the source checkpoint, but to prevent overfitting, it may be configured to automatically select the model at the point in time that recorded the highest score in the verification dataset. In this process, the processor can strictly guarantee the consistency of data input / output specifications between the first phase and the second phase by also checking whether the version matches with the interface code generated in the previous step.

[0372] Once a selected source checkpoint is confirmed, the processor can dynamically generate a second-phase experiment configuration based thereon. By initializing a new experiment from the selected checkpoint, a seamless transition between algorithms or learning paradigms can be made possible. In one embodiment, parameters such as a learning paradigm to be performed, an algorithm to be used, and a deployment environment may be defined in the generated second-phase configuration. For example, the learning paradigm may be configured as online fine-tuning, the algorithm may be configured as SAC, and the deployment environment may be configured as maintaining a simulation environment or changing between a simulation and a real environment. At this time, if the user specifies a heterogeneous target algorithm (e.g., policy-based algorithm MAPPO, etc.) with a different mathematical structure and optimization method from the source model of the first phase (e.g., QMIX, a value-based algorithm) for the second phase, the processor recognizes that an inevitable discrepancy occurs between the two algorithms in terms of neural network structure (presence or absence of Actor, Critic), layer depth, and package internal variable naming rules, and may transfer system control to a structure verification step (S440) through a compatibility checker described later.

[0373] Furthermore, the processor may record the connection relationship between the first phase and the second phase in a database as learning lineage information. The learning lineage information may include genealogical information that leads to the second phase online learning through a transition process after performing the first phase offline learning, and this can be utilized later for the user to clearly track the entire learning process and reproduce the experiment.

[0374] FIG. 16 illustrates a screen during or after a learning process according to one embodiment of the present disclosure. Referring to FIG. 16, a processor of a server computing system may render an experiment information panel (441) by placing it in the left or upper area of ​​the learning execution screen (440). The experiment information panel (441) may be displayed including at least one of current environment information, pre-training model information, algorithm information, and progress status information.

[0375] In addition, the processor may provide a learning curve display area (446) within the learning execution screen (440). The processor may arrange multiple indicator graph boxes, such as reward and loss, in the learning curve display area (446) to visually provide performance trends based on learning metric data received in real time from the training computing system. For example, the indicator graph boxes may be arranged in two to four units and placed side by side for readability. Additionally, the processor may place a learning end control unit (443) in an easily accessible location, such as the upper right corner of the learning execution screen (440). As a manual switching condition, if a user monitors the learning progress through the learning curve display area (446) and executes an input command to move to the next step, such as clicking the learning end control unit (443) when determining that the agent's behavior is sufficiently mastered, the processor may recognize this as a switching signal to the system and transmit a learning stop command to the training computing system.

[0376] Step S440 may be a process in which, when the source algorithm of the first phase and the target algorithm of the second phase are different, the processor of the training computing system verifies in advance the possibility of physical coupling between the two models to prevent transfer errors. As an example, when the algorithms are different, it may include a transition from the TD3 algorithm to the SAC algorithm, or a transition from the PPO algorithm to the CQL algorithm. The processor may execute the verification process through a compatibility checker within the processor to overcome the technical limitation of neural network structure mismatch that occurs when implementing multi-stage reinforcement learning.

[0377] The processor may refer to an algorithm map pre-stored in system memory to initiate the verification. The algorithm map may be a data structure that systematically defines the standard network structure and parameter layer information of each reinforcement learning algorithm supported by the platform. For example, the standard network structure may include an actor-critic structure, the presence or absence of a shared encoder, or the shape of an output head.

[0378] Specifically, the algorithm map may be implemented in the form of a Python dictionary, and may include metadata for each algorithm identifier (Key), such as the type of Actor, Critic, and Value network (e.g., Gaussian distribution, etc.), the number of layers (n_layers), and information on the underlying package used for the algorithm implementation (e.g., PyTorch, JAX, etc.).

[0379] The processor can verify whether the input and output dimensions of the source model and the target model physically match. Here, the input dimension may refer to the observation space, and the output dimension may refer to the behavior control space. Since all learning phases in the previous step can be configured to share a remodeling environment where the same interface code is applied as a wrapper, the processor can be systematically guaranteed that the specifications of the input and output tensors always match even in the case of a transition between heterogeneous algorithms, and can primarily verify and approve this through the compatibility checker.

[0380] Once the input / output specifications are confirmed, the processor can identify compatible layers capable of safely sharing parameters by comparing and analyzing the internal neural network structures of the source model and the target model. At this time, the comparison analysis and verification can be performed hierarchically. First, the processor performs a primary check to determine whether the base packages of the two models are identical by referring to the algorithm map. Second, for a specific network to be transferred, the processor can perform a secondary comparison to determine whether the network type and layer depth defined by the source model and the target algorithm match.

[0381] For example, since the backbone network commonly used by algorithms has the same structure, the processor can immediately identify it as the compatible region. The backbone network may be a feature extractor such as a convolutional neural network or a multilayer perceptron.

[0382] In addition, the processor can detect structural differences in the policy head, which is the output layer, according to the mathematical characteristics of the algorithm. As an example, the TD3 algorithm, which is a deterministic policy, has a single output, while the SAC algorithm, which is a probabilistic policy, can have two outputs, the mean and the standard deviation. The processor can detect structural differences between the layers through the compatibility checker to determine whether a full transition from the source model to the target model is possible, or whether only a partial transition, such as a backbone network, is possible.

[0383] The processor can generate a compatibility report based on the analysis results of the internal neural network structure. If compatibility is confirmed, the processor can generate a list of transferable layers and pass the data to a subsequent step for actually mapping parameters. Conversely, if a structural mismatch is detected where transferability is impossible due to significant structural differences, the processor can transmit the compatibility report to a server computing system. Based on the compatibility report, the processor of the server computing system can prevent system interruption by outputting a warning message to the user terminal or automatically changing settings to start learning using a random initialization method without parameter transfer. For example, the aforementioned structural mismatch where transferability is impossible may occur in situations such as the transition from value-based reinforcement learning to policy-based reinforcement learning.

[0384] Step S450 may be a process in which, after structural compatibility has been verified in the previous step, the processor of the training computing system injects the learned weights of the source model, which is the first phase model, into the corresponding layer of the target model, which is the second phase model, through a model loader within the system. The processor may perform specific detailed procedures to resolve the complexity of parameter transfer that occurs in a multi-stage learning environment.

[0385] When implementing multi-stage reinforcement learning, the naming conventions for internal parameters may differ depending on the algorithm implementation library or the type of algorithm. For example, the aforementioned algorithm implementation library may be StableBaselines3, Tianshow, or CleanRL. Additionally, even for policy neural networks performing the same role, they may be named ActorNet in a specific algorithm, whereas PolicyActor in another. As a more specific example at the parameter level, in the case of algorithms implemented by different developers, even for the same neural network layer, the internal variable name may be defined as actor.0.weight in the source model, whereas it may be defined differently as policy_net.linear1.weight in the target model.

[0386] To systematically resolve this discrepancy, the processor may refer to an algorithm prefix map stored in system memory through the model loader. Specifically, the processor may parse key values ​​included in the state dictionary of the source model through the model loader and automatically convert the key string of the source parameter into a format that conforms to the naming rules of the target model by applying the matching rules of the algorithm prefix map.

[0387] Through the above transformation, the processor can establish a seamless continuous learning pipeline between heterogeneous algorithms by accurately extracting semantically identical parameters (Weights, Biases) and injecting them into the target model, even if the user is unaware of the internal implementation differences of the target algorithm.

[0388] Once the key transformation is complete, the processor can copy and inject the weight tensor values ​​of the source model into the memory space of the target model through the model loader. If it is determined in the previous step that the structures of the source model and the target model are completely identical, the processor can perform a full transition in which the weight and bias values ​​of all layers are transferred as they are. On the other hand, if it is determined in the previous step that only some layers are compatible, the processor can perform a partial transition in which only the weights of specific layers for which compatibility has been approved are selectively injected. For example, the case where only some layers are compatible may include cases where the feature extractor, which is the backbone network, is shared but the policy head is different.

[0389] In addition, the processor may apply a predefined initialization technique to neural network layers that are excluded from the transition target or newly added to the target model of the second phase, thereby initializing the corresponding parameters to random values. For example, the newly added neural network layer may be a new auxiliary head, and the predefined initialization technique may be Xavier initialization or orthogonal initialization. Through this selective transition and initialization mechanism, the second phase model may begin learning while maintaining the pre-learned feature extraction capability acquired in the first phase, while simultaneously securing flexibility for new exploration in a new paradigm.

[0390] Additionally, according to one embodiment, the processor may determine whether to transition not only to weights but also to the optimizer state of the source model. The optimizer state may be momentum or learning rate scheduling information, etc. In scenarios where the continuity of learning must be maximized, such as when resuming after a pause within the same algorithm, the processor may improve learning efficiency by processing the optimizer state to be restored together through the model loader.

[0391] Step S460 may be a process in which the processor of the training computing system performs full-scale learning of Phase 2 or subsequent phases, using the parameters transferred from the previous step as initial values. This step may be performed under a different paradigm or environmental conditions than the previous step.

[0392] For example, if the second phase is online learning, the processor can control the agent to accumulate experience by directly interacting with the simulator within the remodeling environment.

[0393] As an example, this may include an offline-to-online scenario in which online fine-tuning is performed after offline pre-training. Based on the transferred parameters, the processor can control the agent to perform meaningful actions rather than random actions from the beginning of training, and can refine the policy through rewards, which are real-time feedback.

[0394] This parameter transfer-based online fine-tuning can significantly increase sample efficiency compared to pure online learning by utilizing pre-trained policies, and can achieve performance that surpasses pure offline approaches or online approaches that learn from scratch.

[0395] At this time, all actions output by the agent can pass through the behavior model function generated in the previous step, and the processor can convert them into physically safe and efficient control signals and transmit them to the environment. This can provide a systemic effect of preventing the risk of damage to the simulator or agent that may occur during the exploration process.

[0396] As another embodiment, the second phase may be performed in a real physical environment or a digital twin environment rather than a simulation environment.

[0397] This may be an example of an online-to-online transition scenario that is highly valuable for simulation-to-reality transition. The processor can deploy a model learned in a simulation to a real robot and perform additional learning to adapt to actual physical laws using a limited number of real-world interaction data. For example, the actual physical laws may include friction or backlash.

[0398] Through this, the processor can be controlled to achieve a faster convergence speed than learning from scratch. In this process, the processor can be controlled so that the observation and behavior model functions based on the large language model generated in the previous step operate as wrappers in the same way even when deployed in a real environment. Through this, the processor can perfectly maintain interface consistency of input and output data specifications between the simulation and reality, and minimize transition errors.

[0399] Step S470 may be a cyclic management process that records the entire process of multi-stage learning and continues learning by extending to additional phases as needed. This step may include a procedure in which a processor of a server computing system manages the history of a complex training pipeline and packages the final output.

[0400] The processor can generate an integrated learning lineage in a database by linking metadata, such as settings and outputs used in a given phase, whenever each learning phase is completed. The processor can track lineage information by recording the entire path through which the current model was completed—including which data and algorithms—as a data structure. For example, the entire path may include a phase sequence leading from the first phase to the second phase via a parameter transition process, as well as information on the algorithms used. Additionally, the processor can support the user in analyzing the contribution to performance improvement throughout the entire learning process and identifying the optimal combination by integrating and storing the trends of change in performance indicators for each stage. In one embodiment, the performance indicators may include reward or loss values, etc.

[0401] The system according to the embodiment of the present disclosure is not limited to a unidirectional offline-to-online transition, and even after the second phase is completed, learning can be flexibly extended to additional phases, such as a third or fourth phase, according to user settings or automated policies. For example, the processor may configure a complex learning pipeline that sequentially leads to offline pre-learning, simulated online learning, and real-world adaptive learning, and instruct a training computing system to execute it. In this case, the processor may control the process to repeatedly cycle from the transition decision stage to the next phase execution stage, thereby ensuring that knowledge accumulated in the model is continuously transferred to the next stage without interruption.

[0402] When all configured learning phases are successfully completed, the processor can bundle the weights of the acquired final model, the interface code linked in the previous step, and the entire generated learning lineage information into a single integrated package and permanently store them as artifacts. In one embodiment, the interface code may include a wrapper function. Through this, when a similar task is given in the future, the processor can provide a reuse basis that prevents the agent from learning again from the beginning and controls the permanently stored artifact to be loaded into system memory for immediate reuse or to perform additional learning with a small amount of data.

[0403] If all configured learning phases are successfully completed or there is an explicit save command from the user, the processor may perform a procedure to permanently save the final output.

[0404] FIG. 17 is a schematic diagram illustrating the process of heterogeneous reinforcement learning framework integration and target algorithm selection according to one embodiment of the present disclosure.

[0405] Referring to FIG. 17, a series of data flows is illustrated in which, at step S410, the processor interacts with an external framework for multi-stage continuous learning and selects a heterogeneous target algorithm.

[0406] First, the processor loads a serialized configuration file (881, e.g., a YAML file) created and stored through the preceding environment setup steps (S100 to S200). The loaded configuration file (881) is instantiated into a machine learning-enabled simulation environment object through a multi-agent standard API environment wrapper (882, e.g., PettingZoo API).

[0407] As a key feature of the present disclosure, the processor is not dependent on a single algorithm implementation and dynamically interacts with the system and an external heterogeneous multi-agent reinforcement learning framework (883, e.g., EPyMARL) based on the standard API environment wrapper (882).

[0408] When the integration is complete, the processor provides the user with a list of multiple learning algorithms supported by the heterogeneous framework (883) through the target algorithm selection interface (884). The user can specify a specific target algorithm (e.g., MAPPO, QMIX, etc.) for the second phase of learning by manipulating the dropdown list, etc., of the selection interface (884).

[0409] When the user selects a new target algorithm and triggers multi-stage continuous learning, the system prepares to inject weight checkpoints (885) of an artificial intelligence model trained in the first phase or loaded from an external source into the selected algorithm architecture.

[0410] At this time, the processor systematically recognizes that if the target algorithm specified through the selection interface (884) is a heterogeneous algorithm that differs from the source algorithm that generated the checkpoint (885) in mathematical structure (e.g., value-based vs. policy-based) or internal implementation method, there inevitably exist structural inconsistencies in neural network layers and inconsistencies in variable naming rules between the two models.

[0411] Consequently, the flow of FIG. 17 suggests a parameter conflict problem between heterogeneous frameworks (883) that cannot be resolved by the conventional simple parameter copying method, and this provides a key prerequisite and justification for the processor to transfer control to the 'structure verification process using a compatibility checker and algorithm map' in step S440, which will be described later, for the safe transfer of parameters.

[0412] FIG. 18 is a schematic diagram showing the simulation dashboard operation and real-time verification screen of a multi-agent system according to one embodiment of the present disclosure.

[0413] Referring to FIG. 18, a simulation dashboard interface (890) is illustrated that integrally supports the dynamic interaction performance of step S460 and the interactive monitoring process of step S500. When the simulation mode is activated, the processor provides synchronized qualitative behavior visualization and quantitative performance indicators through multiple visually separated panels.

[0414] First, the user can adjust runtime environment parameters, such as the playback speed (FPS) of the simulation, through the control panel (891) on the left, and initiate the interactive learning and simulation of step S460 by operating the Run, Pause, or Step buttons of the simulation control unit (895) located at the bottom.

[0415] In response to the initiation signal, the processor renders the physical behavior of multiple agents in the form of a real-time animation in the main grid view (892) at the top right. As illustrated in FIG. 18, multiple active agent objects (893) injected with a previously specified behavior model (e.g., a reinforcement learning policy or a large language model planner) actively interact with the environment, such as transporting stock shelves to a work station or moving to a charging station. All observations and control commands of the agents generated during this process are processed by passing through observation model functions and behavior model function wrappers generated through the large language model in step S200, thereby ensuring a physically valid runtime without system crashes.

[0416] At the same time, the processor displays a live chart panel (894) at the bottom of the main grid view (892) in synchronization according to the monitoring process of step S500. The live chart panel (894) includes a plurality of graph objects that update key performance indicators in real time according to the simulation progress timestep. As a specific embodiment, the live chart panel (894) intuitively provides the total throughput of the logistics center or factory, a bar graph showing the distribution of the work status of individual agents (e.g., idle, movement, interaction, charging, etc.), the average order picking completion time, and the trend of cumulative rewards when reinforcement learning is applied.

[0417] Through this dual-view-based interactive evaluation environment, the user can immediately visually verify bottlenecks or local collision patterns between robot swarms that are difficult to identify using only simple numerical training logs. The user can determine whether the training in the current phase is sufficient by comprehensively analyzing the qualitative behavior of the main grid view (892) and the quantitative indicators of the live chart panel (894), and can quickly make subsequent decisions to deploy the verified optimal model parameters and interface code to the actual physical robot environment (Sim-to-Real).

[0418] FIG. 19 may illustrate a screen for storing learning results according to one embodiment of the present disclosure.

[0419] Referring to FIG. 19, the processor of the server computing system may activate and display an experiment storage control unit (444) at a location adjacent to the learning end control unit (443) of the learning execution screen (440). When a control signal is received through the user terminal (400) for the user to select the experiment storage control unit (444), the processor may display a storage confirmation popup (445) in the form of a separate box overlay on the center or one side of the screen. The processor may provide guidance text regarding the experiment storage to the user through the storage confirmation popup (445) and request final confirmation. When the user's confirmation input is received through the storage confirmation popup (445), the processor may control the permanent storage of metadata displayed on the experiment information panel (441) and the weight data of the final model corresponding to the learning curve display area (446) in the system's database.

[0420] FIG. 20 is a flowchart illustrating the control result of an artificial intelligence model having a plurality of learning phases according to one embodiment of the present disclosure.

[0421] Referring to FIG. 20, step S510 may be a paradigm-aware visualization and dynamic layout separation process in which a processor of a server computing system logically arranges and displays received heterogeneous learning log data on a web-based user interface. This step may be performed to technically resolve distortion problems that occur when offline learning and online learning data are mixed.

[0422] First, the processor receives training log data streamed in real-time from a training computing system and can identify which training paradigm the currently received log data belongs to. In one embodiment, the processor can identify the training paradigm by parsing experimental configuration information from a previous stage or metadata from the header of the received log data. If the identified paradigm is an offline phase where iterative training is performed on a pre-built dataset, the processor can render the visualization graph by automatically setting the horizontal axis reference to gradient descent steps or epochs rather than environment interaction steps. This allows the user to accurately determine the sample complexity and convergence speed within a limited dataset. On the other hand, if the identified paradigm is an online phase where interaction with the environment is performed, the processor can automatically convert and set the horizontal axis reference of the graph to interaction steps or episodes. This can be utilized as an objective indicator showing the agent's adaptability over time in an actual simulation environment.

[0423] In addition, the processor can dynamically adjust and separate the visualization layout of the user interface to prevent confusion in data interpretation and visual distortion that may occur when offline learning indicators and online learning indicators are simply arranged on a single two-dimensional plane. As an example, the processor can control the display to separate the offline learning result area and the online learning result area into separate panel objects or tab objects that are visually clearly distinguished according to the paradigm attributes of the received log data.

[0424] FIG. 21 may be a drawing illustrating a screen that separately displays on / offline learning performance according to one embodiment of the present disclosure.

[0425] Referring to FIG. 21, the processor of the server computing system can provide a performance comparison screen (450) that allows multiple experimental results to be superimposed and compared through a user terminal (400). The processor can provide multiple experiment tabs (451) at the top of the performance comparison screen (450) and control the user to select desired experimental models to activate the comparison mode.

[0426] The processor can dynamically adjust and separate the visualization layout according to the paradigm-aware visualization principle described above. Specifically, the processor can control the first paradigm learning result area (452) and the second paradigm learning result area (453) to be separated into separate panel objects that are visually clearly distinguished according to the paradigm attributes of the received log data and displayed on the display. For example, if the first paradigm learning result area (452) represents offline learning results for a pre-built dataset, the processor can render the visualization graph by automatically setting the horizontal axis standard to gradient descent steps or epochs rather than environment interaction steps. On the other hand, if the second paradigm learning result area (453) represents online learning results where interaction with the environment is performed, the processor can render the graph in a separate area by automatically setting the horizontal axis standard to interaction steps or episodes.

[0427] Through this, when the processor displays the performance of different experiments superimposed in comparison mode, it renders by aligning the axes only between sections with the same paradigm, thereby fundamentally preventing misanalysis situations where the user incorrectly compares graphs with different data systems.

[0428] Step S520 may be a process in which the processor of the server computing system visualizes the connections of the multi-stage learning performed in the preceding multi-stage learning orchestration step as a diagram in the form of a learning lineage and renders it on the user interface. This step may be performed to help the user intuitively grasp the entire lineage of complexly intertwined multi-stage reinforcement learning and identify the optimal model combination.

[0429] The processor can parse learning step metadata recorded in the system's database and generate a graphic object of a logical hierarchical node structure extending from a parent node, which is a pre-trained model, to a child node, which is a fine-tuned model, through links representing parameter transitions. Additionally, the processor can control the system to automatically summarize and map data on the specific reinforcement learning algorithm and dataset used in the corresponding learning phase, the learning progress status, and the performance indicator data achieved in that phase to each generated node object. In one embodiment, the reinforcement learning algorithm may be TD3, SAC, etc., and the performance indicator may be the final reward or the highest reward. Through this, the processor can control the system so that the user can grasp the position and results of individual experiments within the overall learning pipeline at a glance.

[0430] FIG. 22 may be a drawing illustrating a multi-phase learning status and a Lineage management interface according to one embodiment of the present disclosure.

[0431] Referring to FIG. 22, the processor of the server computing system can provide a dashboard screen (600) through a user terminal in which a list of experiments of multi-stage learning is listed in a table format. The processor can control the display of algorithm information (610), pre-training model information (620), and progress status information (630) for each experiment on the dashboard screen (600) so that the user can intuitively grasp the entire lineage of complexly intertwined multi-stage reinforcement learning.

[0432] Specifically, the processor can render whether the experiment is based on an offline paradigm or an online paradigm through the algorithm information (610) so that it can be intuitively identified along with the algorithm name. Additionally, the processor can specify through the pre-training model information (620) which previous experiment's checkpoint parameters were transferred from to initialize the experiment. Through this, the user can check the lineage of the entire multi-stage learning, such as what the parent model of a specific experiment is, at a glance on the table without a separate complex tracking process. Furthermore, the processor can provide real-time updates on the current status of each experiment, such as success, in progress, or failure, through the progress status information (630).

[0433] Furthermore, by rendering the generated learning lineage view on a display, the processor can continuously track and provide how the agent's performance has progressively improved, starting from an initial random policy and passing through interface code generated by a large language model in the previous stage. Through the provided visualization interface, the user can intuitively identify at which phase transition point the greatest performance leap occurred. Ultimately, the processor can provide a visual analysis tool that helps the researcher, as the user, effectively identify and design the optimal algorithm sequence and learning paradigm combination for a specific task.

[0434] Step S530 may be a process in which, when a user selects a specific point on the interactive learning curve, the processor of the server computing system immediately synchronizes and provides quantitative metrics and qualitative behavioral video of the model at that point. This step may be performed to support the user in visually evaluating the agent's actual physical behavior, which is difficult to grasp solely from numerical reward metrics, and in determining whether to transition to a phase based on this evaluation.

[0435] The processor can render an interactive learning curve on a web interface based on metric data received from a training computing system. Users can monitor the training progress of ongoing or completed multi-stage learning in real time through an integrated interface. At this time, the processor can control the visualization of the learning curve to map and display multiple checkpoints generated at regular intervals as selectable graphic markers. For example, the graphic markers may be in the form of nodes or points.

[0436] The processor may receive a first user input from a user terminal, which selects a specific checkpoint on the learning curve through input means such as a mouse click or hovering. In one embodiment, the user may select a checkpoint at a specific step point. Upon receiving the first user input, the processor may query a database to identify model weights and simulation environment configuration data mapped to the corresponding checkpoint, and request a training computing system to perform a simulation for behavior visualization. When a behavior video is received from the training computing system in response to the request, the processor may control the display of the behavior video on a user interface.

[0437] Additionally, the processor may create an evaluation panel on one side of the web interface and synchronize the loaded behavioral video with the quantitative metrics at the corresponding point in time to display them on the display. Specifically, the processor may control the evaluation panel so that a video player playing the specific behavioral patterns of the agent is provided alongside key performance indicators recorded at the corresponding checkpoint in numerical form. In one embodiment, the key performance indicators may include average reward, average normalized score, or episode length, etc. Furthermore, the user may perform additional parameter settings, such as adjusting the number of rollout episodes to be evaluated in the video playback environment.

[0438] The user can comprehensively analyze the behavioral visualization video and indicators provided in synchronization to intuitively evaluate whether the agent's training at the corresponding checkpoint has been sufficiently performed. In one embodiment, if the average reward value is high but the agent's walking posture in the video is unstable or unnecessary shaking is observed, the user may determine that the learning is still insufficient. If the evaluation result indicates that the training is sufficient, the processor may receive an input commanding the user to directly initialize a new experiment on the interface. In response to the input, the processor may control the smooth transition to a new algorithm or a different paradigm by confirming a selected specific checkpoint as the initial value for the next phase and starting the next learning step without delay.

[0439] Step S540 may be a process in which the processor of the training computing system immediately runs the agent's simulation and renders the behavioral video in response to a specific checkpoint-based evaluation request received in the previous step. This step may be performed with specific data processing procedures to minimize latency and ensure perfect consistency between the training environment and the evaluation environment.

[0440] When the processor receives the evaluation request from the server computing system, it can dynamically load the model weights at that point in time from the database into system memory. At the same time, the processor can call the interface code generated and finalized through the large language model in the previous step. In one embodiment, the interface code may include an observation model function and a behavior model function. The processor can control the immediate reconstruction of the remodeling environment for evaluation by combining the called interface code in the form of a wrapper with the simulation environment to be evaluated. Through this step, inconsistencies in the specifications of input / output data tensors between the training phase and the evaluation phase can be fundamentally prevented, and the processor can thereby strictly guarantee physical reliability during evaluation using parameters transferred from the previous step.

[0441] In addition, the processor can perform agent simulations a number of times specified by the user by linking the model network loaded in the memory with the reconstructed remodeling environment. For example, the specified number may be 10 episodes rolled out. At this time, to provide a near-real-time user experience, the processor can be controlled to minimize computation waiting time in a complex simulation environment by assigning multiple evaluation episodes to multiple computation nodes or processes and processing them in parallel instead of single-threaded computation.

[0442] During the simulation, the processor can capture an array of image frames representing the interaction state between the agent and the physical environment in real time from the simulator's rendering pipeline into memory. The captured frame data can be sequentially encoded through a video codec within the processor and finally converted into an action video file that can be played immediately in a web environment.

[0443] Once computation and encoding are completed, the processor can transmit the generated video file and quantitative indicators obtained from the simulation to a server computing system in the form of an integrated data packet. In one embodiment, the quantitative indicators may include average reward, normalized score, or episode length. The processor of the server computing system can asynchronously transmit the received data to a user terminal, and the user terminal can parse the received data and immediately display an evaluation pop-up window as an overlay on the screen. Through the provided video, the user can intuitively determine qualitative elements, such as whether the agent falls or accurately approaches the target, by analyzing the behavior. Based on this, if the processor of the server computing system determines that the learning of the current phase is sufficient, it can receive a decision input signal instructing a transition to the next phase and control the subsequent steps to be processed quickly.

[0444] FIG. 23 may be a drawing illustrating a user interface for checkpoint-based interactive evaluation and verification according to one embodiment of the present disclosure.

[0445] Referring to FIG. 23, a processor of a server computing system may provide an evaluation interface (700) for verifying performance at a specific point in time during multi-stage learning through a user terminal. When a user selects a specific checkpoint on a learning curve, the processor may display time point information of the corresponding model (e.g., Step_50K) in a checkpoint identification unit (710) within the evaluation interface (700). Additionally, the processor may provide an evaluation parameter setting unit (720) to receive parameters such as the number of rollout episodes to be simulated from the user and receive an evaluation execution command.

[0446] When the above evaluation execution command is received and simulation and rendering are completed in the training computing system, the processor can play a video showing the agent's actual behavior on the screen through the behavior visualization rendering unit (730). In synchronization with this, the processor can numerically display and provide quantitative metrics such as the average reward, normalized score, and episode length obtained from the evaluation on the performance indicator display unit (740). Furthermore, the processor can place an evaluation history panel (750) on one side of the screen to accumulate and provide evaluation results performed for different checkpoints in a table format, thereby supporting the user in intuitively comparing and determining the optimal phase transition point.

[0447] FIG. 24 may be a diagram illustrating a performance comparison graph before and after the application of a multi-stage learning pipeline according to an embodiment of the present disclosure. Referring to FIG. 24, performance indicators of an offline-to-online transition scenario may be displayed, in which pre-training is performed using an offline dataset in the first phase, and then fine-tuning is performed by interacting with an online environment in the second phase. As the model parameters of the first phase are successfully transferred to the target model of the second phase through compatibility verification via the processor of the training computing device of the present disclosure, the agent can obtain high reward values ​​and begin meaningful actions from the beginning of the second phase of training. This demonstrates a technical effect that overcomes the performance limitations of conventional purely offline approaches and dramatically improves sample efficiency and learning convergence speed compared to a control group that performs online training from the beginning in a random initialization state.

[0448] Additionally, referring to FIG. 25, performance indicators of a simulation-to-reality transition scenario may be displayed, in which a first phase online learning is performed in a simulation environment, and then a second phase online learning is additionally performed in a real physical environment or a heterogeneous dynamic environment where gravity values, etc., are changed. Through parameter transfer between heterogeneous algorithms and interface code consistency maintenance techniques, the processor can control the agent to adapt quickly and stably to new environmental conditions with only limited interaction data. Consequently, the present disclosure can provide the effect of achieving excellent convergence speed and scalability, which allows the target performance to be reached with significantly fewer interaction steps than learning from scratch in a new physical environment.

[0449]

[0450] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be modified into one or more software modules to perform processing according to the present disclosure, and vice versa.

[0451] The specific embodiments described in this disclosure are examples and do not limit the scope of this disclosure in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated as “essential,” “importantly,” etc., a component may not be strictly necessary for the application of this disclosure.

[0452] Furthermore, although the detailed description of the present disclosure has been explained with reference to preferred embodiments of the present disclosure, those skilled in the art or those with ordinary knowledge in the art will understand that the present disclosure can be modified and changed in various ways without departing from the spirit and technical scope of the present disclosure as set forth in the claims below. Accordingly, the technical scope of the present disclosure should not be limited to the contents described in the detailed description of the specification but should be determined by the claims.

[0453] The present invention can be directly utilized in a simulation system for integrated control and optimization of multiple autonomous driving robots (AGVs) and automation equipment in the smart factory and smart logistics industries. It enables the construction of an intuitive environment without complex hardcoding and rapid pre-verification of heterogeneous control algorithms, thereby significantly reducing the time and cost required to build automation infrastructure.

Claims

1. As a simulation method for an industrial automation environment executed by a computer, At least one processor of the computer visually arranges a plurality of infrastructure components in a virtual workspace and sets detailed operational parameters through a visual editor provided to a user computing device, and receives user input regarding a task to be performed by an agent cluster; The step of the above at least one processor loading a hardware specification file corresponding to the infrastructure components stored in at least one memory of the computer based on the received input; The step of the above at least one processor combining spatial arrangement information of the arranged components and operational parameter values ​​to generate data in a standardized structural text format having a hierarchical structure; The above at least one processor parses the loaded hardware specification file to extract the state variable space and actuator space of the agent cluster and normalizes them into machine-readable metadata; The step of the at least one processor inputting the normalized metadata and user input for the task into at least one pre-trained artificial intelligence model to generate an interface code that mediates data exchange between the agent cluster and the simulation environment; and The above-mentioned at least one processor links at least one behavioral model determined via the generated interface code to a simulation environment, and implements content representing the interaction result of the agent cluster performed in the simulation environment through the user interface of the user computing device; Industrial automation environment simulation method.

2. In Paragraph 1, The step of receiving user input for the above task is characterized by including the step of receiving input selecting any one of a standard industrial scenario template, which includes at least one of order picking, inventory loading, and automated production facilities pre-built in the system, in order to define a control objective function and interaction rules to be performed by the agent cluster within the workspace. Industrial automation environment simulation method.

3. In Paragraph 1, The step of receiving user input for the above task is, It further includes a step of receiving a task description in the form of unstructured natural language text regarding a specific interaction rule of the above agent cluster, and The above task description is characterized by being utilized as a constraint of a prompt injected into the above artificial intelligence model. Industrial automation environment simulation method.

4. In Paragraph 1, The step of generating data in the above-mentioned standardized structural text format is characterized by including the step of serializing the above-mentioned layout information and operational parameter values ​​into a file in YAML (YAML Not Markup Language) format and saving it in an export form, or loading a previously saved YAML file to restore the layout on the above-mentioned virtual workspace. Industrial automation environment simulation method.

5. In Paragraph 1, The above hardware specification file is a file in XML or URDF format, and The step of parsing the above hardware specification file is, Characterized by including the step of identifying kinematic connectivity of the agent cluster by tracing a hierarchical tree of multiple links, joints, sensors, and actuators constituting the robot. Industrial automation environment simulation method.

6. In Paragraph 1, The step of normalizing into the above machine-readable metadata is, Characterized by including the step of identifying tensor dimension and array-based index mapping information for each variable of the extracted state variable space and actuator space, and combining the identified information with the data in the form of annotations along with data types and physical meanings. Industrial automation environment simulation method.

7. In Paragraph 6, The above at least one pre-trained artificial intelligence model includes a large-scale language model, and The above-described large-scale language model is characterized by being controlled to block hallucinations regarding system variable references by referencing the metadata combined with the tensor dimension and index mapping information when generating the interface code. Industrial automation environment simulation method.

8. In Paragraph 1, The step of generating the above interface code is, The method is characterized by including the step of automatically generating, through the artificial intelligence model, an observation model function that converts raw data collected from the simulation environment into a form recognizable by the behavior model, and a behavior model function that maps control commands determined by the behavior model to the physical specifications of the actuator space. Industrial automation environment simulation method.

9. In paragraph 1, the step of linking the behavior model to a simulation environment is, Characterized by including the step of wrapping the observation transmission and behavior input processes of the above behavior model in accordance with the international standard API specifications for multi-agent interaction and dynamically binding them with the above simulation environment object. Industrial automation environment simulation method.

10. In Paragraph 1, The above behavioral model is characterized as being a heterogeneous model comprising at least one of a user-defined Python script-based rule-based heuristic algorithm, a multi-agent reinforcement learning (MARL) policy model directly linked with an external open-source framework, and an autonomous planner based on a giant language model (LLM). Industrial automation environment simulation method.

11. In Paragraph 10, The step of linking the above behavioral model to a simulation environment is, The method is characterized by including the step of assigning the rule-based heuristic algorithm to a first group among the agent clusters within a single simulation environment, and assigning the multi-agent reinforcement learning policy model or the autonomous planner to a second group in a mixed manner to enable precise verification of interaction and collision avoidance performance. Industrial automation environment simulation method.

12. In Paragraph 1, The step of implementing the above content is, The method is characterized by providing a dual-view-based multi-window structure dashboard comprising a main grid view area that renders the physical behavior of the agent cluster in the form of a real-time animation, and a live chart panel area that displays at least one of the total throughput of the agent cluster, the distribution of the work status of individual agents, and the cumulative reward trend in synchronization according to the simulation progress time step. Industrial automation environment simulation method.

13. In Paragraph 2, The step of receiving user input for the above task is, It includes the step of being assigned a composite scenario in which the above order picking task and the above inventory loading task are performed simultaneously within a single layout, and An industrial automation environment simulation method characterized by the above behavioral model operating to simultaneously optimize conflicting goals of minimizing total order completion time and minimizing loading completion time for the above agent cluster.

Images