Coal mine XR hybrid decision-making system and method based on cooperative game and interactive feedback
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2024-11-26
- Publication Date
- 2026-06-04
Smart Images
Figure CN2024134558_04062026_PF_FP_ABST
Abstract
Description
Coal Mine XR Hybrid Decision System and Method Based on Cooperative Game Theory and Interactive Feedback Technical Field
[0001] This application relates to the field of human-machine hybrid augmented intelligence, and in particular to a coal mine XR hybrid decision-making system and method based on cooperative game theory and interactive feedback. Background Technology
[0002] With the development of technology, people have clearly realized that for a long time to come, no level of artificial intelligence can completely replace humans. Therefore, the concept of human-machine hybrid augmented intelligence has emerged: introducing human roles or cognitive models into physical intelligent systems, and constructing a higher level of intelligence through the deep integration of human intelligence and machine intelligence.
[0003] In the coal mining sector, with the development of intelligent coal mining, coal mining robots are gradually being applied underground, aiming to reduce the labor intensity of coal mine operators and improve production efficiency. However, the underground coal mine environment is highly uncertain, and the tasks are open and complex, making it impossible for coal mining robots to handle all tasks. When faced with complex and highly random tasks, deep involvement of coal mine operators is still necessary. Therefore, coal mines can be considered a typical application scenario for human-machine hybrid augmented intelligence.
[0004] The process of completing complex tasks in the underground coal mine environment can be divided into three levels: perception, decision-making, and execution. Decision-making is the core, serving as the bridge between perceived information and actual action. Within the framework of human-machine hybrid augmented intelligence, the task decision-making process should combine human intuition, judgment, and creativity with the robot's efficient and accurate data processing capabilities to achieve human-machine hybrid decision-making. However, humans and robots are inherently heterogeneous, differing in their working methods, communication styles, and perceptual levels. Therefore, current human-machine hybrid decision-making is still in the exploratory stage.
[0005] Current human-machine hybrid decision-making methods rely on prior rules, confidence thresholds, model-based decision-making, or fixed task assignments, lacking true real-time human-machine interaction and adaptability, making them difficult to apply to highly complex and open-ended tasks in underground coal mines. Summary of the Invention
[0006] The purpose of this application is to provide a coal mine XR hybrid decision-making system and method based on cooperative game theory and interactive feedback, so as to realize real-time deep human-machine interaction, improve the flexibility of hybrid decision-making, and be applicable to highly complex and open operation tasks in underground coal mines.
[0007] To achieve the above objectives, this application provides the following solution:
[0008] In the first aspect, this application provides a coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback, including: a coal mine physical operation subsystem, a VR dynamic scheduling subsystem and an AR interactive feedback subsystem;
[0009] The coal mine physical operation subsystem includes the operating environment, the tasks to be completed, and the smart miners and coal mine operation robots that participate in the tasks to be completed.
[0010] The smart miner is an underground operator in a coal mine equipped with auxiliary computing equipment, AR glasses, and a smart explosion-proof helmet; the auxiliary computing equipment is the operating platform of the VR dynamic scheduling subsystem; the AR glasses are the operating platform of the AR interactive feedback subsystem; and the smart explosion-proof helmet is used to monitor the physiological state of the smart miner.
[0011] The coal mine operation robot is equipped with sensing elements and a host computer; the sensing elements include a 3D LiDAR, a depth camera, and an industrial camera; the sensing elements are used to monitor environmental and task perception information;
[0012] The VR dynamic scheduling subsystem is used to perform task allocation, behavior optimization, and spatiotemporal simulation based on the physiological state of the smart miner, the environmental and task perception information, and the tasks to be completed.
[0013] The AR interactive feedback subsystem is used by the smart miner to provide feedback on the optimal behavior-level allocation scheme of the VR dynamic scheduling subsystem, generate feedback opinions, and transmit the feedback opinions to the VR dynamic scheduling subsystem.
[0014] Secondly, this application provides a coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback. This method is applied to the aforementioned coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback. The method includes:
[0015] Based on environmental and task awareness information, a hierarchical task network is used to recursively decompose the tasks to be completed into task level, behavior level, and action level; the task level is the overall task objective; the behavior level is the sub-objective of the overall task objective; and the action level is the execution action to complete the sub-objective.
[0016] Based on the sub-objectives, the optimal behavior-level allocation scheme is determined using a mixed-integer linear programming algorithm;
[0017] Based on the feedback, a multi-agent reinforcement learning algorithm is used to determine preference weights, task allocation suggestions, and utility functions; the feedback is obtained by the intelligent miner in response to the optimal behavior-level allocation scheme.
[0018] Based on the preference weights, the task allocation suggestions, and the utility function, the optimal behavior-level allocation scheme is incrementally adjusted using a dynamic allocation algorithm to obtain an optimized behavior-level allocation scheme.
[0019] Based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm are determined using the D*Lite path planning algorithm and the artificial bee colony-particle swarm optimization algorithm.
[0020] The intelligent miner plans its own actions and executes the task to be completed based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm.
[0021] The coal mine operation robot executes the task to be completed based on its walking path and the movement trajectory of its robotic arm.
[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0023] This application provides a coal mine XR hybrid decision-making system and method based on cooperative game theory and interactive feedback. The system includes a coal mine physical operation subsystem, a VR dynamic scheduling subsystem, and an AR interactive feedback subsystem. The coal mine physical operation subsystem includes the working environment, tasks to be completed, and intelligent miners and coal mine operation robots participating in the tasks. The intelligent miner is an underground operator equipped with auxiliary computing equipment, AR glasses, and an intelligent explosion-proof helmet. The auxiliary computing equipment is the operating platform for the VR dynamic scheduling subsystem; the AR glasses are the operating platform for the AR interactive feedback subsystem; the intelligent explosion-proof helmet is used to monitor the physiological state of the intelligent miner; the coal mine operation robot is equipped with sensing elements and a host computer; the sensing elements are used to monitor environmental and task perception information; the VR dynamic scheduling subsystem is used to perform task allocation, behavior optimization, and spatiotemporal extrapolation based on the intelligent miner's physiological state, environmental and task perception information, and tasks to be completed; the AR interactive feedback subsystem is used for the intelligent miner to provide feedback on the optimal behavior-level allocation scheme of the VR dynamic scheduling subsystem, generate feedback opinions, and transmit the feedback opinions to the VR dynamic scheduling subsystem. This system can extend the human-machine hybrid decision-making process to the virtual space. Through the transformation and transmission of heterogeneous information flow between the three subsystems, it can rationally integrate the decisions of smart miners and coal mine operation robots while meeting the requirements of the operation tasks, forming a virtual-real integration and human-machine co-intelligence operation mode. This enables real-time deep human-machine interaction, improves the flexibility of hybrid decision-making, and can be applied to highly complex and open operation tasks in underground coal mines. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 is a diagram of the overall architecture of a coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback provided in this application.
[0026] Figure 2 shows the layout and dependency relationship of the coal mine physical operation subsystem, VR dynamic scheduling subsystem and AR interactive feedback subsystem of this application.
[0027] Figure 3 is a schematic diagram of the task hierarchy divided by the task decomposition module in this application;
[0028] Figure 4 is a schematic diagram of the task hierarchy in the task decomposition module of this application, taking the replacement of the cutting teeth of the coal mining machine drum as an example.
[0029] Figure 5 is a schematic diagram of the operation principle of the behavior-level allocation module in this application;
[0030] Figure 6 is a schematic diagram of the behavior-level allocation scheme for the replacement of the cutting teeth of the coal mining machine drum in this application;
[0031] Figure 7 is a schematic diagram of the operation principle of the cooperative game module in this application;
[0032] Figure 8 shows the interaction relationship between the modules of the AR interactive feedback subsystem and the VR dynamic scheduling subsystem of this application;
[0033] Figure 9 is a schematic diagram of the operation principle of the communication network architecture of this application;
[0034] Figure 10 is a flowchart of the overall operation of a coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback provided in this application.
[0035] Figure 11 shows the information flow diagram in the S2-S6 operation process. Detailed Implementation
[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] This application provides a coal mine XR hybrid decision-making system and method based on cooperative game theory and interactive feedback, aiming to solve the problems proposed in the background technology, such as the lack of real-time deep human-machine hybrid decision-making systems or methods, poor flexibility of decision-making hybrid mechanisms, and limited adaptability in dynamic and uncertain application scenarios.
[0039] In an exemplary embodiment, as shown in Figure 1, a coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback is provided, including: a coal mine physical operation subsystem, a VR dynamic scheduling subsystem, an AR interactive feedback subsystem, and a communication network module (communication network architecture). The arrangement and dependency relationship of the coal mine physical operation subsystem, the VR dynamic scheduling subsystem, and the AR interactive feedback subsystem are shown in Figure 2.
[0040] The coal mine physical operation subsystem includes the operating environment, the tasks to be completed, and the smart miners and coal mine operation robots that participate in the tasks to be completed.
[0041] The smart miner is a local operator in a coal mine equipped with auxiliary computing equipment, AR glasses, and a smart explosion-proof helmet.
[0042] The auxiliary computing device is a lightweight edge computer housed in an explosion-proof container. It serves as the operating platform for the VR dynamic scheduling subsystem and can be carried by the smart miner. The auxiliary computing device is equipped with embedded CPU and GPU modules, possessing strong computing and graphics processing capabilities, and also supports wireless network communication.
[0043] The head-mounted AR glasses are the operating platform of the AR interactive feedback subsystem. They have three interaction modes: gesture, voice, and gaze. They support wireless network communication and serve as the interface for smart miners to deeply and in real-time participate in the human-machine hybrid process.
[0044] The intelligent explosion-proof helmet integrates an EEG detection module and an EEG information processing module, which can be used by the VR dynamic scheduling subsystem to monitor the physiological state of the smart miner.
[0045] In one embodiment, the EEG detection module is used to monitor the EEG signals of the smart miner; the EEG information processing module is used to process the EEG signals.
[0046] Coal mine operation robots are tracked, wheeled, or legged mobile robots equipped with robotic arms. These robots are equipped with sensing elements and a host computer; the sensing elements are used to monitor environmental and task-related information. These sensing elements include 3D LiDAR, depth cameras, and industrial cameras, and possess a certain degree of autonomous perception and control capabilities.
[0047] The VR dynamic scheduling subsystem is used to perform task allocation, behavior optimization, and spatiotemporal simulation based on the physiological state of the smart miner, the environmental and task perception information, and the tasks to be completed.
[0048] The VR dynamic scheduling subsystem is developed using Unity3d as its core engine and includes modules for task decomposition, behavior-level allocation, cooperative game theory, behavior-level optimization, action-level planning, hyperspace simulation, and online evaluation and learning evolution.
[0049] In one embodiment, the VR dynamic scheduling subsystem includes:
[0050] The task decomposition module is used to recursively decompose the task to be completed into task level, behavior level and action level based on environmental and task awareness information and using a hierarchical task network. The task level is the overall task objective of the task to be completed; the behavior level is the sub-objective of the overall task objective; and the action level is the execution action to complete the sub-objective.
[0051] The task decomposition module is based on the Hierarchical Task Network (HTN). It can recursively decompose tasks into three levels: task level, behavior level, and action level, based on the environmental and task perception information provided by the AR glasses on the smart miner and the sensing elements on the coal mining robot, as well as the overall operation objectives, as shown in Figure 3.
[0052] The task level is the highest level of decomposition, representing the overall operational goals that the system needs to achieve.
[0053] Behavioral level is an intermediate level of decomposition. It breaks down the task level into a set of more specific steps, which can be seen as sub-goals of the task.
[0054] The action level is the lowest level of decomposition, which includes specific, executable actions and cannot be further decomposed. Since coal mine operators have a certain degree of flexibility and can plan their own actions according to specific behaviors, all behaviors are considered to be executed by the coal mine operation robot during action-level decomposition. If a behavior is assigned to the intelligent miner in subsequent behavior-level allocation, the action-level decomposition result is ignored, and the intelligent miner plans its own actions. Ultimately, the action level is essentially reflected in the movement of the coal mine operation robot and the rotation of the robotic arm joints.
[0055] As shown in Figure 4, taking the replacement of the cutting tooth on the coal mining machine drum as an example, after being decomposed by the task decomposition module, the task level is "replace a certain cutting tooth on the coal mining machine drum", the behavior level is a set of specific steps including "confirm the position of the cutting tooth", "grab the new cutting tooth", "move to the position of the cutting tooth", "remove the cutting tooth", "install the new cutting tooth" and "confirm the new cutting tooth is installed in place", and the action level is the movement and mechanical arm joint rotation required for the coal mine operation robot to perform each step.
[0056] The behavior-level allocation module is used to determine the optimal behavior-level allocation scheme based on the sub-objective using a mixed-integer linear programming algorithm.
[0057] The behavior-level assignment module is based on the Mixed-Integer Linear Programming (MILP) algorithm, and its operating principle is shown in Figure 5. Its operation process is as follows: define decision variables, objective function, and constraints; then use a solver to solve the problem to find the solution that optimizes the objective function, thus achieving optimal behavior-level assignment.
[0058] The decision variable is a binary variable x. ij x ij =1 indicates that behavior j is assigned to participant i (intelligent miner or coal mine robot), x ij =0 indicates that behavior j was not assigned to participant i.
[0059] The objective function is to minimize job completion time, maximize job efficiency, balance workload, and maximize job execution quality. Its expression is:
[0060] Where N represents the number of participants, including smart miners and coal mine robots, M represents the total number of actions, and T ij E represents the time required for participant i to complete behavior j. ij L represents the work efficiency of participant i when performing behavior j. i Q represents the workload of participant i. ij This represents the quality of the task performed by participant i in executing behavior j. The weighting coefficients α1, α2, α3, and α4 are used to adjust the priority of different objectives.
[0061] The constraints include constraint 1, constraint 2, and constraint 3. Constraint 1 ensures that each behavior must be assigned; that is, for any behavior j, Constraint 2 takes into account the capability limitations of intelligent miners and coal mine operation robots, namely, complex and delicate operations and adjustments must be completed by intelligent miners, while simple repetitive and high-intensity operations must be completed by coal mine operation robots; Constraint 3 takes into account the physiological state of intelligent miners provided by intelligent explosion-proof helmets.
[0062] The solver should provide support for C# or be integrated with C# via a relevant interface to facilitate connection with Unity3d. In one embodiment, Gurobi is used as the solver, which provides a .NET API accessible via C#.
[0063] As shown in Figure 6, taking the replacement of the cutting tooth of a coal mining machine drum as an example, the task is assumed to be jointly performed by a smart miner and a coal mining robot. With N=2, the weighting coefficients α1, α2, α3, and α4 are set to 0.3, 0.3, 0.2, and 0.2, respectively. After Gurobi calculation, the three actions of "confirming the position of the cutting tooth", "removing the cutting tooth", and "confirming the new cutting tooth is installed in place" are assigned to the smart miner, while the three actions of "grabbing the new cutting tooth", "moving to the location of the cutting tooth", and "installing the new cutting tooth" are assigned to the coal mining robot.
[0064] The cooperative game theory module is used to determine preference weights, task allocation suggestions, and utility functions based on the feedback from the AR interactive feedback subsystem using a multi-agent reinforcement learning algorithm.
[0065] The cooperative game module has a built-in cooperative game model that can integrate feedback from smart miners and coal mine operation robots on behavior-level allocation schemes to obtain output information, so that the behavior-level optimization module can effectively adjust the behavior allocation. Its operating principle is shown in Figure 7.
[0066] The cooperative game theory model is based on Multi-Agent Reinforcement Learning (MARL). The training process involves constructing a simulated environment where intelligent miners and coal mining robots interact continuously. Q-learning is used for policy optimization. Each iteration provides feedback on individual and overall behavior through a reward function. After multiple iterations, the model gradually converges, ultimately forming the optimal policy. This ensures that the cooperation among agents maximizes common interests while satisfying their individual needs.
[0067] Feedback indicates whether the optimal behavior-level allocation scheme is accepted or not. If accepted, the optimal behavior-level allocation scheme is approved; if not accepted, the expected adjustment direction is given.
[0068] The output information is a combination of preference weights, task allocation suggestions, and utility functions.
[0069] The behavior-level optimization module is used to incrementally adjust the optimal behavior-level allocation scheme based on the preference weights, the task allocation suggestions, and the utility function using a dynamic allocation algorithm, so as to obtain the optimized behavior-level allocation scheme.
[0070] The behavior-level optimization module is based on a dynamic allocation algorithm. It can make incremental adjustments to the allocation results of the behavior-level allocation module based on the output information of the cooperative game module, and optimize the optimal behavior-level allocation scheme in a short time.
[0071] The action-level planning module is used to determine the walking path of the coal mine operation robot and the motion trajectory of the robotic arm based on the optimized behavior-level allocation scheme, using the D*Lite path planning algorithm and the artificial bee colony-particle swarm optimization algorithm.
[0072] The motion-level planning module has built-in path planning functions for coal mine robots based on the D*Lite path planning algorithm and trajectory planning functions for the robotic arm of coal mine operation robots based on the Artificial Bee Colony-Particle Swarm Optimization (ABC-PSO) algorithm.
[0073] The coal mine operation robot executes the task to be completed based on the walking path and the motion trajectory of the robotic arm; the smart miner plans its own actions and executes the task to be completed based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm.
[0074] The hypersonic simulation module is used to deduce the completion status of the task to be completed and the next stage movement of the coal mine operation robot in a virtual reconstruction scenario based on the optimized behavior-level allocation scheme. The virtual reconstruction scenario includes a twin agent of the coal mine operation robot and a twin agent of the working environment and the task to be completed.
[0075] The main body of the hyperspace simulation module is a real-time virtual reconstruction scene that includes a twin intelligent agent of the coal mine operation robot and twin agents of the local environment and operation tasks. It can simulate the completion status of the current operation task and the next stage of the coal mine operation robot's movement.
[0076] The twin agent of the coal mine operation robot is obtained through pre-modeling and can be updated in real time based on the sensor data of the coal mine operation robot. Pre-modeling includes the construction of geometric and mechanistic models. The geometric model is generated using computer-aided design (CAD) software, while the mechanistic model is constructed by writing inverse kinematics scripts based on the parameters and motion laws of the coal mine operation robot. The sensor data includes position information, attitude information, velocity information, and the joint angles and force conditions of the robotic arm.
[0077] The local environment and task twin are obtained from the perception information provided by the smart miner and the coal mine operation robot, mainly including changes in the local environment and the progress of the task.
[0078] The prediction of the current task completion status is based on the real-time status of the coal mine operation robot twin agent and the local environment and task twin agent; the real-time status of the coal mine operation robot twin agent is updated by data driven by the sensor elements of the coal mine operation robot, and the real-time status of the local environment and task twin agent is updated by the perception information of the smart miner; the completion status of the task is judged by comparing the real-time status update with the task objective.
[0079] The prediction of the next stage of the coal mine operation robot is based on the completion status of the current operation task. That is, the completion status of the current operation task is compared with the behavior-level optimization results to obtain the behavior of the coal mine operation robot in the next stage and its corresponding action-level planning results, and pre-execution in the virtual reconstruction scenario.
[0080] An online evaluation and learning evolution module is used to record feedback and the completion results of tasks to be completed, and to learn the preference trends of the smart miner and the coal mine operation robot using a preference learning algorithm.
[0081] The online evaluation and learning evolution module can record the choices, feedback, and completion results of tasks to be completed by the smart miner and the coal mining robot at different decision-making stages during the human-machine hybrid decision-making process. Through preference learning algorithms, it gradually learns the preference trends of the smart miner and the coal mining robot and continuously optimizes the human-machine hybrid decision-making strategy.
[0082] The AR interactive feedback subsystem is used by the smart miner to provide feedback on the optimal behavior-level allocation scheme of the VR dynamic scheduling subsystem, generate feedback opinions, and transmit the feedback opinions to the VR dynamic scheduling subsystem.
[0083] The AR interactive feedback subsystem is developed using Unity3d as its core engine and includes an adaptive push module and an online feedback module. The interaction relationship between the AR interactive feedback subsystem and the VR dynamic scheduling subsystem is shown in Figure 8.
[0084] In one embodiment, the AR interactive feedback subsystem includes:
[0085] An adaptive push module is used to push the optimal behavior-level allocation scheme, the optimized behavior-level allocation scheme, the path planning results, and the spatiotemporal simulation results to the smart miner through voice, graphics, and animation; the path planning results are the walking path of the coal mine operation robot and the motion trajectory of the robotic arm; the spatiotemporal simulation results are the completion status of the task to be completed and the next stage movement of the coal mine operation robot.
[0086] The adaptive push module has the functions of pushing behavior-level allocation results, behavior-level optimization results, path planning results, and time-space extrapolation results. It can also present relevant push information to the smart miner in real time through voice, graphics, and animation according to the progress of the human-machine hybrid decision-making process.
[0087] The behavior-level allocation result push function refers to presenting the allocation results of the behavior-level allocation module in the form of a Gantt chart on the head-mounted AR glasses, along with voice explanations for the smart miner to understand.
[0088] The behavior-level optimization result push function refers to presenting the optimization results of the behavior-level optimization module in the form of a Gantt chart in the head-mounted AR glasses, along with voice explanations for the smart miner to understand.
[0089] The time-space simulation result push function refers to presenting the simulation results of the coal mine operation robot in the time-space simulation module in the form of animation in the head-mounted AR glasses, and at the same time, it is accompanied by voice broadcast of the next stage of the coal mine operation robot's movement, so that the smart miner can plan its own actions in advance.
[0090] The online feedback module is used by smart miners to express their feedback on the optimal behavior-level allocation scheme in a multimodal interactive manner, and transmit it to the cooperative game module; the multimodal interaction includes gesture interaction, voice interaction and gaze interaction.
[0091] The online feedback module is used by smart miners to express their opinions on the allocation scheme in a multimodal interactive manner after obtaining the behavioral-level allocation results through the adaptive push module, and then transmits them to the cooperative game module.
[0092] Multimodal interaction includes gesture interaction, voice interaction, and gaze interaction. Smart miners can drag and drop bars in the Gantt chart through gesture or gaze interaction to change them to the desired time sequence. They can also use voice interaction to state in natural language how they want to allocate the results and the reasons for it.
[0093] In one embodiment, a communication network module is also included; the communication network module includes MQTT communication, Socket communication, and ROS# communication. The communication network architecture (communication network module) includes three network communication methods: MQTT, Socket, and ROS#, and its operating principle is shown in Figure 9.
[0094] The MQTT communication method is used for network communication between modules within the VR dynamic scheduling subsystem.
[0095] The Socket communication method is used for network communication between the VR dynamic scheduling subsystem and the AR interactive feedback subsystem.
[0096] The ROS# communication method is used for network communication between the VR dynamic scheduling subsystem and the host computer of the coal mine operation robot.
[0097] In an exemplary embodiment, this application provides a coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback. This method is applied to the aforementioned coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback. The method includes:
[0098] Based on environmental and task-aware information, a hierarchical task network is used to recursively decompose the tasks to be completed into task level, behavior level, and action level. The task level is the overall task objective of the task to be completed; the behavior level is the sub-objective of the overall task objective; and the action level is the execution action to complete the sub-objective.
[0099] Based on the sub-objectives, the optimal behavior-level allocation scheme is determined using a mixed-integer linear programming algorithm.
[0100] Based on the feedback, a multi-agent reinforcement learning algorithm is used to determine preference weights, task allocation suggestions, and utility functions; the feedback is obtained by the intelligent miner in response to the optimal behavior-level allocation scheme.
[0101] Based on the preference weights, the task allocation suggestions, and the utility function, the optimal behavior-level allocation scheme is incrementally adjusted using a dynamic allocation algorithm to obtain an optimized behavior-level allocation scheme.
[0102] Based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm are determined using the D*Lite path planning algorithm and the artificial bee colony-particle swarm optimization algorithm.
[0103] The intelligent miner plans its own actions and executes the tasks to be completed based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot, and the motion trajectory of the robotic arm.
[0104] The coal mine operation robot executes the task to be completed based on its walking path and the movement trajectory of its robotic arm.
[0105] In one embodiment, it further includes:
[0106] In the virtual reconstruction scenario, the completion status of the task to be completed and the next stage movement of the coal mine operation robot are deduced based on the optimized behavior-level allocation scheme; the virtual reconstruction scenario includes the twin agent of the coal mine operation robot and the twin agent of the working environment and the task to be completed.
[0107] As shown in Figure 10, a coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback includes the following steps: S1: Task decomposition; S2: Behavior-level allocation; S3: Human-machine feedback; S4: Cooperative game theory; S5: Behavior-level optimization; S6: Cyclic collaborative optimization; S7: Action-level planning; S8: Spatiotemporal extrapolation and push; S9: Online evaluation and learning evolution.
[0108] S1 includes the following steps:
[0109] S101: Input environmental and task awareness information and overall operational objectives into the task decomposition module.
[0110] S102: Based on HTN, the tasks to be completed are recursively decomposed into task level, behavior level and action level.
[0111] S103: Send the task decomposition results to the behavior-level allocation module.
[0112] S2 includes the following steps:
[0113] S201: Define decision variables, objective function, and constraints.
[0114] S202: Use a solver to solve the problem.
[0115] S203: Find the solution that optimizes the objective function and achieve optimal behavior-level allocation.
[0116] S204: The optimal behavior-level allocation scheme is transformed into a Gantt chart and voice explanation, and transmitted to the adaptive push module to be presented to the smart miner; the optimal behavior-level allocation scheme is transformed into a behavior sequence readable by the ROS system and transmitted to the host computer of the coal mine operation robot.
[0117] S3 includes the following steps:
[0118] S301: The intelligent miner observes and understands the Gantt chart of the optimal behavioral-level allocation scheme in the head-mounted AR glasses, and provides strategy suggestions to the cooperative game module through the online feedback module.
[0119] S302: The coal mine operation robot reads the behavior sequence instructions and feeds back strategy suggestions to the cooperative game module through ROS standard messages.
[0120] S4 includes the following steps:
[0121] S401: The cooperative game theory module integrates feedback from smart miners and coal mine operation robots regarding the optimal behavior-level allocation scheme.
[0122] S402: Output information is obtained through a MARL-based cooperative game model.
[0123] S403: Transmits the output information to the behavior-level tuning module.
[0124] S5 includes the following steps:
[0125] S501: The behavior-level optimization module optimizes the optimal behavior-level allocation scheme based on the output information of the cooperative game module.
[0126] S502: Transmit the optimization results (optimized behavior-level allocation scheme) to the smart miner and coal mine operation robot in the same manner as S204.
[0127] S6 includes the following steps:
[0128] S601: The intelligent miner and coal mine operation robot provide feedback to the cooperative game module on the optimization results in the same way as S3.
[0129] S602: The cooperative game module integrates the feedback results from the smart miner and the coal mine operation robot in the same way as S4, and conducts a re-game.
[0130] S603: The behavior-level tuning module re-tunes the optimal behavior-level allocation scheme in the same way as S5.
[0131] S604: Cycle through S601-S603 until both the task objective and the needs of both the human and machine are met simultaneously.
[0132] S605: Transmit the final optimization results to the task decomposition module.
[0133] The information flow in the S2-S6 operation process is shown in Figure 11.
[0134] S7 includes the following steps:
[0135] S701: The task decomposition module further decomposes the behavior-level units allocated to the coal mine operation robot in the behavior-level optimization module into action-level units, which are several action sequences that can be directly completed by the actuators of the coal mine operation robot.
[0136] S702: The task decomposition module transmits the decomposed action sequence to the action-level planning module.
[0137] S703: The motion-level planning module generates walking paths and robotic arm trajectories for each motion sequence of the coal mine robot using the D*Lite algorithm and the ABC-PSO algorithm.
[0138] S704: The motion-level planning module encapsulates the walking path and robotic arm trajectory data of the coal mining robot into a JSON format file and transmits it to the hyperspace simulation module; it also encapsulates the walking path and robotic arm trajectory data of the coal mining robot into a ROS standard message and transmits it to the host computer of the coal mining robot.
[0139] S8 includes the following steps:
[0140] S801: During the execution of the task, the time-space simulation module combines the kinematic parameters of the coal mine operation robot with the local environment and the status of the task twin to simulate the completion status of the current task in real time.
[0141] S802: The hypersonic simulation module simulates the next stage of the coal mine operation robot's movement in real time based on the completion status of the current task, and pushes it to the adaptive push module in the form of animation and voice broadcast, presenting it to the smart miner so that the smart miner can plan its own actions in advance.
[0142] S9 includes the following steps:
[0143] S901: During the human-machine hybrid decision-making and task execution process, the online evaluation and learning evolution module continuously optimizes the behavior-level allocation strategy.
[0144] S902: Transmit the optimized behavior-level allocation strategy to the behavior-level allocation module so that the system can continuously improve the quality of decision-making.
[0145] This application provides a coal mine XR hybrid decision-making system and method based on cooperative game theory and interactive feedback, which has the following advantages compared with related technologies:
[0146] (1) This application introduces VR technology and utilizes Unity3d’s powerful scripting capabilities, visualization spatiotemporal simulation capabilities and simulation computing capabilities to develop a VR dynamic scheduling subsystem, extending the human-machine hybrid decision-making process to the virtual space. It utilizes the coupled operation of multiple parallel modules to provide a computational and deductive basis for human-machine hybrid decision-making, enabling iteration, optimization, learning and evolution in the virtual space to achieve the best decision-making effect.
[0147] (2) This application deploys the VR dynamic scheduling subsystem in an auxiliary computing device that can be carried by smart miners, realizing a high degree of spatial integration of the coal mine physical operation subsystem, the VR dynamic scheduling subsystem and the AR interactive feedback subsystem, avoiding complex network infrastructure deployment and long-distance network communication, and significantly improving the efficiency of human-machine hybrid decision-making.
[0148] (3) This application introduces AR technology and uses the AR interactive feedback subsystem as the interface for smart miners to deeply participate in human-machine hybrid decision-making. This enables smart miners to understand the decision results intuitively and transparently, express their opinions and feedback naturally and conveniently, and communicate and collaborate directly with the VR dynamic scheduling subsystem and the coal mine operation robot in real time under cross-carrier and cross-cognitive conditions, which greatly enhances the proactive role of smart miners in actual decision-making.
[0149] (4) This application introduces cooperative game theory into the human-machine hybrid decision-making process. Under the premise of meeting the requirements of the task and the needs of both the human and the machine, the MARL algorithm is used to carry out feedback-game-optimization-refeedback-game-optimization loop optimization, and finally realize the human-machine two-way game interaction and problem solving of convergent complex problems, so that the human-machine hybrid decision-making results are more flexible and reliable in the complex and dynamic environment of coal mine.
[0150] (5) This application decomposes the tasks to be completed into three levels: task level, behavior level, and action level, so that the task allocation has a sense of hierarchy and precision, which facilitates the redistribution of tasks and local optimization. At the same time, it enables smart miners and coal mine operation robots to clearly understand the details of the tasks, so as to express their opinions and needs more accurately in the feedback and cooperative game process, thereby improving the flexibility and accuracy of human-machine hybrid decision-making.
[0151] (6) This application effectively combines the intelligent judgment ability of smart miners with the efficient computing ability of robots through the transformation and transmission of heterogeneous information flow between the three subsystems, so as to achieve complementarity and form a virtual-real integration and human-machine co-intelligence operation mode. It can play a role in human-machine collaboration in various complex tasks in coal mines, and can provide reference for human-machine hybrid decision-making in other fields.
[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0153] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback, characterized in that, include: Coal mine physical operation subsystem, VR dynamic scheduling subsystem, and AR interactive feedback subsystem; The coal mine physical operation subsystem includes the operating environment, the tasks to be completed, and the smart miners and coal mine operation robots that participate in the tasks to be completed. The smart miner is an underground operator in a coal mine equipped with auxiliary computing equipment, AR glasses, and a smart explosion-proof helmet; the auxiliary computing equipment is the operating platform of the VR dynamic scheduling subsystem; the AR glasses are the operating platform of the AR interactive feedback subsystem; and the smart explosion-proof helmet is used to monitor the physiological state of the smart miner. The coal mine operation robot is equipped with sensing elements and a host computer; the sensing elements include a 3D LiDAR, a depth camera, and an industrial camera; the sensing elements are used to monitor environmental and task perception information; The VR dynamic scheduling subsystem is used to perform task allocation, behavior optimization, and spatiotemporal simulation based on the physiological state of the smart miner, the environmental and task perception information, and the tasks to be completed. The AR interactive feedback subsystem is used by the smart miner to provide feedback on the optimal behavior-level allocation scheme of the VR dynamic scheduling subsystem, generate feedback opinions, and transmit the feedback opinions to the VR dynamic scheduling subsystem.
2. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback as described in claim 1, characterized in that, The VR dynamic scheduling subsystem includes: The task decomposition module is used to recursively decompose the task to be completed into task level, behavior level and action level based on environmental and task awareness information and using a hierarchical task network; the task level is the overall task objective; the behavior level is the sub-objective of the overall task objective; and the action level is the execution action to complete the sub-objective. The behavior-level allocation module is used to determine the optimal behavior-level allocation scheme based on the sub-objective using a mixed-integer linear programming algorithm. The cooperative game module is used to determine preference weights, task allocation suggestions, and utility functions based on the feedback from the AR interactive feedback subsystem using a multi-agent reinforcement learning algorithm. The behavior-level optimization module is used to incrementally adjust the optimal behavior-level allocation scheme based on the preference weights, the task allocation suggestions, and the utility function using a dynamic allocation algorithm to obtain the optimized behavior-level allocation scheme. The action-level planning module is used to determine the walking path of the coal mine operation robot and the motion trajectory of the robotic arm based on the optimized behavior-level allocation scheme, using the D*Lite path planning algorithm and the artificial bee colony-particle swarm optimization algorithm. The hypersonic simulation module is used to deduce the completion status of the task to be completed and the next stage movement of the coal mine operation robot in a virtual reconstruction scenario based on the optimized behavior-level allocation scheme. The virtual reconstruction scenario includes a twin agent of the coal mine operation robot and a twin agent of the working environment and the task to be completed.
3. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback as described in claim 2, characterized in that, The coal mine operation robot executes the task to be completed based on the walking path and the motion trajectory of the robotic arm; the smart miner plans its own actions and executes the task to be completed based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm.
4. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback as described in claim 2, characterized in that, The VR dynamic scheduling subsystem also includes: An online evaluation and learning evolution module is used to record feedback and the completion results of tasks to be completed, and to learn the preference trends of the smart miner and the coal mine operation robot using a preference learning algorithm.
5. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback according to claim 2, characterized in that, The AR interactive feedback subsystem includes: An adaptive push module is used to push the optimal behavior-level allocation scheme, the optimized behavior-level allocation scheme, the path planning results, and the spatiotemporal simulation results to the smart miner through voice, graphics, and animation; the path planning results are the walking path of the coal mine operation robot and the motion trajectory of the robotic arm; the spatiotemporal simulation results are the completion status of the task to be completed and the next stage movement of the coal mine operation robot; The online feedback module is used by smart miners to express their feedback on the optimal behavior-level allocation scheme in a multimodal interactive manner, and transmit it to the cooperative game module; the multimodal interaction includes gesture interaction, voice interaction and gaze interaction.
6. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback according to claim 1, characterized in that, The intelligent explosion-proof helmet has a built-in EEG detection module and an EEG information processing module; the EEG detection module is used to monitor the EEG signals of the intelligent miner; the EEG information processing module is used to process the EEG signals.
7. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback according to claim 1, characterized in that, The auxiliary computing device is a lightweight edge computer placed inside an explosion-proof container.
8. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback according to claim 1, characterized in that, The coal mine operation robot is a tracked, wheeled, or legged mobile robot equipped with an operating robotic arm.
9. The coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback according to claim 1, characterized in that, It also includes a communication network module; the communication network module includes MQTT communication mode, Socket communication mode and ROS# communication mode; MQTT communication is used for network communication between modules within the VR dynamic scheduling subsystem. Socket communication is used for network communication between the VR dynamic scheduling subsystem and the AR interactive feedback subsystem; The ROS# communication method is used for network communication between the VR dynamic scheduling subsystem and the host computer of the coal mine operation robot.
10. A hybrid XR decision-making method for coal mines based on cooperative game theory and interactive feedback, characterized in that, The coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback is applied to the coal mine XR hybrid decision-making system based on cooperative game theory and interactive feedback as described in any one of claims 1-9. The coal mine XR hybrid decision-making method based on cooperative game theory and interactive feedback includes: Based on environmental and task awareness information, a hierarchical task network is used to recursively decompose the tasks to be completed into task level, behavior level, and action level; the task level is the overall task objective; the behavior level is the sub-objective of the overall task objective; and the action level is the execution action to complete the sub-objective. Based on the sub-objectives, the optimal behavior-level allocation scheme is determined using a mixed-integer linear programming algorithm; Based on the feedback, a multi-agent reinforcement learning algorithm is used to determine preference weights, task allocation suggestions, and utility functions; the feedback is obtained by the intelligent miner in response to the optimal behavior-level allocation scheme. Based on the preference weights, the task allocation suggestions, and the utility function, the optimal behavior-level allocation scheme is incrementally adjusted using a dynamic allocation algorithm to obtain an optimized behavior-level allocation scheme. Based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm are determined using the D*Lite path planning algorithm and the artificial bee colony-particle swarm optimization algorithm. The intelligent miner plans its own actions and executes the task to be completed based on the optimized behavior-level allocation scheme, the walking path of the coal mine operation robot and the motion trajectory of the robotic arm. The coal mine operation robot executes the task to be completed based on its walking path and the movement trajectory of its robotic arm.