Information processing device, information processing system, information processing method, and information processing program

Abstract

This information processing device comprises a reception unit, a robot control unit, and a switching unit. The reception unit receives observation information of a control robot that is to be controlled. The robot control unit directs control of the control robot on the basis of the observation information received by the reception unit. The switching unit switches, from observation information observed in the control robot to observation information obtained by observing the control robot from an observation robot that is capable of observing the control robot, observation information used by the robot control unit to control the control robot when the quality of communication with the control robot deteriorates.

Description

Information Processing Apparatus, Information Processing System, Information Processing Method, and Information Processing Program

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and an information processing program.

[0002] In recent years, a multimodal large language model, which is a machine learning model capable of handling multiple types of data (modalities) such as images, voices, and videos in addition to text, is known. Also, techniques for controlling the actions (also referred to as operations) of robots and autonomous driving using multimodal large language models are known. For example, a technique is known in which a source code for controlling the actions of a robot is generated by a multimodal large language model, and the actions of the robot are controlled by the generated source code.

[0003] Wenlong Huang, et al., "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models", November 2023, [Searched on November 19, 2024], Internet <URL: https: / / arxiv.org / pdf / 2307.05973>

[0004] The information processing apparatus according to the present application includes a receiving unit, a robot control unit, and a switching unit. The receiving unit receives observation information of a control robot, which is a robot to be controlled. The robot control unit issues a control instruction for the target robot based on the observation information received by the receiving unit. The switching unit switches the observation information used by the robot control unit for controlling the control robot when the communication quality with the control robot deteriorates, from the observation information observed by the control robot to the observation information observed by an observation robot, which is a robot capable of observing the control robot, from the observation information observed by the control robot.

[0005] Figure 1 is a schematic diagram illustrating the information processing system according to the embodiment. Figure 2 is a schematic diagram illustrating the information processing according to the embodiment. Figure 3 is a diagram showing an example configuration of the information processing device according to the embodiment. Figure 4 is a diagram showing an example of a communication quality prediction map according to the embodiment. Figure 5 is a diagram showing an example of a communication area according to the embodiment. Figure 6 is a diagram showing an example configuration of a robot according to the embodiment. Figure 7 is a flowchart showing the processing procedure executed by the information processing device according to the embodiment. Figure 8 is a flowchart showing the processing procedure executed by the information processing device according to the embodiment. Figure 9 is a flowchart showing the processing procedure of a robot according to the embodiment. Figure 10 is a flowchart showing the processing procedure of a robot according to the embodiment. Figure 11 is a diagram showing an example of a hardware configuration.

[0006] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing device, information processing system, information processing method, and information processing program according to the present application. Note that these embodiments do not limit the information processing device, information processing system, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0007] (Embodiment) [1. Introduction] First, an overview of the information processing system according to the embodiment will be described using Figure 1. Figure 1 is an overview diagram of the information processing system according to the embodiment. As shown in Figure 1, the information processing system S comprises an information processing device 1 and a plurality of robots 50.

[0008] The information processing device 1 is, for example, a MEC (Multi-access Edge Computing) server. An MEC server is an edge server built in a location closer to the robot 50, such as a network before it reaches the internet or a local 5G network.

[0009] The information processing device 1 controls the actions (also called movements) of the robot 50 that are within its communication range. Specifically, the information processing device 1 controls the actions of the robot 50 using a multimodal large-scale language model (hereinafter sometimes referred to as "multimodal LLM").

[0010] As shown in Figure 1, the robot 50 is, for example, a quadruped robot that has been trained to follow a person. Note that the robot 50 is not limited to the example in Figure 1. The robot 50 operates under the control of the information processing device 1.

[0011] Specifically, the information processing device 1 receives observation information from the robot 50, generates control instructions based on the received observation information, and transmits them to the robot 50. The observation information includes various types of information observed by the robot 50, such as images taken by the robot 50 and coordinate information of each joint point of the robot 50.

[0012] The information processing device 1 generates control instructions for the robot 50 by inputting the observation information received from the robot 50 into the multimodal LLM. For example, consider a case where the task the robot 50 is performing is a task in which it follows a person.

[0013] In this case, the information processing device 1 inputs observational information, including images of a person taken by the robot 50 and coordinate information of each joint of the robot 50, to the multimodal LLM and generates control instructions for the robot 50 to follow the person.

[0014] Here, it is conceivable that communication delays may occur when the information processing device 1 receives observation information from the robot 50, such as when the communication bandwidth is congested. In such cases, the information processing device 1 may not be able to properly receive observation information from the robot 50 and may not be able to generate appropriate control instructions for the robot 50.

[0015] Therefore, in the information processing system S according to this embodiment, if the communication quality with robot 50 deteriorates, observation information of robot 50 is received from another robot 50. Here, an overview of the information processing according to this embodiment will be explained using Figure 2.

[0016] Figure 2 is an overview diagram illustrating the information processing according to the embodiment. As shown in Figure 2, robots 50a and 50b are located in degraded area Aa, where communication quality deteriorates, and normal area Ab, where communication quality does not deteriorate. Normal area Ab is the area where no communication delay occurs when robot 50 uploads observation information.

[0017] When the information processing device 1 receives observation information from the robot 50a in the degraded area Aa, it takes time to receive the observation information, and furthermore, the spectral efficiency of the observation information decreases. Such communication delays mean that it takes a long time for the control values ​​generated by the multimodal LLM to reach the robot 50a. In this case, the speed command value to be output changes depending on whether or not there is a communication delay, even if the image input to the multimodal LLM is the same. Specifically, if a communication delay of 120 msec occurs, it is necessary to generate a control instruction 120 msec later than when there is no 120 msec communication delay, so even if the image input to the multimodal LLM is the same, the speed command value to be output changes depending on whether or not there is a communication delay.

[0018] Therefore, the information processing system S according to this embodiment receives observation information of robot 50a in degraded area Aa from robot 50b in normal area Ab. Specifically, as will be described later, if robot 50a is in degraded area Aa, the information processing device 1 requests robot 50b in normal area Ab to observe robot 50a.

[0019] As a result, robot 50b, which is in the normal area Ab, observes robot 50a from the normal area Ab and transmits the observation information of robot 50a to the information processing device 1. For example, the observation information includes images taken by robot 50b of robot 50a or its surroundings (for example, a person to be followed).

[0020] As a result, the information processing device 1 can receive observation information from robot 50a in degraded area Aa without any communication delay. The information processing device 1 then uses the observation information received from robot 50b in normal area Ab to generate control instructions for robot 50a and transmits them to robot 50a.

[0021] This enables the robot 50a to operate based on control instructions received from the information processing device 1.

[0022] In this manner, the information processing system S according to this embodiment receives observation information of the control robot from an observation robot capable of observing the control robot when the communication quality with the control robot deteriorates.

[0023] Therefore, in the information processing system S according to this embodiment, observation information of the robot 50a in the degraded area Aa can be received without communication delay, so the robot 50 can be appropriately controlled under conditions where communication quality deteriorates.

[0024] [2. Configuration of the Information Processing Device] Figure 3 is a diagram showing an example of the configuration of the information processing device 1 according to the embodiment. As shown in Figure 3, the information processing device 1 according to the embodiment comprises a communication unit 2, a storage unit 3, and a control unit 4.

[0025] (Communication Unit 2) The communication unit 2 is wirelessly connected to a mobile network (cellular network) and transmits and receives information with the robot 50. For example, the communication unit 2 is implemented by a NIC (Network Interface Card) or an antenna. For example, the mobile network may be a 5G mobile network or a 6G mobile network. For example, the mobile network may include network equipment that provides functions for realizing the mobile network. The network equipment may be a base station (BS) or equipment that constitutes a core network (CN). The network equipment may also be an entity called an application function. The core network may be a 5G core network, an EPC (Evolved Packet Core), or a 6th generation or later core network. The base station forms a cell (area cell) and uses this cell to wirelessly communicate with the information processing device 1 and the robot 50. The communication unit 2 may include, for example, a Bluetooth® or Wi-Fi® antenna.

[0026] For example, the information processing device 1 may be a server device that implements the functions of a RIC (RAN Intelligent Controller). For example, the information processing device 1 may be a server device that implements the functions of an AI-RAN (Artificial Intelligence Radio Access Network). For example, the information processing device 1 may be a server device located within a mobile network that implements the functions of a base station within the mobile network using software. For example, the communication unit 2 provides functions for realizing a mobile network. For example, the information processing device 1 may form a cell (area cell) and use this cell to communicate wirelessly with the robot 50. For example, the communication unit 2 may form a cell (area cell) and use this cell to communicate wirelessly with the robot 50.

[0027] (Storage Unit 3) The storage unit 3 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. Specifically, the storage unit 3 stores information related to the multimodal LLM 31 and the communication quality prediction map 32. The storage unit 3 also stores the information processing program according to the embodiment.

[0028] The multimodal LLM31 is a large-scale language model that can handle multiple types of data simultaneously, such as text, images, audio, and video. For example, the multimodal LLM31 includes a modality encoder, an input projector, an LLM backbone, an output projector, and a modality generator.

[0029] A modality encoder is an encoder that takes images, audio, and video as input and extracts feature quantities from images, audio, and video. For example, a modality encoder takes instruction information indicating a person's instructions to the robot 50 and sensor information acquired by the robot 50 as input and extracts feature quantities from the instruction information and sensor information. For example, the sensor information may include at least one of either images or point cloud information relating to the environment around the robot 50.

[0030] The input projector converts the feature vectors output from the modality encoder into a data format that large language models (LLMs) can understand. For example, the input projector converts the feature vectors of instruction information and sensor information into a data format that large language models can understand. The input projector generates the feature vectors of instruction information and sensor information converted into a data format that large language models can understand.

[0031] The LLM backbone is a machine learning model that includes Large Language Models (LLMs). The LLM backbone uses data output from an input projector as input to the Large Language Model and generates text from the data output from the input projector. For example, the LLM backbone uses the feature quantities of instruction information and sensor information, which have been converted into a data format that the LLM can understand, as input to the LLM and generates text corresponding to the operation information based on the instruction information and sensor information.

[0032] The output projector converts the text output from the LLM backbone into a data format that the modality generator can understand. The output projector then generates operation information in a data format that the modality generator can understand.

[0033] A modality generator generates modality data other than text. For example, a modality generator generates the actions of robot 50. For example, a modality generator generates the actions of robot 50 by taking action information, which has been converted into a data format that the modality generator can understand, as input to the modality generator.

[0034] Furthermore, the multimodal LLM31 may be a machine learning model that includes a large-scale vision language model (VLM). For example, the multimodal LLM31 may be a machine learning model that includes GPT-4, Gemini, CogVLM, AnyGPT, or LLaMA3.

[0035] The communication quality prediction map 32 is a map that predicts future communication quality. Here, a specific example of the communication quality prediction map 32 will be explained using Figure 4. Figure 4 is a diagram showing an example of the communication quality prediction map 32 according to the embodiment.

[0036] Figure 4 shows a communication quality prediction map 32 for a certain area. As shown in Figure 4, the communication quality prediction map 32 is a map that predicts future communication quality for each area divided into a mesh-like structure.

[0037] Figure 4 shows the differences in communication quality based on the density of each mesh. By referring to this communication quality prediction map 32, the information processing device 1 can determine in advance whether the robot 50 will pass through a degraded area Aa where communication quality deteriorates.

[0038] Furthermore, the communication quality prediction map 32 may also indicate the communication area of ​​each information processing device 1. Figure 5 is a diagram showing an example of a communication area according to the embodiment. The communication area of ​​each information processing device 1 changes according to the number of users in the area, etc.

[0039] Therefore, if the communication bandwidth in a certain area becomes congested, the robot 50 can prevent the congestion by utilizing the information processing device 1 in other areas. Figure 5 shows the communication area at a certain time.

[0040] For example, Area 1 A1, Area 2 A2, and Area 3 A3 each represent communication areas of different information processing devices 1. Since these Area 1 A1, Area 2 A2, and Area 3 A3 change over time, for example, at a certain time, a portion of Area 1 A1 may change into Area 2 A2.

[0041] Furthermore, when the robot 50 moves across areas, it transmits observation information to the information processing device 1 corresponding to the area it has moved to, and receives control instructions from the information processing device 1.

[0042] (Control Unit 4) The control unit 4 is a controller, and is realized by executing various programs (for example, information processing programs) stored in the storage device inside the information processing device 1 using RAM or VRAM as the working area, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array).

[0043] The control unit 4 has a receiving unit 41, a robot control unit 42, a generating unit 43, a switching unit 44, and a transmitting unit 45 as functional units, and may realize or execute the operations of information processing described below. Note that the internal configuration of the control unit 4 is not limited to the configuration shown in FIG. 3, and any other configuration may be used as long as it can perform the information processing described later. Also, each functional unit represents the function of the control unit 4 and does not necessarily have to be physically distinct.

[0044] (Receiving Unit 41) The receiving unit 41 receives the observation information observed by the robot 50. More specifically, the receiving unit 41 acquires the observation information from the robot 50 via the communication unit 2. For example, the sensor information may be an image. Also, the observation information may be the coordinate information of the joint points of the robot 50 or the point cloud information acquired by LIDAR (Light Detection And Ranging).

[0045] Further, the receiving unit 41 receives the instruction information indicating the instruction of the person to the robot 50. For example, the instruction information may be text. More specifically, the communication unit 2 forms a cell (in-circle cell) and wirelessly communicates with the terminal device used by the person using this cell. Also, the receiving unit 41 receives the instruction information from the terminal device used by the person via the communication unit 2.

[0046] (Robot Control Unit 42) The robot control unit 42 issues a control instruction for the control robot based on the observation information received by the receiving unit 41. The robot control unit 42 generates a control instruction for the control robot by utilizing the multimodal LLM 31.

[0047] Specifically, the robot control unit 42 inputs the observation information received by the receiving unit 41 to the multimodal LLM 31 and generates a control instruction for the control robot. Also, when the receiving unit 41 receives the observation information of the control robot from the observation robot around the control robot, the robot control unit 42 inputs the observation information received by the receiving unit 41 from the observation robot to the multimodal LLM 31 and generates a control instruction for the control robot.

[0048] In addition, the robot control unit 42 also generates a control instruction for the observation robot that observes the control robot based on the observation information of the observation robot itself.

[0049] (Generator unit 43) The generator unit 43 generates a communication quality prediction map 32. For example, the generator unit 43 predicts the future communication quality in each area based on each robot 50 in the communication area, the communication status with the communication terminal, the position information of each robot 50 and the communication terminal, etc., and generates the communication quality prediction map 32. Note that the communication quality prediction map 32 may be generated by a higher-level device or the like.

[0050] (Switching unit 44) When the communication quality with the control robot deteriorates, the switching unit 44 switches the observation information used by the robot control unit 42 for controlling the control robot from the observation information observed by the control robot to the observation information obtained by observing the control robot from the observation robots around the control robot.

[0051] For example, when the control robot passes through a deterioration area Aa where the communication quality deteriorates in the communication quality prediction map 32, the switching unit 44 switches the observation information. Note that the switching of the observation information by the switching unit 44 is a process of switching the reception destination received by the reception unit 41, but it may also be a process of switching the observation information used when the robot control unit 42 generates a control instruction for the control robot. That is, in the latter case, while continuing to receive the observation information transmitted from the control robot, the robot control unit 42 generates a control instruction for the control robot based on the observation information transmitted from the observation robot.

[0052] For example, when it is predicted that the control robot will pass through the deterioration area Aa, the switching unit 44 makes the observation robots around the control robot wait for the observation of the control robot.

[0053] For example, the switching unit 44 predicts that the control robot will pass through the degraded area Aa when the control robot moves to the vicinity of the degraded area Aa in the normal area Ab. At this time, the switching unit 44 selects, for example, an observation robot from among the robots 50 surrounding the control robot to observe the control robot. The observation robot may be, for example, pre-positioned in the vicinity of the degraded area Aa.

[0054] The selection of the observation robot may be carried out according to any arbitrary rules. For example, the switching unit 44 selects as the observation robot one of the robots 50 in the vicinity of the control robot that is performing a low-priority task.

[0055] For example, in this case, the robot 50 selected as the observation robot will interrupt any task it is currently performing and begin its task as an observation robot. The observation robot may be, for example, a robot 50 specifically for observation. That is, the observation robot may be a robot 50 whose task is specifically to observe the control robot. Furthermore, the observation robot 50 may be a robot 50 with relatively high mobility, or it may be a robot 50 that can float, such as a drone.

[0056] Once the switching unit 44 has finished selecting the observation robot, it determines the standby position of the observation robot. For example, the standby position is a position where the control robot passing through the normal area Ab and the degraded area Aa can be observed.

[0057] For example, the switching unit 44 determines a standby position within the normal area Ab where there are no obstacles such as buildings when observing the control robot. Alternatively, the switching unit 44 may select multiple observation robots and decide to observe the control robot from different positions.

[0058] When the switching unit 44 determines the standby position of the observation robot, it transmits information such as the characteristics of the robot to be observed and information about the standby position to the robot 50 that will become the observation robot, via the communication unit 2.

[0059] Furthermore, the switching unit 44 instructs the observation robot to start observation when the control robot enters the deterioration area Aa.

[0060] As a result, the robot control unit 42 can generate control instructions for the control robot based on the observation information transmitted from the observation robot. Furthermore, by having the switching unit 44 stand by the observation robot in advance, it can quickly receive observation information from the observation robot when the control robot enters the degraded area Aa.

[0061] The switching unit 44 may also be configured to switch a portion of the observation information used by the robot control unit 42 to control the control robot from observation information obtained from the observation robot to observation information obtained from the control robot.

[0062] In this case, the observation robot observes information to interpolate the observation information from the control robot. For example, in this case, the switching unit 44 may select the type of information to receive from the control robot and the type of information to receive from the observation robot according to the communication quality of the area where the control robot is located. Specifically, the system may receive observation information of a relatively small capacity from the control robot so as not to cause communication delay even in degraded area Aa, and receive observation information of a relatively large capacity from the observation robot.

[0063] In this way, by using the observation robot to interpolate the observation information from the control robot, it is possible to prevent communication delays with the control robot while appropriately receiving the observation information necessary for controlling the control robot.

[0064] [3. Robot Configuration] Figure 6 shows an example of the configuration of the robot 50 according to the embodiment. The robot 50 according to the embodiment has a communication unit 52, a sensor unit 53, a drive unit 54, a storage unit 55, and a control unit 47.

[0065] (Communication Unit 52) ​​The communication unit 52 is wirelessly connected to the mobile network and transmits and receives information with the information processing device 1. For example, the communication unit 52 is implemented by a NIC, an antenna, etc. The communication unit 52 may also include, for example, a Bluetooth® or Wi-Fi® antenna.

[0066] (Sensor Unit 53) The sensor unit 53 has the function of collecting various sensor information in the environment surrounding the robot 50. For example, the sensor unit 53 includes an external sensor. The external sensor is a sensor that acquires information such as the shape of objects present around the robot 50 and the distance and direction to objects present around the robot 50, under the control of the control unit 47. For example, the external sensor includes at least one of the following: a camera, LiDAR (Light Detection And Ranging), Sonar, or ToF (Time of Flight). For example, the external sensor includes a 2D-LiDAR that acquires the environment as two-dimensional information or a 3D-LiDAR that acquires the environment as two-dimensional information. The sensor information also includes images generated by the camera. The images include still images and moving images. The sensor information also includes point cloud information generated by the LiDAR. For example, the sensor information includes point cloud information which is three-dimensional or two-dimensional information.

[0067] Furthermore, the sensor unit 53 includes an internal sensor. The internal sensor is a sensor that acquires information such as the distance traveled, speed, direction of movement, and posture of the robot 50 under the control of the control unit 47. For example, the internal sensor includes an inertial measurement unit (IMU) for detecting the orientation and acceleration of the robot 50's movement, and an encoder (or potentiometer) for detecting the amount of drive of the actuator. In addition to these, the internal sensor may also include acceleration sensors, angular velocity sensors, etc. Furthermore, the information detected by the external sensor and the internal sensor corresponds to an example of observed information.

[0068] (Drive Unit 54) The drive unit 54 has the function of driving the physical components of the robot 50. The drive unit 54 has the function of performing the movements of the robot 50. For example, the drive unit 54 has the function of moving the position of the robot 50. For example, the drive unit 54 controls the movement of the robot 50 in accordance with the control of the motion control unit 49. The drive unit 54 is, for example, an actuator or a motor driver. The drive unit 54 may have any configuration as long as the robot 50 can perform the desired movements. For example, the drive unit 54 moves the robot 50 and changes the position of the robot 50 by driving the movement mechanism of the robot 50 in response to instructions from the motion control unit 49.

[0069] (Storage Unit 55) The storage unit 55 is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 55 stores various programs that the robot 50 will execute.

[0070] (Control Unit 47) The control unit 47 is a controller, and is realized by executing various programs stored in the storage device inside the robot 50 using RAM or VRAM as the work area, for example, by a CPU, MPU, GPU, ASIC or FPGA.

[0071] The control unit 47 has an acquisition unit 58 and an operation control unit 49 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 47 is not limited to the configuration shown in Figure 7, and other configurations are also possible as long as they perform the information processing described later. In addition, each functional unit represents the function of the control unit 47 and does not necessarily have to be physically separated.

[0072] (Acquisition Unit 58) The acquisition unit 58 acquires observation information. Specifically, the acquisition unit 58 acquires observation information from the sensor unit 53. For example, the observation information may be an image. The sensor information may be point cloud information acquired by LIDAR. The acquisition unit 58 also acquires control instructions from the information processing device 1 via the communication unit 2.

[0073] (Motion Control Unit 49) The motion control unit 49 controls the operation of the robot 50 to execute an instructed task based on the control information. Specifically, the motion control unit 49 controls the operation of the robot 50 based on the control information so that the robot 50 outputs an operation corresponding to the control information. For example, the motion control unit 49 controls the operation of the robot 50 based on the control information received in response to a control instruction received from the information processing device 1.

[0074] [4. Processing Flow] Next, the processing procedure performed by the information processing system S according to the embodiment will be described using Figures 7 to 10. First, the processing procedure performed by the information processing device 1 will be described using Figures 7 and 8. Figures 7 and 8 are flowcharts showing the processing procedure performed by the information processing device 1 according to the embodiment.

[0075] As shown in Figure 7, the information processing device 1 first determines whether or not there is a robot 50 that is scheduled to enter the degraded area Aa where communication delays occur (step S101). If the information processing device 1 determines that there is a robot 50 that is scheduled to enter the degraded area Aa (step S101; Yes), it selects an observation robot to observe the robot 50 in the degraded area Aa (step S102).

[0076] Next, the information processing device 1 determines the observation position for the observation robot (step S103) and instructs the observation robot to wait (step S104). Subsequently, the information processing device 1 determines whether or not the robot 50 has entered the degraded area Aa (step S105), and if it has entered (step S105; Yes), it instructs the observation robot to start observation (step S106).

[0077] Furthermore, if the information processing device 1 determines in step S105 that the robot 50 has not entered the deterioration area Aa (step S105; No), it continues to perform the determination in step S105. In this case, if the possibility of the robot 50 entering the deterioration area Aa decreases, such as by moving away from the deterioration area Aa, the processing may be terminated.

[0078] Subsequently, the information processing device 1 determines whether the robot 50 has evacuated from the degraded area Aa (step S107). If it has evacuated (step S107; Yes), it instructs the observation robot to end the observation (step S108) and terminates the process.

[0079] Furthermore, if the information processing device 1 determines in step S107 that the robot has not been evacuated (step S107; No), it continues to perform the determination in step S107. Also, if the information processing device 1 determines in step S101 that there are no robots 50 that are scheduled to enter the degraded area Aa (step S101; No), it terminates the process.

[0080] Next, the control flow of the robot 50 by the information processing device 1 will be explained using Figure 8. As shown in Figure 8, the information processing device 1 receives observation information of the control robot from the control robot to be controlled or from the observation robot (step S111).

[0081] Next, the information processing device 1 generates control instructions for the control robot based on the observation information (step S112). Then, the information processing device 1 transmits control information, including the control instructions, to the control robot (step S113), and terminates the process.

[0082] Next, the processing flow by the robot 50 will be explained using Figures 9 and 10. Figures 9 and 10 are flowcharts illustrating the processing procedure of the robot 50 according to this embodiment. Figure 9 describes the processing procedure when the robot 50 is a control robot, and Figure 10 describes the processing procedure when the robot 50 is an observation robot.

[0083] As shown in Figure 9, the control robot, robot 50, first generates observation information (step S201) and transmits the observation information to the information processing device 1 (step S202). Subsequently, the control robot, robot 50, receives a control instruction from the information processing device 1 (step S203), operates according to the control instruction (step S204), and then terminates the process.

[0084] Next, using Figure 10, the processing procedure when robot 50 is an observation robot will be explained. Robot 50, which is an observation robot, generates observation information by observing the control robot (step S211) and transmits the observation information to the information processing device 1 (step S212).

[0085] Next, the observation robot 50 determines whether or not it has received an instruction from the information processing device 1 to end the observation (step S213). If the observation robot 50 has not received an instruction to end the observation (step S213; No), it repeatedly executes the processes from step S211 onwards.

[0086] Furthermore, if the robot 50, which is an observation robot, receives an instruction from the information processing device 1 to end the observation (step S213; Yes), it will terminate its processing as an observation robot.

[0087] [5. Effects] As described above, the information processing device 1 includes a receiving unit 41 that receives observation information of the control robot 50 which is the robot to be controlled, a robot control unit 42 that issues control instructions to the control robot based on the observation information received by the receiving unit 41, and a switching unit 44 that, in the event of a deterioration in the communication quality with the control robot, switches the observation information used by the robot control unit 42 for controlling the control robot from observation information observed by the control robot to observation information observed by the observation robot 50 which is the robot capable of observing the control robot.

[0088] Furthermore, the switching unit 44 switches the observation information that the robot control unit 42 uses to control the robot when the control robot passes through a degraded area in the communication quality prediction map, which predicts the communication quality of each area, where it is predicted that the communication quality with the control robot will deteriorate.

[0089] Furthermore, the switching unit 44 instructs the observation robot to wait for the control robot to begin observation when it is predicted that the control robot will pass through a degraded area.

[0090] Furthermore, the switching unit 44 positions the control robot in a location where it can be observed by the observation robot when it is anticipated that the control robot will pass through a degraded area.

[0091] Furthermore, the switching unit 44 allows the observation robot to observe the control robot from a stable area where the communication quality with the observation robot is stable.

[0092] Furthermore, the switching unit 44 switches a portion of the observation information that the robot control unit 42 uses to control the control robot from observation information that has been observed by the observation robot to observation information that has been observed by the control robot.

[0093] Furthermore, the observation robot is the control robot, which is the observation robot 50.

[0094] Furthermore, the robot control unit 42 generates control instructions for the observed information using a multimodal large-scale language model.

[0095] Furthermore, the information processing system S according to the embodiment is an information processing system S that includes a plurality of robots 50 and an information processing device 1 for remotely controlling the robots, wherein the information processing device 1 includes a receiving unit 41 that receives observation information of the control robot, which is the robot to be controlled, a robot control unit 42 that gives control instructions to the control robot based on the observation information received by the receiving unit 41, and a switching unit 44 that, in the event that the communication quality with the control robot deteriorates, switches the observation information that the robot control unit 42 uses to control the control robot from observation information observed by the control robot to observation information observed by an observation robot, which is a robot capable of observing the control robot.

[0096] Through any or a combination of the above-described processes, the information processing device according to the present application can appropriately control a robot even under conditions where communication quality deteriorates. The technology of this disclosure enables appropriate remote control of a robot even under conditions where communication delays occur, and since it forms an innovative technological foundation for remotely controlled robots, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0097] [6. Hardware Configuration] The information processing device 1 and robot 50 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 11. The following explanation will use the information processing device 1 as an example. Figure 11 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0098] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0099] The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various calculations, such as RAM (Random Access Memory) or VRAM (Video Random Access Memory). The secondary storage device 1050 is a storage device in which data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), a file server, etc.

[0100] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010 that output various types of information, such as displays, projectors, and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners. It is implemented using, for example, USB.

[0101] Furthermore, the output interface 1060 and the input interface 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. In other words, the output device 1010 and the input device 1020 may be wireless devices.

[0102] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0103] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0104] The network interface 1080 receives data from other devices via the network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via the network N.

[0105] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0106] For example, when computer 1000 functions as information processing device 1, the arithmetic unit 1030 of computer 1000 realizes the functions of control unit 4 by executing a program loaded onto primary storage device 1040. Alternatively, computer 1000's arithmetic unit 1030 may load a program obtained from another device via network interface 1080 onto primary storage device 1040 and execute the loaded program. Furthermore, computer 1000's arithmetic unit 1030 may cooperate with other devices via network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0107] [7. Others] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0108] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0109] For example, the information processing device 1 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via API (Application Programming Interface) or network computing, so the configuration can be flexibly changed.

[0110] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0111] 1 Information Processing Device 2, 52 Communication Unit 3, 55 Storage Unit 4, 47 Control Unit 31 Multimodal LLM 32 Communication Quality Prediction Map 41 Receiving Unit 42 Robot Control Unit 43 Generation Unit 44 Switching Unit 45 Transmitting Unit 49 Motion Control Unit 50 Robot 53 Sensor Unit 54 Drive Unit 58 Acquisition Unit S Information Processing System

Claims

1. An information processing device comprising: a receiving unit that receives observation information of a control robot, which is a robot to be controlled; a robot control unit that issues control instructions to the control robot based on the observation information received by the receiving unit; and a switching unit that, when the communication quality with the control robot deteriorates, switches the observation information used by the robot control unit for controlling the control robot from observation information observed by the control robot to observation information observed by an observation robot, which is a robot capable of observing the control robot.

2. The information processing apparatus according to claim 1, wherein the switching unit switches the observation information used by the robot control unit to control the control robot when the control robot passes through a degraded area in a communication quality prediction map that predicts the communication quality of each area, where it is predicted that the communication quality with the control robot will deteriorate.

3. The information processing device according to claim 2, wherein the switching unit causes the observation robot to wait for the control robot to perform observation when it is predicted that the control robot will pass through the deterioration area.

4. The information processing device according to claim 3, wherein the switching unit causes the control robot to wait in a position where it can be observed by the observation robot when it is predicted that the control robot will pass through the deterioration area.

5. The information processing device according to claim 4, wherein the switching unit causes the observation robot to observe the control robot from a stable area where the communication quality with the observation robot is stable.

6. The information processing device according to claim 1, wherein the switching unit switches a portion of the observation information used by the robot control unit for controlling the control robot from observation information of the observation robot to observation information of the control robot.

7. The information processing device according to claim 1, wherein the observation robot is the control robot for observation.

8. The information processing device according to claim 1, wherein the robot control unit generates the control instructions for the observation information using a multimodal large-scale language model.

9. An information processing system comprising a plurality of robots and an information processing device for remotely controlling the robots, wherein the information processing device includes: a receiving unit for receiving observation information of a control robot which is the robot to be controlled; a robot control unit for issuing control instructions to the control robot based on the observation information received by the receiving unit; and a switching unit for switching the observation information used by the robot control unit for controlling the control robot from observation information observed by the control robot to observation information observed by an observation robot which is the robot capable of observing the control robot, when the communication quality with the control robot deteriorates.

10. An information processing method executed by an information processing device, comprising: a receiving step of receiving observation information of a control robot which is a robot to be controlled; a robot control step of giving control instructions to the control robot based on the observation information received by the receiving step; and a switching step, when the communication quality with the control robot deteriorates, of switching the observation information used by the robot control step for controlling the control robot from observation information observed by the control robot to observation information observed by an observation robot which is a robot capable of observing the control robot.

11. An information processing program that causes a computer to execute a receiving procedure for receiving observation information of a control robot, which is a robot to be controlled; a robot control procedure for issuing control instructions to the control robot based on the observation information received by the receiving procedure; and a switching procedure, in the event that the communication quality with the control robot deteriorates, to switch the observation information used by the robot control procedure for controlling the control robot from observation information observed by the control robot to observation information observed by an observation robot, which is a robot capable of observing the control robot.

Images