Information processing method, information processing device, and program

Abstract

An information processing method according to the present disclosure is executed by an information processing system that supports remote control by an operator with respect to at least one autonomous mobile body. In the information processing method, when an operation instruction to start remote monitoring is acquired, the operator side of the information processing system performs the following: transitioning from a browsing mode in which the operator does not have control responsibility to a monitoring mode in which the operator has control responsibility; validating input of an operation instruction for a remote operation for remotely controlling a target mobile body, that is to be remotely monitored, from among the at least one autonomous mobile body; and when an operation instruction to end remote monitoring is acquired, transitioning from the monitoring mode to the browsing mode and invalidating input of the operation instruction for remote operation.

Description

Information Processing Method, Information Processing Apparatus, and Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program.

[0002] In recent years, in order to solve the labor shortage, the introduction of autonomous driving mobility has been promoted in various places such as roads, factories, airports, and ports. Among these, the realization of completely unmanned operation without human intervention is very difficult from the perspective of safety, and human intervention from a distance is required when hazards or abnormalities occur.

[0003] For example, in Patent Document 1, a technique is disclosed that aims to enable appropriate operation of a plurality of remotely monitored moving bodies, such as a bus operated by an autonomous driving technology.

[0004] Japanese Patent No. 6980609

[0005] However, in the remote control of autonomous driving mobility, when an operator gives an operation instruction remotely, for example, if an unintended instruction is given due to a system defect, processing delay, or communication delay, there is a risk of a decrease in safety.

[0006] The present disclosure has been made in view of the above, and one of its objectives is to improve the safety of remote control for autonomous driving mobility.

[0007] The information processing method according to the present disclosure is an information processing method executed by an information processing system that supports remote control by an operator for at least one autonomous moving body. In the information processing method, when the operator side of the information processing system acquires an operation instruction to start remote monitoring, it causes a transition from a browsing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, and enables the input of an operation instruction for remote control to remotely control the target moving body that is the target of the remote monitoring among the at least one autonomous moving body. When an operation instruction to end the remote monitoring is acquired, it causes a transition from the monitoring mode to the browsing mode and invalidates the input of the operation instruction for the remote control.

[0008] This disclosure can improve the safety of remote control of autonomous mobility. The effects described herein are not necessarily limited to those described herein and may include any of the effects described herein.

[0009] Figure 1 is a diagram showing an example of the schematic configuration of a remote control system according to the embodiment. Figure 2 is a diagram showing an example of the configuration of each device included in the remote control system according to the embodiment. Figure 3 is a diagram showing an example of the hardware configuration of an information processing device that realizes each function of each device included in the remote control system according to the embodiment. Figure 4 is a flowchart showing an example of the flow of operation initiation of remote monitoring in remote control according to the embodiment. Figure 5 is a flowchart showing an example of the flow of operation in remote monitoring in remote control according to the embodiment. Figure 6 is a flowchart showing an example of the flow of operation in termination of remote monitoring in remote control according to the embodiment. Figure 7 is a flowchart showing an example of the processing flow during automatic driving in remote control according to the embodiment. Figure 8 is a diagram showing an example of the screen display of an operator terminal in remote control according to the embodiment. Figure 9 is a diagram showing another example of the screen display of an operator terminal in remote control according to the embodiment. Figure 10 is a sequence diagram showing an example of the information processing flow at the start of remote monitoring executed by each part of the remote control system according to the embodiment. Figure 11 is a sequence diagram showing an example of the information processing flow regarding communication delay monitoring when remote monitoring is performed, executed by each part of the remote control system according to the embodiment. Figure 12 is a sequence diagram showing an example of the information processing flow related to operator operation instructions during remote monitoring, executed by each part of the remote control system according to the embodiment. Figure 13 is a sequence diagram showing an example of the information processing flow at the end of remote monitoring, executed by each part of the remote control system according to the embodiment. Figure 14 is a sequence diagram showing another example of the information processing flow at the start of remote monitoring, executed by each part of the remote control system according to the embodiment. Figure 15 is a sequence diagram showing another example of the information processing flow at the start of remote monitoring, executed by each part of the remote control system according to the embodiment. Figure 16 is a diagram showing another example of the screen display of the operator terminal in remote control according to the embodiment.

[0010] Hereinafter, embodiments of the information processing method, information processing apparatus, information processing system, mobile device, program, and recording medium related to this disclosure will be described in detail with reference to the attached drawings.

[0011] In this disclosure, components having the same or substantially the same function as those described above in previously shown drawings are denoted by the same reference numerals, and explanations may be omitted as appropriate. Furthermore, even when representing the same or substantially the same parts, the dimensions and proportions may be shown differently in different drawings. In addition, for example, from the viewpoint of ensuring the readability of the drawings, reference numerals may be assigned only to the main components in the explanation of each drawing, and reference numerals may not be assigned to components having the same or substantially the same function as those described above in previously shown drawings.

[0012] In addition, in the descriptions of this disclosure, components having the same or substantially the same function may be distinguished by adding alphanumeric characters and / or symbols to the end of the reference numeral. Alternatively, if multiple components having the same or substantially the same function are not to be distinguished, they may be described together by omitting the alphanumeric characters and / or symbols to the end of the reference numeral.

[0013] In this disclosure, “remote control” is classified into three modes: “remote viewing,” “remote assistance,” and “remote control,” each of which differs in responsibility, possible operations, and safety function activation conditions depending on the form of control over the target mobile object. Here, “remote control” refers to at least one of “remote viewing,” “remote assistance,” and “remote control” over at least one mobile object, each configured to move autonomously and perform a predetermined task.

[0014] In this disclosure, "remote viewing" refers to a mode of remote control in which mobile information relating to at least one target mobile object and its surrounding environment is presented to an operator in a remotely viewable (verifiable) manner, or in which the operator views (verifies) this information. This mobile information includes camera images from cameras mounted on the mobile object, information from various sensors mounted on the mobile object, sensing information generated by processing such information, and / or information representing the state of the mobile object itself, such as vehicle speed. An operator is a person who remotely monitors a mobile object such as a vehicle and remotely controls the mobile object as needed. Here, "remote viewing" is an example of a mode of remote control in which a mobile object is remotely monitored by transmitting mobile information such as images relating to at least one target mobile object.

[0015] Furthermore, in this disclosure, "remote assistance" means, in addition to "remote viewing," remotely instructing (remotely controlling) the control of at least one target mobile body by transmitting a control signal (driving instruction signal) in response to an operation instruction to that target mobile body. This operation instruction is when an operator remotely instructs the operation of an autonomously driven mobile body, such as starting or stopping the automatic driving of the mobile body, but may also include instructions related to MRM (Minimal Risk Maneuver), including emergency stops, and operations unrelated to driving. MRM will be described later. Here, "remote assistance" is an example of a first transmission mode in which the target mobile body is controlled by a control signal transmitted as a single instruction request.

[0016] Furthermore, in this disclosure, "remote control" means remotely controlling (operating remotely) a target mobile body by transmitting a control signal (operation signal) in response to an operation instruction to that target mobile body among at least one mobile body. Here, "remote control" is an example of a second transmission mode in which the target mobile body is controlled by control signals transmitted as a series of instruction requests.

[0017] The UI (User Interface) display includes the display of mobile object information and surrounding environment information in each of the remote control modes: "remote viewing," "remote assistance," and "remote control."

[0018] Operator control responsibilities are "none" in the "remote viewing" remote control mode, but "present" in the "remote assistance" and "remote control" remote control modes.

[0019] The operations that an operator can perform (remote control) are "none" in the "remote viewing" remote control mode, but "yes" in the "remote assistance" and "remote control" remote control modes. For example, in the "remote assistance" remote control mode, the operations that an operator can perform include sending commands for safe stopping of the mobile body, automatic stopping, automatic restart, and route changes. For example, in the "remote control" remote control mode, the operations that an operator can perform include longitudinal control such as operating the accelerator and brakes of the mobile body, and lateral control such as steering operations (e.g., steering).

[0020] The simultaneous control constraint is "none" in the "remote viewing" remote control mode, but "present" in the "remote assistance" and "remote control" remote control modes. For example, the simultaneous control constraint in the "remote assistance" and "remote control" remote control modes is that only one operator can operate a vehicle in either "remote assistance" or "remote control" mode.

[0021] The conditions for activating safety features are "none" in the "remote viewing" remote control mode, while they are "present" in the "remote assistance" and "remote control" remote control modes. For example, the conditions for activating safety features in the "remote assistance" and "remote control" remote control modes are that the video and vehicle information transmission and display functions are operating correctly without delay, and / or that instruction commands are transmitted and executed correctly without delay.

[0022] Thus, in the “remote control” of this disclosure, “remote assistance” and “remote control” allow for a reduction in the level of autonomous driving of the moving object, in exchange for the operator assuming control responsibility. In other words, during “remote assistance” and “remote control” in the “remote control” of this disclosure, the remote system ensures that the operator can fulfill control responsibility.

[0023] In this disclosure, the remote control mode of "remote viewing" may also be simply referred to as "viewing mode." Furthermore, in this disclosure, the remote control modes of "remote assistance" and / or "remote control" may be combined and referred to as "remote monitoring" or "monitoring mode." Therefore, "remote monitoring" means that an operator remotely monitors the status of a mobile object using mobile object information (video and various other information) and, if necessary, remotely controls the target mobile object by giving remote operation instructions. In other words, in this disclosure, "viewing mode" is a remote control mode in which the operator does not assume control responsibility, and "monitoring mode" is a remote control mode in which the operator assumes control responsibility.

[0024] In this disclosure, the operation in "remote viewing," "remote assistance," and "remote control" modes may also be described as "(remote) viewing state," "(remote) assistance state," and "(remote) control state," respectively. Furthermore, the operation in "monitoring mode" may also be described as "(remote) monitoring state."

[0025] (Example of Remote Control System Configuration) Figure 1 is a diagram showing an example of the schematic configuration of a remote control system 1 according to an embodiment. As shown in Figure 1, the remote control system 1 includes a front end 2, a back end 3, and an edge 4.

[0026] In the remote control system 1, the backend 3 is connected to multiple frontends 2 and multiple edge devices 4. In the remote control system 1, multiple remote operators simultaneously control multiple vehicles 46 (mobile devices).

[0027] Front end 2 forms the remote operator side of the remote control system 1. Front end 2 includes an operator terminal 20 and a controller 25. The operator terminal 20 and the controller 25 may be configured as a single unit.

[0028] The backend 3 relays the transmission and reception of information between the corresponding frontend 2 and edge 4 pairs. For example, the backend 3 receives video and vehicle information transmitted from edge 4 and transmits the received video and vehicle information to the corresponding frontend 2. For example, the backend 3 receives various control signals, including mode switching and driving instructions, transmitted from frontend 2 and transmits the received control signals to the corresponding edge 4. The backend 3 includes a server 30. Note that the backend 3 may be realized through the cooperation of multiple servers 30. Here, vehicle information is an example of information related to a mobile object.

[0029] Edge 4 forms the side of the system that is remotely controlled by the remote control system 1. Edge 4 includes an edge device 40, a camera 45, and a vehicle 46. At least two of the edge device 40, camera 45, and vehicle 46 may be integrated into a single unit. For example, camera 45 may include an on-board camera in vehicle 46. For example, edge device 40 may be implemented by a computer mounted in vehicle 46.

[0030] The front-end 2, back-end 3, and edge 4 operate in cooperation via communication, but strictly speaking, they are asynchronous. For example, the front-end 2 and back-end 3 are connected to each other in a way that allows them to communicate with one another via any telecommunications line, such as a LAN (Local Area Network). Note that communication between the front-end 2 and back-end 3 may be achieved via the Internet. The back-end 3 and edge 4 are connected to each other in a way that allows them to communicate with one another via any telecommunications line, such as the Internet.

[0031] Here, the remote control system 1 according to this embodiment is an information processing system (remotely operated vehicle operation system) that performs remote control of an autonomously roaming vehicle 46 (autonomous mobile body) by remotely viewing, assisting, and / or operating it through an operator. The remote control system 1 provides various services through its remote control, such as delivery, security, cleaning, childcare, nursing care, sales, agricultural work, manufacturing, cargo handling, transportation, and construction. In this remote control system 1, an operator is assigned in response to a request for support (remote request) from a vehicle 46 that requires remote control, and the assigned operator intervenes in the control of the vehicle 46, such as remote assistance or remote operation. The server 30 according to this embodiment is an information processing device (remote control support device) that supports the remote control of the vehicle 46. The operator terminal 20 according to this embodiment is an information processing device (terminal device) operated by an operator in a remote control room installed in a control center or the like. The vehicle 46 according to this embodiment is an example of an autonomously roaming mobile body and is used to provide various services.

[0032] As an example, the remote control system 1 according to this embodiment can be constructed by applying edge computing. In this case, for example, edge 4 is used as the network periphery (edge), but other devices may be used as the edge.

[0033] For example, vehicle 46 is an example of a mobile body that performs various tasks, including autonomous driving, related to various services provided by the remote control system 1, such as delivery, security, cleaning, childcare, nursing care, sales, agricultural work, manufacturing, cargo handling, transportation, and construction. For example, vehicle 46 is a mobile body configured to move autonomously and perform predetermined tasks. For example, vehicle 46 is a mobile body configured to move according to the remote control of an operator monitoring multiple vehicles 46 and perform predetermined tasks.

[0034] Furthermore, the mobile device is not limited to a vehicle; various types of mobile devices configured to be movable in response to remote operation by an operator can be used as appropriate. The mobile device may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Alternatively, the mobile device may be an Automated Guided Vehicle (AGV), or various robots such as construction machinery, agricultural machinery, or drones. Moreover, these mobile devices are not limited to transporting people; they may transport objects other than people, or they may provide specific services in addition to transportation.

[0035] For example, if vehicle 46 becomes unable to autonomously drive, for instance, when it detects an obstacle in its path, such as a fallen object or a parked vehicle, it sends a remote control support request (remote request) to server 30. This support request may be an assistance request requesting remote assistance, or a control request requesting remote operation. For example, server 30 transmits images captured by a camera mounted on vehicle 46 to the corresponding operator terminal 20. For example, when server 30 receives a support request from vehicle 46, it sends a remote operation request to operator terminal 20 requesting remote assistance or remote operation (remote control) from the operator. For example, operator terminal 20 displays a display screen including images (video) captured by a camera 45, such as a camera mounted on vehicle 46. The display screen includes at least one image (video) relating to at least one vehicle 46 that is operated by the operator operating operator terminal 20. The operator monitors the status of at least one vehicle 46 assigned to them while viewing the display screen of the operator terminal 20. This monitoring is an example of remote viewing of multiple vehicles 46 by an operator based on video data transmitted from each of the multiple vehicles 46. For example, when the operator terminal 20 receives a remote operation request (remote request) from the server 30, it notifies the operator operating the terminal that remote operation assistance has been requested. The operator operates the controller 25 connected to the operator terminal 20 while viewing the display screen and provides support such as moving the vehicle 46 by remotely assisting or remotely controlling the vehicle 46 that requested assistance. This remote control assistance of the vehicle 46 is an example of remote operation by an operator based on video data transmitted from the vehicle 46 that is the target of remote operation among multiple vehicles 46.

[0036] The display screen of the operator terminal 20 may be a screen or image generated by the operator terminal 20 based on display information from the server 30, or it may display an image (display information) generated by the server 30.

[0037] Thus, the remote control system 1 according to this embodiment is configured to be able to execute an information processing method (remote control support method) that supports remote control, which is performed based on images for remote control captured by the camera 45, and involves an operator remotely viewing, assisting with, and / or controlling a vehicle 46.

[0038] Figure 2 shows an example of the configuration of each device included in the remote control system 1 according to the embodiment. Although Figure 2 illustrates one operator terminal 20 and one edge device 40, the configurations of other operator terminals 20 and edge devices 40 included in the remote control system 1 are similar. The configurations of the operator terminal 20, server 30, and edge device 40 will be described below with reference to Figure 2.

[0039] (Example of operator terminal configuration) The operator terminal 20 has a communication unit 201, a calculation unit 202, a storage unit 203, and an operation input unit 204.

[0040] The communications unit 201 communicates with the server 30.

[0041] For example, the communication unit 201 may request vehicle information such as camera images from the server 30, and may receive vehicle information transmitted from the vehicle 46 at predetermined intervals via the server 30. Transmission information may be attached to this vehicle information at the edge 4. In other words, the communication unit 201 may receive transmission information attached to the vehicle information.

[0042] As an example, the communication unit 201 may transmit a control signal corresponding to an operator's operation input to the controller 25 to the server 30. This control signal may be a mode switching signal corresponding to the input of the operator's mode switching instruction. Further, the control signal may be a travel instruction signal (automatic travel start signal or automatic travel stop signal) corresponding to the operation instruction (input) of the operator's remote assistance. Further, the control signal may be a steering signal corresponding to the operation instruction (input) of the operator's remote control. Further, the control signal may be an emergency stop signal corresponding to the input of the operator's emergency stop instruction (operation instruction). This operator's emergency stop instruction may be an operation instruction for instructing the execution of MRM.

[0043] As an example, the communication unit 201 may periodically return the transmission information attached to the latest vehicle information to the edge 4. Further, the transmission information of the latest vehicle information may be attached to the control signal corresponding to the operator's operation instruction. That is, the communication unit 201 returns the transmission information of the latest vehicle information attached to the control signal corresponding to the operator's operation instruction to the edge 4. Here, the transmission information is, for example, the transmission time from the edge 4, but may be the information acquisition time from various sensors including the camera 45, may be a transmission counter, or may be an identifier of the information. This identifier refers to something that can uniquely identify the transmission information, such as a random character string.

[0044] The arithmetic unit 202 comprehensively controls the operation of the operator terminal 20.

[0045] As an example, the arithmetic unit 202 may monitor whether a plurality of software applications for remote control are not activated in the operator terminal 20, and may prohibit the activation of the second application.

[0046] For example, the calculation unit 202 may control the screen display on the operator terminal 20's display. This screen display may include, for example, the display of received vehicle information (mobile information). Also, for example, the screen display may include operation buttons for inputting operator operation instructions. These operation buttons may include, for example, at least one remote monitoring button for inputting operation instructions from the operator to start and end remote monitoring of a vehicle 46 among a plurality of vehicles 46 that is the target of remote monitoring. For example, the operation buttons may include at least one remote operation button for inputting operation instructions for remotely controlling the target vehicle 46. These operation buttons are examples of operation buttons on the controller 25. Also, for example, the screen display may include operator information indicating the operator remotely monitoring each vehicle 46. This operator remotely monitoring each vehicle 46 is the operator whose operation is acquired by the operator terminal 20 (monitoring front end) that has transitioned to monitoring mode.

[0047] As an example, the calculation unit 202 may switch the operation buttons of the controller 25 on or off. Here, enabling / disabling the operation buttons of the controller 25 may mean enabling / disabling the input of operation instructions represented by the operation of the operation button. Note that enabling / disabling the input of operation instructions may mean making it possible / impossible to input operation instructions physically or in software, or allowing / denying input operation instructions. For example, in monitoring mode, the calculation unit 202 may enable the remote operation buttons related to the vehicle 46 that is the target of remote monitoring. For example, in viewing mode, the calculation unit 202 may disable the remote operation buttons that are enabled in remote mode. For example, in monitoring mode, the calculation unit 202 may prohibit other operators from operating the remote monitoring button that inputs an operation instruction to end remote monitoring for the vehicle 46 that is the target of remote monitoring, and operations on the remote operation buttons. For example, the calculation unit 202 may disable the remote monitoring button that inputs an operation instruction to start remote monitoring for the vehicle 46 that is the target of remote monitoring by another operator terminal 20.

[0048] As an example, the arithmetic unit 202 may monitor whether vehicle information of the vehicle 46, which is the target moving object for which remote monitoring is to be performed, can be received. For example, when the vehicle information can be received normally, the arithmetic unit 202 may determine whether mode switching is possible based on the reception status of the vehicle information, such as permitting switching to the monitoring mode.

[0049] As an example, the arithmetic unit 202 may hold information indicating whether the edge 4 is in the remote assistance state or the remote control state, for example, by the storage unit 203, and determine whether a control signal such as a travel instruction signal or a control signal can be transmitted based on the held information.

[0050] As an example, the arithmetic unit 202 may switch the mode of remote control according to an operation instruction by an operator. For example, when the arithmetic unit 202 acquires an operation instruction for starting remote monitoring by the operator, the arithmetic unit 202 may transition the mode of remote control from the browsing mode in which the operator has no control responsibility to the monitoring mode in which the operator has control responsibility. For example, when the arithmetic unit 202 acquires an operation instruction for ending remote monitoring by the operator, the arithmetic unit 202 may transition the mode of remote control from the monitoring mode to the browsing mode.

[0051] As an example, the arithmetic unit 202 may add transmission information of the latest vehicle information received from the edge 4 to a control signal according to an operation instruction of the operator.

[0052] As an example, the calculation unit 202 may perform tasks such as verifying the integrity of vehicle information, monitoring delays, monitoring disconnections of the controller 25, and monitoring whether multiple controllers 25 are connected. For example, the calculation unit 202 may perform MRM, including an emergency stop, if it detects that the vehicle information is irregular or has a delay exceeding a specified value, or if it detects an abnormality in the controller connection. Here, irregular vehicle information includes the received vehicle information and / or the attached transmission information not being in the specified format, or not being received at all. Performing MRM is an example of restricting the movement of the target vehicle 46 (movement of the moving object), for example, by having the vehicle 46 perform MRM. This MRM is a behavior that automatically transitions the autonomous vehicle to a safe state, including an emergency stop on the spot, and is an example of a pre-configured measure that automatically transitions the target moving object to a safe state. Note that restricting movement may mean stopping the vehicle (not allowing it to move) by performing MRM including an emergency stop, limiting its speed, or restricting the execution of other various functions related to movement.

[0053] The memory unit 203 stores various information related to the operation of the operator terminal 20.

[0054] The operation input unit 204 receives signals from the controller 25. For example, the operation input unit 204 may receive an instruction from the operator to start driving the vehicle 46. For example, the operation input unit 204 may receive an instruction from the operator to stop driving the vehicle 46. For example, the operation input unit 204 may receive an operation instruction from the operator to remotely control the vehicle 46 that is the subject of remote monitoring. For example, the operation input unit 204 may receive an operation instruction from the operator to perform an emergency stop on the vehicle 46 that is the subject of remote monitoring. This emergency stop operation instruction from the operator may also be an operation instruction to execute MRM.

[0055] (Example of controller configuration) Controller 25 is an input device connected to the operator terminal 20. Controller 25 receives operations from the operator. Specifically, Controller 25 generates an operation signal corresponding to the operator's operation and outputs the generated operation signal to the operator terminal 20.

[0056] For example, the controller 25 may be composed of dedicated operating components such as a steering wheel, accelerator pedal, and brake pedal. Alternatively, the controller 25 may be composed of a touch panel (touch panel display) provided on the display of the operator terminal 20. Furthermore, the controller 25 may be composed of a combination of these.

[0057] For example, the controller 25 may include operation buttons for inputting operator operation instructions. These operation buttons may include, for example, at least one remote monitoring button for inputting operation instructions by the operator to start and end remote monitoring of a vehicle 46 among a plurality of vehicles 46 that is the target of remote monitoring. For example, the operation buttons may include at least one remote control button for inputting operation instructions for remote control of the target vehicle 46. For example, the controller 25 may have a remote monitoring start button for inputting an instruction to start remote monitoring for which the operator assumes control responsibility. For example, the controller 25 may have a remote monitoring end button for inputting an instruction to end remote monitoring. For example, the controller 25 may have a drive start instruction button for the operator to input a drive start instruction for the vehicle 46. For example, the controller 25 may have a drive stop instruction button for the operator to input a drive stop instruction for the vehicle 46. The controller 25 may have an emergency stop instruction button for the operator to input an emergency stop instruction for the vehicle 46. This emergency stop instruction from the operator may be an operation instruction to execute MRM.

[0058] (Example of server configuration) As shown in Figure 2, the server 30 has a communication unit 301, a calculation unit 302, and a storage unit 303.

[0059] The communication unit 301 communicates with the operator terminal 20 and the edge device 40, respectively. For example, the communication unit 301 may transmit vehicle information (such as video) of the vehicle 46 sent from the edge device 40 to the operator terminal 20, which is remotely controlling remote viewing, remote assistance, and remote operation. For example, the communication unit 301 may transmit control signals from the operator terminal 20 to the corresponding edge device 40.

[0060] The arithmetic unit 302 comprehensively controls the operation of the server 30. For example, the arithmetic unit 302 may monitor whether multiple operator terminals 20 are sending control signals to a single vehicle 46. For example, the operator terminals 20 (monitoring front end) that transition to monitoring mode for a single vehicle 46 may be limited to just one operator terminal 20. For example, the arithmetic unit 302 may process the signals that were started first to be sent to the target vehicle 46.

[0061] The memory unit 303 stores various information related to the operation of the server 30.

[0062] (Example of edge device configuration) As shown in Figure 2, the edge device 40 includes a communication unit 401, a calculation unit 402, a storage unit 403, a vehicle information input unit 404, and a vehicle control unit 405.

[0063] The communication unit 401 communicates with the server 30. For example, the communication unit 401 may transmit vehicle information with transmission information attached to it to the operator terminal 20 via the server 30 at predetermined intervals. For example, the communication unit 401 may receive transmission information returned from the operator terminal 20, or control signals returned from the operator terminal 20 with transmission information attached to them.

[0064] The arithmetic unit 402 comprehensively controls the operation of the edge device 40.

[0065] As an example, the calculation unit 402 may add transmission information such as the acquisition time to the vehicle information from the camera 45 and / or the vehicle 46.

[0066] As an example, the arithmetic unit 402 may perform integrity verification and delay monitoring of control signals from the server 30. For example, the arithmetic unit 402 may calculate a round-trip delay time, including the operator's response time, using the transmission and reception times of the transmitted information. Here, the operator's response time included in the round-trip delay time may be the time from when vehicle information is presented to the operator until the operator's operation instructions based on the vehicle information are executed. In other words, the round-trip delay time may be the time from when "vehicle information is acquired from the camera 45 and / or vehicle 46" until the operator's operation instructions based on that vehicle information are executed "in the vehicle 46". Alternatively, the round-trip delay time may be the time from when "vehicle information is transmitted from the edge device 40" until the operator's operation instructions based on that vehicle information are "instructed from the edge device 40 to the vehicle 46". For example, the arithmetic unit 402 may calculate an estimated round-trip delay time, including an estimated response time, using the transmission and reception times of the transmitted information. Here, the estimated response time may be the time from when vehicle information is presented to the operator until the operator's operation instructions based on the vehicle information are executed. This estimated response time may be predetermined for each operator and stored in the storage unit 403.

[0067] For example, if the calculated round-trip delay time is within a predetermined time, the calculation unit 402 may execute control of the vehicle 46 by the vehicle control unit 405 in accordance with the control signal from the operator terminal 20 in the corresponding monitoring mode. This predetermined time for the round-trip delay time may be predetermined and stored in the storage unit 403. This predetermined time may vary depending on the type of support request, the operator or the operator's experience, the type of vehicle 46, the state of the vehicle 46, the vehicle speed, the location, or the surrounding environment. The state of the vehicle 46 may be the occurrence or absence of malfunctions or errors, the level of autonomous driving (Lv2, 3, 4), etc. The surrounding environment may be the type, number, positional relationship, and speed relationship of objects around the vehicle 46, etc.

[0068] As an example, if the information received from the server 30 is irregular or delayed by more than a specified value, the calculation unit 402 may perform MRM, including emergency stopping of the vehicle 46. For example, if the calculated round-trip delay time exceeds a predetermined time, the calculation unit 402 may not permit the control of the vehicle 46 that follows the control signal from the operator terminal 20 in the corresponding monitoring mode. In this case, the calculation unit 402 may also perform MRM, including emergency stopping of the vehicle 46, for the vehicle 46 in question. For example, if the calculated estimated round-trip delay time exceeds a predetermined time, the calculation unit 402 may perform MRM, including emergency stopping of the vehicle 46, for the vehicle 46 in question. This predetermined time for the estimated round-trip delay time may be predetermined and stored in the storage unit 403, for example. This predetermined time may vary depending on the type of support request, the operator, or the operator's experience. For example, the calculation unit 402 does not have to perform MRM, including emergency stopping of the vehicle 46, for the vehicle 46 in question based on the estimated round-trip delay time for other operator terminals 20 in the corresponding monitoring mode. Not performing an MRM including an emergency stop based on this estimated round-trip delay time means ignoring information from other front-end 2 in the calculation, not calculating the estimated round-trip delay time, or calculating it and then not performing an MRM including an emergency stop. In this disclosure, an emergency stop is mainly used as an example of an MRM, but other safe evacuation actions besides an emergency stop are also acceptable.

[0069] For example, the calculation unit 402 may monitor whether multiple operator terminals 20 are transmitting control signals to a single vehicle 46. For example, the operator terminal 20 (monitoring front end) that transitions to monitoring mode for a single vehicle 46 may be limited to just one operator terminal 20. For example, the calculation unit 402 may process the operator terminal 20 that started receiving signals first as the operator terminal 20 that performs remote monitoring of the target vehicle 46. For example, the calculation unit 402 may reject control signals from other operator terminals 20 to the operator terminal 20 that performs remote monitoring.

[0070] For example, the calculation unit 402 may deactivate the monitoring mode if the driver directly operates on the vehicle 46 being remotely monitored, and / or directly operates on the vehicle 46's automated driving system.

[0071] For example, the calculation unit 402 may switch whether or not to send a control signal to the vehicle 46 depending on the remote control mode.

[0072] For example, the calculation unit 402 may compare the content of the instructions and control given to the vehicle 46 by the vehicle control unit 405 (instruction content) with the information from the vehicle 46 (execution content). For example, if the comparison results do not match, the calculation unit 402 may determine that there is an abnormality and perform MRM, including emergency stopping of the vehicle 46.

[0073] The memory unit 403 stores various information related to the operation of the edge device 40.

[0074] The vehicle information input unit 404 receives various vehicle information of the vehicle 46, such as camera images and vehicle body information from the camera 45 and / or the vehicle 46. This vehicle body information may include sensor information from GNSS (Global Navigation Satellite System) systems such as LiDAR (Light Detection and Ranging), radar, sonar, and GPS (Global Positioning System) attached to the vehicle body of the vehicle 46, as well as sensing information such as target information, location information, and map information processed from this sensor information.

[0075] The vehicle control unit 405 controls the operation of the vehicle 46 by transmitting control signals to the vehicle 46.

[0076] (Example of camera configuration) Camera 45 is an input device connected to the edge device 40. Camera 45 acquires images for remote control and outputs the acquired images to the edge device 40. These images may be still images or moving images (video). These images may be the same as or different from the images used for the autonomous driving of the vehicle 46.

[0077] (Example of vehicle configuration) Vehicle 46 is a mobile unit connected to edge device 40. As described above, vehicle 46 is a mobile unit that performs various tasks, including autonomous driving, related to the various services provided by the remote control system 1. As an example, vehicle 46 is a remotely operated miniature car.

[0078] Figure 3 shows an example of the hardware configuration of an information processing device 8 that realizes each function of each device included in the remote control system 1 according to the embodiment. The information processing device 8 is a computer that comprehensively controls the operation of the entire device in each device included in the remote control system 1.

[0079] The information processing device 8 that realizes each function of the vehicle 46 may be a computer such as an ECU (Electronic Control Unit) or a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit), which is an OBU.

[0080] As shown in Figure 3, the information processing device 8 includes a processor 81, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 83, and a device I / F (interface) unit 84.

[0081] The processor 81 is, for example, a CPU (Central Processing Unit), but in addition to or instead of a CPU, at least one of various processors such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array) can be used as appropriate. Here, the processor 81 according to this embodiment is an example of at least one processor in the information processing device 8.

[0082] For example, the processor 81 of the operator terminal 20 realizes the functions of the communication unit 201, arithmetic unit 202, storage unit 203, and operation input unit 204 as illustrated in Figure 2 by executing a program stored in, for example, ROM 82. For example, the processor 81 of the server 30 realizes the functions of the communication unit 301, arithmetic unit 302, and storage unit 303 as illustrated in Figure 2 by executing a program stored in, for example, ROM 82. For example, the processor 81 of the edge device 40 realizes the functions of the communication unit 401, arithmetic unit 402, storage unit 403, vehicle information input unit 404, and vehicle control unit 405 as illustrated in Figure 2 by executing a program stored in, for example, ROM 82.

[0083] In the example shown in Figure 2, only the functions necessary for explaining the main parts of the embodiment are illustrated, but the functions of each device included in the remote control system 1 are not limited to these. In the embodiment, the processor 81 executes a program stored in the ROM 42 to realize each function of each device, including the functions of each part described above. However, it is not limited to this, and some or all of these functions may be realized by dedicated hardware circuits. In addition, each device of the remote control system 1 may integrate two or more functions to realize a single function. Similarly, each device of the remote control system 1 may divide a single function to realize two or more functions. In addition, in the remote control system 1, the functions of two or more devices may be integrated to realize at least one function of any of the devices. Similarly, in the remote control system 1, the functions of one device may be divided to realize two or more functions of two or more devices.

[0084] ROM 82 is a non-volatile memory and is an auxiliary storage device that stores various types of information, including programs executed by the processor 81. The memory of the information processing device 8 is not limited to ROM 82; various recording media and devices such as HDD (Hard Disk Drive), SSD (Solid State Drive), and Flash memory can be used as appropriate. RAM 83 is a volatile memory that has a working area for the processor 81 and is the main memory. Here, ROM 82 and RAM 83 in this embodiment are examples of at least one memory in the information processing device 8. The device I / F unit 84 is an interface for connecting the information processing device 8 to other devices in the remote control system 1, such as communication devices (not shown), display devices (not shown), and input devices (not shown).

[0085] For example, the device interface 84 of the operator terminal 20 is connected to the controller 25. For example, the device interface 84 of the edge device 40 is connected to the camera 45 and the vehicle 46.

[0086] Next, examples of operation of each embodiment of the remote control system 1 configured as described above will be explained with reference to the drawings. Note that the operation procedures and processing flows described below are examples only, and the order of steps can be changed, some steps can be deleted, or other steps can be added as desired.

[0087] (Regarding assumed use cases) Here, we will describe the use cases of the remote control system 1 assumed by the embodiments of this disclosure.

[0088] Figure 4 is a flowchart illustrating an example of the flow of operations to initiate remote monitoring in a remote control system according to an embodiment. In the flow shown in Figure 4, the remote control system is initiated in the viewing mode. First, the operator selects the vehicle 46 to be remotely monitored and presses the monitoring start / end button 616 (see Figure 8) (S101). When the operator presses the monitoring start / end button 616 in viewing mode, the operator terminal 20 receives this as an operation instruction to start monitoring. The operator terminal 20 transitions the remote control system to monitoring mode in response to this monitoring start operation instruction (S102). As a result, the remote control system 1 transitions to monitoring mode. In other words, the vehicle 46 transitions to the monitored state.

[0089] Figure 5 is a flowchart showing an example of the operation flow for remote monitoring in remote control according to the embodiment. The flow in Figure 5 starts when the remote control mode is in monitoring mode. First, the operator checks the surroundings of the vehicle 46 that is the target of remote monitoring based on the vehicle information included in the screen display of the operator terminal 20 (S201). If no danger is found (S202: No), the operator continues to check the surroundings. For example, during the surroundings check, the operator checks the camera images from the front, rear, left, and right of the vehicle 46 (image 611, see Figure 8), sensor information, notifications, etc. Then, if the operator finds a dangerous event during the surroundings check (S202: Yes), the operator presses the vehicle operation instruction button 612a (see Figure 8) to input an operation instruction to stop driving into the operator terminal 20 (S203). As a result, the vehicle 46 being remotely monitored is temporarily stopped. In this way, in remote monitoring, the operator checks the safety of the moving vehicle 46 and issues a stop instruction when a dangerous event occurs. Subsequently, the operator checks the surroundings of the vehicle 46 being remotely monitored based on the vehicle information displayed on the operator terminal 20 screen (S204). If safety cannot be confirmed (S205: No), the operator continues the surroundings check. If safety is confirmed during the surroundings check (S205: Yes), the operator presses the vehicle operation instruction button 612a (see Figure 8) again to input an operation instruction to start driving into the operator terminal 20 (S206). As a result, the vehicle 46 being remotely monitored resumes driving.

[0090] Figure 6 is a flowchart showing an example of the flow of operations to terminate remote monitoring in remote control according to the embodiment. In the flow of Figure 6, the remote control mode is started in monitoring mode. First, the operator selects the vehicle 46 to which remote monitoring is to be terminated and presses the monitoring start / end button 616 (see Figure 8) (S301). The operator terminal 20 receives the monitoring start / end button 616 pressed by the operator in monitoring mode as an operation instruction to terminate monitoring. The operator terminal 20 transitions the remote control mode to viewing mode in response to the operation instruction to terminate monitoring (S302). As a result, the remote control system 1 transitions to viewing mode. In other words, the vehicle 46 transitions to the normal state (autonomous driving state). Note that for vehicles 46 that are not capable of autonomous driving in viewing mode, the transition to the normal state does not mean a transition to the autonomous driving state. This normal state may be, for example, the state immediately before the monitoring mode is started, or the state before a request for assistance (remote request) is made.

[0091] Figure 7 is a flowchart showing an example of the processing flow during automated driving in remote control according to the embodiment. Figure 7 illustrates the flow when the remote control system 1 issues an MRM execution instruction when the operator is unable to perform monitoring duties. The edge device 40 determines whether a communication error has occurred (S401). If no communication error has occurred (S401: No), the flow in Figure 7 ends. If a communication error has occurred (S401: Yes), the edge device 40 determines whether the vehicle 46 is in a dangerous location (S402). Here, a dangerous location includes, for example, a place where the vehicle 46 should not stop, such as a pedestrian crossing. If the vehicle 46 is in a dangerous location (S402: Yes), the edge device 40 causes the monitored vehicle 46 to execute an MRM (S403), and after the vehicle 46 has crossed the dangerous location (S404), it executes an MRM including an emergency stop (S405). On the other hand, if vehicle 46 is not in a dangerous location (S402: No), the edge device 40 performs MRM, including emergency stop (S405). After that, the flow shown in Figure 7 ends. Note that the flow shown in Figure 7 may also be performed by the server 30.

[0092] (Regarding screen display) Here, we will describe the screen display for remote control related to the use case envisioned by the embodiments of this disclosure.

[0093] Figure 8 shows an example of the screen display of an operator terminal in remote control according to an embodiment. The display screen 610 in Figure 8 is an example of the screen display presented to the operator of the operator terminal 20 performing remote control. The display screen 610 in Figure 8 includes at least one image 611 for remote control and a map 613 of the surrounding area of ​​the vehicle 46 that is the target of remote control. In the example of Figure 10, a front image 611a, a left image 611b, a right image 611c, and a rear image 611d are exemplified, but it is not limited to these. The image 611 in the display screen 610 may be any one of these, or it may include other images. The display screen 610 in Figure 8 also includes an operator information display 614 that displays information of the remote monitor (operator) and a vehicle status display 617. The operator information display 614 displays information that identifies the operator, such as the ID and name of the operator who remotely monitors the vehicle 46 displaying vehicle information such as the image 611. In the example in Figure 8, the operator information display 614 shows "Operator 1," indicating the operator remotely monitoring "Vehicle 1." The vehicle status display 617 shows the vehicle status of Vehicle 46, such as "Stopped" or "Automatic Driving."

[0094] Furthermore, the operator information display 614 may be displayed in a manner that corresponds to the monitoring status of the vehicle 46. For example, the operator information display 614 may be left blank if the vehicle 46 displaying vehicle information is not under monitoring, that is, if no operator is performing remote monitoring. For example, the operator information display 614 may display information indicating the operator, such as the name or ID of the operator performing remote monitoring of the vehicle 46 displaying vehicle information. For example, the display 416 may be displayed in a way that distinguishes between when the operator of the own terminal is performing monitoring and when the operator of another terminal is performing monitoring. For example, if the vehicle 46 displaying vehicle information is being remotely monitored by the operator of the own terminal, the display 416 may highlight the information indicating the operator. Highlighting may be achieved by changing the color of the text or background of the display, or by displaying a frame or icon. In addition, the display 416 may not be limited to highlighting only one side, but may also distinguish between the display color, frame, or icon between the self and the other to indicate whether the vehicle is being monitored by the self or the other. Thus, the display 416 may also be a display of the monitoring operator's name that facilitates identification of oneself and others.

[0095] Furthermore, the display screen 610 in Figure 8 includes a vehicle list display 615. The vehicle list display 615 displays information about each vehicle 46 that is subject to remote control by the remote control system 1. In the example in Figure 8, the vehicle list display 615 includes information to uniquely identify each vehicle 46, such as vehicle ID and vehicle name, and information indicating the status of each vehicle 46, such as stopped, automatic driving, or power off. The order in which the vehicles 46 are displayed in the vehicle list display 615 may be changed depending on whether an assistance request has been made, whether monitoring is being performed, the vehicle status, etc. The vehicle list display 615 may also include an operation button for inputting remote monitoring operation instructions (a steering wheel icon in Figure 8) and an operation button for inputting remote viewing operation instructions (a camera icon in Figure 8), as illustrated in Figure 8. Furthermore, the display screen 610 in Figure 8 includes a vehicle operation instruction button 612a and a monitoring start / end button 616. The vehicle operation instruction button 612a is an operation button for inputting remote assistance operation instructions (start driving instruction / stop driving instruction) for the vehicle 46 that is displaying vehicle information. In the example in Figure 8, the vehicle operation instruction button 612a is implemented by the forward image 611a. The monitoring start / end button 616 is an operation button for inputting operation instructions to start and / or end remote monitoring for the vehicle 46 that is displaying vehicle information.

[0096] Figure 9 shows another example of the screen display of an operator terminal in remote control according to the embodiment. The display screen 620 in Figure 9 is an example of the screen display presented to the operator of the operator terminal 20 performing remote control. The display screen 620 in Figure 9 includes a plurality of screens 610a to 610d corresponding to a plurality of vehicles 46. Each of the plurality of screens 610a to 610d is the same as, for example, the display screen 610 in Figure 8. The display screen 620 in Figure 9 includes a simultaneous monitoring start / end button 621 for the vehicles 46 on the screen and an operator information display 624 for the terminal. The simultaneous monitoring start / end button 621 is an operation button that corresponds to the simultaneous operation of all monitoring start / end buttons 616 on the plurality of screens 610a to 610d. In the example of Figure 9, an example is shown in which an integrated vehicle list display 625 is displayed instead of the vehicle list display 615 on each screen.

[0097] Figure 10 is a sequence diagram showing an example of the information processing flow at the start of remote monitoring, executed by each part of the remote control system 1 according to the embodiment. When the operator terminal 20 receives an operation instruction to start monitoring (S501), it transitions to monitoring mode (S502). If the vehicle 46 to be remotely monitored is offline, the operator terminal 20 waits until it comes online, and then proceeds to processing from S504 onwards (S503).

[0098] If the vehicle 46 to be monitored is online, the operator terminal 20 sends a command to the server 30 to start monitoring the vehicle 46 (S504). Upon receiving this command, the server 30 transmits the command to start monitoring from the operator terminal 20 to the edge device 40 that controls the vehicle 46 (S505).

[0099] If the vehicle 46 is not in monitoring mode, the edge device 40, upon receiving the monitoring start instruction, transitions to monitoring mode, starts delayed monitoring, and begins accepting operation instruction commands (control signals) (S506a). The edge device 40 also sends an OK response to the server 30 notifying that it has entered a state of waiting for control signals (S507). Upon receiving this, the server 30 receives monitoring information from the edge device 40 indicating that it is being monitored by the user who initiated the intervention instruction (S508), then transitions to monitoring mode and starts delayed monitoring (S509a). Subsequently, the server 30 sends an OK response to the operator terminal 20 notifying that it has entered a state of waiting for control signals (S510). Upon receiving this, the operator terminal 20 activates the vehicle operation instruction button 612a, begins sending back information regarding the transmission of vehicle information, and switches the screen display to the monitoring status display (S511a). Furthermore, after sending an OK response to the server 30, the edge device 40 and / or the server 30 distribute monitoring user information, indicating which user is monitoring each vehicle, to all users (S512-S513). The operator terminal 20 that receives the monitoring user information displays the operator information monitoring each vehicle 46 (S514) (see Figures 8-9).

[0100] If the vehicle 46 in question is being remotely monitored by another operator terminal 20, the remote start command from the operator terminal 20 is rejected by the edge device 40. In this case, the edge device 40 transmits an NG response to the operator terminal 20 via the server 30 (S515-S516). Upon receiving the NG response, the operator terminal 20 cancels the monitoring mode (S517).

[0101] Figure 11 is a sequence diagram showing an example of the information processing flow related to communication delay monitoring during remote monitoring, which is performed by each part of the remote control system 1 according to the embodiment. In monitoring mode, the edge device 40 transmits vehicle information with transmission information attached to it to the operator terminal 20 via the server 30 (S601-S602). The operator terminal 20, upon receiving this, displays the vehicle information (S603) and sends back the transmission information attached to the received vehicle information to the edge device 40 via the server 30 (S604-S605). The edge device 40, upon receiving this, checks whether the transmission information sent back from the operator terminal 20 via the server 30 was transmitted from its own device (S606). If the transmission information matches that transmitted from its own device, the edge device 40 calculates the estimated round-trip delay time from the time the vehicle information is transmitted until it is received, based on the transmission information (S607a). If the estimated round-trip delay time exceeds a predetermined threshold, the edge device 40 performs MRM such as an emergency stop, and after canceling the monitoring mode (S608), it distributes the information that monitoring has been canceled to the server 30 (S609). Upon receiving this, the server 30 cancels the monitoring mode (S610) and distributes the information that monitoring has been canceled to the operator terminal 20 (S611). Upon receiving this, the operator terminal 20 disables the vehicle operation instruction button 612a that instructs the start / stop of driving and displays an error (S612). Subsequently, the operator terminal 20 re-executes the monitoring start sequence (S613).

[0102] Note that the threshold for the estimated round-trip delay time may be set to be longer only for the first time. This initial threshold may be set taking into account the delay related to the internal state switching process.

[0103] Furthermore, if the edge device 40 does not receive vehicle information transmission data, i.e., returned transmission data, for a certain period of time, it may perform MRM and cancel the monitoring mode.

[0104] The edge device 40 may periodically compare the transmission time of the last received vehicle information transmission data with the current time. If the comparison result (time difference) is greater than or equal to a predetermined threshold, the edge device 40 may execute MRM and deactivate the monitoring mode. This delay monitoring allows for the detection of both cases—delays and failure to receive transmission information—in a single process.

[0105] Furthermore, the execution of MRM and the deactivation of monitoring mode may be performed if the time information within the edge device 40 has been rolled back by a certain period of time or more. This makes it possible to respond when the internal time of the edge device 40 becomes abnormal. Whether the time information within the edge device 40 has been rolled back by a certain period of time or more may be determined by comparing the time information before and after correction when the time correction is performed in the edge device 40 based on communication with the outside, or by comparing it with the time information within each terminal in the server 30 or operator terminal 20 that receives the transmitted information from the edge device 40, or by comparing it with the transmitted information already sent by the edge device 40.

[0106] Figure 12 is a sequence diagram showing an example of the information processing flow related to operator operation instructions during remote monitoring, as performed by each part of the remote control system 1 according to the embodiment. Here, the differences from the flow in Figure 11 will be mainly explained. In monitoring mode, the operator terminal 20 receives vehicle information with transmission information attached from the edge device 40 via the server 30 (S601-S602) and displays the received vehicle information (S603). Also, when the operator terminal 20 obtains an operation instruction from the operator (S701), it attaches the latest transmission information to the operation instruction command (control signal) and sends it back to the edge device 40 via the server 30 (S702-S703). The edge device 40, upon receiving this, checks whether the transmission information sent back from the operator terminal 20 via the server 30 was sent from its own device (S606), and if the transmission information matches that sent from its own device, it calculates the actual round-trip delay time (S607b). If the round-trip delay time does not exceed a predetermined threshold, the edge device 40 controls the vehicle 46 according to the control signal corresponding to the operator's operation instruction (S704). Note that the processing when the round-trip delay time exceeds a predetermined threshold (S608-S613) is the same as the flow shown in Figure 11, so the explanation is omitted here.

[0107] Figure 13 is a sequence diagram showing an example of the flow of information processing at the end of remote monitoring, executed by each part of the remote control system 1 according to the embodiment. In monitoring mode, the operator terminal 20 receives vehicle information, including information indicating whether the vehicle is in motion, from the edge device 40 via the server 30 (S801-S802), and displays the received vehicle information (S803). When the operator terminal 20 receives an operation instruction from the operator to end monitoring (S804), if the vehicle 46 being remotely monitored is in motion, it displays a confirmation dialog (notification) to ask if it is OK to end remote monitoring (S805). After the confirmation dialog is displayed, if the operator terminal 20 receives an operation instruction from the operator to end monitoring (S806), it sends a monitoring termination instruction to the edge device 40 via the server 30 (S807-S808). Upon receiving this, the edge device 40 cancels the monitoring mode (S809) and then sends an OK response to the server 30 notifying that the monitoring mode has been canceled in accordance with the instruction (S810). Upon receiving this, the server 30 cancels the monitoring mode (S811) and sends an OK response to the operator terminal 20 (S812). Upon receiving this, the operator terminal 20 disables the vehicle operation instruction button 612a, terminates the return transmission of vehicle information, and ends the display of monitoring (S813). Subsequently, the operator terminal 20 cancels the monitoring mode (S814). After sending the OK response to the server 30, the edge device 40 distributes monitoring mode cancellation information to all users, indicating that the monitoring mode has been canceled (S815-S816). Upon receiving the monitoring mode cancellation information, the operator terminal 20 updates the operator information monitoring each vehicle 46 (S817) (see Figures 8-9).

[0108] Regarding the display of the confirmation dialog (notification) (S805), the statement that the remotely monitored vehicle 46 is in motion may mean that the vehicle speed (vehicle speed) of vehicle 46 is not zero, or that the automatic driving system has control over the vehicle 46. In other words, whether the remotely monitored vehicle 46 is in motion may be determined by acquiring the vehicle speed as vehicle information, or by acquiring the automatic driving status as vehicle information.

[0109] Here, the right of operation may refer to the authority to directly operate the steering wheel, accelerator, brakes, etc., of the vehicle 46. This right of operation may be held by the vehicle 46's automated driving system, a remote operator, or a driver riding in the vehicle 46. For example, during remote control, the operator has the right of operation. For example, during remote assistance, the automated driving system has the right of operation. In addition, the operator during remote assistance can operate the automated driving system.

[0110] For example, the confirmation dialog (notification) displayed (S805) may display something like, "Are you sure you want to cancel? [OK] [Cancel]". If [OK] is selected in this confirmation dialog, the operator terminal 20 may receive this as an instruction from the operator to terminate monitoring after the confirmation dialog was displayed. On the other hand, if [Cancel] is selected in this confirmation dialog, the processing from S807 onwards in Figure 13 does not need to be executed.

[0111] For example, the confirmation dialog (notification) displayed (S805) may display something like, "Are you sure you want to cancel? [OK]". If [OK] is selected in this confirmation dialog, the operator terminal 20 may receive this as an instruction from the operator to terminate monitoring after the confirmation dialog has been displayed. On the other hand, if [OK] is not selected in this confirmation dialog within a predetermined time, the display of the confirmation dialog ends, and the processing from S807 onwards in Figure 13 does not need to be executed.

[0112] The display of the confirmation dialog (notification) (S805) is not a mandatory configuration and can be omitted as appropriate. In this case, if the vehicle 46 being remotely monitored is in motion, the operator terminal 20 may only acquire the operation instruction to end monitoring by the operator when the corresponding operation button is pressed and held. In other words, in the example of Figure 8, the operator terminal 20 acquires a tap operation on the vehicle operation instruction button 612a as an instruction to start remote monitoring in viewing mode, while if the vehicle 46 is in motion during remote monitoring, it may disable the tap operation on the vehicle operation instruction button 612a and enable only the long-press operation. Note that the configuration may also enable a swipe operation instead of a long-press operation. Furthermore, the operation for disabling and / or enabling may differ depending on whether the vehicle 46 is in motion or stopped during remote monitoring. In other words, if the vehicle 46 is in motion during remote monitoring, the monitoring mode may be deactivated only by a different operation than when the vehicle is stopped.

[0113] Furthermore, the confirmation dialog (notification) may be displayed in the same manner not only when a remote monitoring termination command is received, but also when a remote monitoring start command is received. In addition, the remote monitoring start command may be received not only by a tap operation, but also by a long press or swipe operation on the vehicle operation command button 612a.

[0114] In addition, in monitoring mode, the operator terminal 20 may disable the monitoring start / end button 616 if the vehicle 46 being remotely monitored is in motion.

[0115] In the above embodiment, the example given is that the system transitions to monitoring mode when the operator remotely initiates monitoring using the monitoring start / end button 616 in viewing mode (see Figure 10), but the system is not limited to this. Monitoring mode may also be initiated only when a monitoring request is received from the vehicle 46. Figure 14 is a sequence diagram showing another example of the information processing flow at the start of remote monitoring, executed by each part of the remote control system 1 according to the embodiment. Here, the differences from the flow in Figure 10 will be mainly explained. If the vehicle 46 is not in monitoring mode, the edge device 40, which has received the monitoring start instruction (S505), transitions to monitoring mode (S506b). Subsequently, the edge device 40 sends an OK response to the server 30 (S507). After receiving this, the server 30 receives monitoring information from the edge device 40 indicating that it is being monitored by the user who initiated intervention via the monitoring start instruction (S508), and then transitions to monitoring mode (S509b). Subsequently, the server 30 sends an OK response to the operator terminal 20 (S510). Furthermore, upon receiving this, the operator terminal 20 switches the screen display to the monitoring status (S511b). The edge device 40 then sends an assistance request to the server 30 if the vehicle 46 is unable to continue automatic driving (S521). Upon receiving this, the server 30 transmits the assistance request to the operator terminal 20 that sent the remote monitoring instruction for the target vehicle 46 (S522a). Upon receiving this, the operator terminal 20 displays the received assistance request and, in the same manner as the process in S511a of Figure 10, activates the vehicle operation instruction button 612a and begins sending back information regarding the transmission of vehicle information (S523). Subsequently, an auxiliary start notification is transmitted from the operator terminal 20 to the edge device 40 via the server 30 (S524-S525), and the server 30 and the edge device 40 transition to monitoring mode in the same manner as the processes in S506a and S509a of Figure 10, and the edge device 40 starts delay monitoring and receiving control signals in the same manner as the process in S506a of Figure 10 (S526). Note that delay monitoring is not limited to this step, but may be started at an earlier point, such as S506b.

[0116] If vehicle 46 is capable of automatically resuming driving, the assistance request or the transition to monitoring mode may be canceled on either the vehicle 46 side or the server 30 side.

[0117] In a configuration where monitoring mode is initiated only when a monitoring request is also received from the vehicle 46 (see Figure 14), the operator may remotely initiate monitoring using the monitoring start / end button 616 after a monitoring request is received from the vehicle 46. Figure 15 is a sequence diagram showing another example of the information processing flow at the start of remote monitoring, executed by each part of the remote control system 1 according to the embodiment. Here, the differences from the flow in Figure 10 or Figure 14 will be mainly explained. The edge device 40 sends an auxiliary request to the server 30 when the vehicle 46 is unable to continue automatic driving (S521). The server 30, upon receiving this, transmits the auxiliary request to all operator terminals 20 (S522b). Subsequently, if an operation instruction to start monitoring is obtained at an operator terminal 20 that received the auxiliary request (S501), that operator terminal 20 transitions to monitoring mode (S502). Note that if a monitoring request is also received from the vehicle 46, the vehicle 46 is online, so the flow in Figure 15 does not include the processing in S5033. The operator terminal 20, having transitioned to monitoring mode, sends a command to the server 30 to start monitoring the vehicle 46 (S504). Upon receiving this, the server 30 transmits the command to start monitoring from the operator terminal 20 to the edge device 40 that controls the vehicle 46 (S505). Steps S506 to S510 are the same as in Figure 10, so their explanation is omitted. Upon receiving the OK response from the server 30 regarding the transition to monitoring mode, the operator terminal 20 switches the screen display to the monitoring status display, similar to the flow in Figure 14 (S511b).

[0118] Furthermore, if a monitoring request is received from the vehicle 46, or if an assistance request is received from the edge device 40, one of the operator terminals 20 that is remotely viewing the target vehicle 46 may be automatically switched to monitoring mode.

[0119] In addition, in the remote control screen display according to the above embodiment (see Figures 8 to 9), monitoring level displays 619a and 619b may be shown. Figure 16 is a diagram showing another example of the screen display of the operator terminal 20 in the remote control according to the embodiment. The display screen 640 in Figure 16 is an example of a screen display presented to the operator on the operator terminal 20 in monitoring mode performing remote monitoring. As shown in Figure 16, the display screen 640 includes monitoring level displays 619a and 619b. When no operator is performing remote monitoring on the vehicle 46 displaying vehicle information, these monitoring level displays 619a and 619b do not display anything, as illustrated in Figures 8 and 9. On the other hand, when an operator on their own terminal is performing remote monitoring on the vehicle 46 displaying vehicle information, the operator terminal 20 displays the monitoring level, as illustrated in Figure 16. Here, the monitoring level is information that indicates what the operator can do with respect to the vehicle 46 displaying vehicle information. As an example, the monitoring level indicators 619a and 619b are configured to distinguish between cases where the operator can perform remote assistance operation instructions and cases where remote control operation instructions can be performed by using different display characteristics such as the color, shape, and background color of the display.

[0120] In the remote control screen display according to the above embodiment (see Figures 8-9), a selection of operation instructions that the operator can choose according to the result of the situation assessment may be displayed. When the display screen 640 in Figure 16 receives a support request from the vehicle 46, in addition to the vehicle operation instruction button 612a assigned to start / end driving, vehicle operation instruction buttons 612b to 612d assigned to other operation instructions may be displayed. The number of vehicle operation instruction buttons to which other operation instructions are assigned may be two or four or more. In the example in Figure 16, the vehicle operation instruction button 612b assigned to the operation instruction "dodge to the left" is realized by the left image 611b. The vehicle operation instruction button 612c assigned to the operation instruction "dodge to the right" is realized by the right image 611c. The vehicle operation instruction button 612d assigned to the operation instruction "standby" is realized by the rear image 611d. Furthermore, the number of vehicle operation instruction buttons, their assignments, and their enabled / disabled states may be dynamically changed according to the type of support request (type of support) from the vehicle 46 and the status of the vehicle 46. In addition to situations where a support request is made from the vehicle 46, there may also be cases where the operator initiates remote operation instructions even when no support request is received from the vehicle 46, such as when the operator actively begins monitoring after viewing a screen display on the operator terminal 20, such as video. In such cases, the number of vehicle operation instruction buttons, their assignments, and their enabled / disabled states can be dynamically changed according to the status of the vehicle 46. Alternatively, when remote assistance or remote control is initiated when no support request has been received from the vehicle 46, the operator may be asked to input the type of support, or presented with options for the operator to select. The number of vehicle operation instruction buttons, their assignments, and their enabled / disabled states may then be dynamically changed according to the type of support. In short, the number of vehicle operation instruction buttons, their assignments, and their enabled / disabled states may be dynamically changed according to the type of support received from the vehicle 46 and the status of the vehicle 46.

[0121] For example, if the vehicle 46 is in motion and assistance is requested due to the detection of a parked vehicle ahead, the operator terminal 20 may display only the vehicle operation instruction button 612d assigned to the "standby" operation instruction, instead of the vehicle operation instruction button 612a assigned to "start / end driving". For example, if the vehicle 46 is stopped and assistance is requested due to the detection of a parked vehicle ahead, the operator terminal 20 may display, in addition to the vehicle operation instruction button 612a assigned to "start / end driving", the vehicle operation instruction button 612b assigned to the "avoid to the left" operation instruction and the vehicle operation instruction button 612c assigned to the "avoid to the right" operation instruction. For example, if the operator terminal 20 is requested due to the presence of a pedestrian crossing in front of the vehicle 46, the operator terminal 20 may display only the vehicle operation instruction button 612a assigned to "start / end driving".

[0122] In the above embodiment, camera footage was used as an example of vehicle information to which transmission information is attached, but this is not limited to this. For example, the edge device 40 may attach transmission information to other vehicle information in the camera footage, such as information indicating whether the vehicle is in autonomous driving mode (vehicle system status information), for vehicles 46 that are not being monitored by video. In other words, the operator terminal 20 may return the transmission information attached to the camera footage for vehicles 46 that are being monitored by video, and return the transmission information attached to other information in the camera footage (for example, vehicle system status information) for vehicles 46 that are not being monitored by video. The choice of which information to attach transmission information to may be changed depending on the display state of the UI, such as whether the camera footage is being displayed on the operator terminal 20.

[0123] Furthermore, the above-mentioned variations can be combined in any way.

[0124] Thus, by implementing a monitoring mode, the remote control system according to this embodiment clarifies the operator's responsibility for driving and ensures reliable execution of operation instructions. In other words, the remote control system according to this embodiment can improve the safety of remote control for autonomous driving mobility.

[0125] Furthermore, some or all of the functions of each device of the remote control system 1 relating to the present disclosure (operator terminal 20, controller 25, server 30, edge device 40, and vehicle 46) may be implemented by other devices of the remote control system 1.

[0126] In the embodiments described above, the determination of "whether or not it is A" may be achieved by determining only whether it is "A", by determining only whether it is "not A", or by determining both.

[0127] In the embodiments described above, "any of A" means "at least one of A".

[0128] Furthermore, the programs executed by each device of the remote control system 1 according to the above embodiment may be provided as files in an installable or executable format, recorded on a computer-readable non-transient recording medium such as a CD-ROM, floppy disk, CD-R, or DVD.

[0129] Furthermore, the programs executed by each device of the remote control system 1 according to the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. Alternatively, the programs executed by each device of the remote control system 1 according to the above embodiment may be provided or distributed via a network such as the Internet.

[0130] Furthermore, the program executed by each device of the remote control system 1 according to the above embodiment may be pre-installed and provided in ROM or the like.

[0131] According to at least one embodiment described above, the safety of remote control of autonomous mobility can be improved.

[0132] While embodiments of this disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0133] (Note) The above description of embodiments discloses the following technologies: (1) An information processing method performed by an information processing system that supports remote control of at least one autonomous mobile body by an operator, wherein the operator side of the information processing system, upon receiving an operation instruction to start remote monitoring, transitions from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, and enables the input of an operation instruction for remote control of the target mobile body among the at least one autonomous mobile body that is the subject of the remote monitoring; and upon receiving an operation instruction to end the remote monitoring, transitions from the monitoring mode to the viewing mode and disables the input of an operation instruction for remote control. (2) The information processing method according to (1) above, wherein the viewing mode is a mode of remote control in which mobile body information relating to the at least one autonomous mobile body is presented to the operator in a remotely viewable manner. (3) The information processing method according to (1) or (2) above, wherein the monitoring mode is a mode of remote control which includes a first transmission mode in which the target mobile object is remotely controlled by transmitting a single control signal to the target mobile object in response to the remote operation instructions. (4) The information processing method according to any one of (1) to (3) above, wherein the monitoring mode is a mode of remote control which includes a second transmission mode in which the target mobile object is remotely controlled by transmitting a series of control signals to the target mobile object in response to the remote operation instructions. (5) The information processing method according to any one of (1) to (4) above, wherein the remote operation instructions include at least one operation instruction by the operator to start driving and stop driving to the target mobile object.(6) The information processing method according to any one of (1) to (5) above, wherein the target mobile body side of the information processing system transmits transmission information to at least one operator side with transmission information attached to mobile body information relating to the target mobile body at predetermined intervals; when the operator side receives an operation instruction for the remote operation in the monitoring mode, it attaches the transmission information that was attached to the mobile body information from the target mobile body side to a control signal corresponding to the acquired remote operation instruction and sends it back to the target mobile body side; the target mobile body side calculates a round-trip delay time including the operator's response time from when the mobile body information is presented to the operator until the operator's remote operation instruction based on the mobile body information is executed, using the transmission and reception times of the transmission information, and executes control of the target mobile body in accordance with the control signal from the operator side if the round-trip delay time is within a predetermined time; (7) The information processing method according to (6) above, wherein if the round-trip delay time exceeds a predetermined time, the side of the target mobile body does not permit control of the target mobile body in accordance with the control signal from the operator, and performs a pre-configured measure to automatically transition the target mobile body to a safe state. (8) The information processing method according to (6) above or (7) above, wherein the side of the target mobile body rejects the control signal in response to the remote operation instruction from another operator on the side of the operator that has transitioned to the monitoring mode.(9) The information processing method according to any one of the above paragraphs (1) to (8), wherein the side of the information processing system for the target mobile body transmits transmission information to at least one operator side with transmission information attached to the mobile body information relating to the target mobile body at predetermined intervals; the operator side periodically returns the transmission information attached to the mobile body information from the side of the target mobile body in the monitoring mode to the side of the target mobile body; the side of the target mobile body calculates an estimated round-trip delay time, including the estimated response time from when the mobile body information is presented to the operator until the operator's remote operation instruction based on the mobile body information is executed, using the transmission and reception times of the transmission information; and if the estimated round-trip delay time exceeds a predetermined time, it executes a pre-configured measure to automatically transition the target mobile body to a safe state. (10) The information processing method according to the above paragraph (9), wherein the side of the target mobile body does not execute the pre-configured measure based on the estimated round-trip delay time for the other operators on the side of the operator that has transitioned to the monitoring mode. (11) The information processing method according to any one of (1) to (10) above, wherein the server that relays the transmission and reception of information between the side of the information processing system to the target mobile body, or between each side of the at least one autonomous mobile body and the side of at least one operator, restricts the side of the operator that causes the target mobile body to transition to the monitoring mode to one of the at least one operator's side, and the other operator's side of the operator that has transitioned to the monitoring mode disables the input of an operation instruction to start the remote monitoring of the target mobile body. (12) The information processing method according to any one of (1) to (11) above, wherein the operator's side of the information processing system disables the input of an operation instruction to end the remote monitoring of the target mobile body when the target mobile body is moving in the monitoring mode.(13) The information processing method according to any one of (1) to (11) above, wherein the operator side of the information processing system receives an instruction to terminate remote monitoring of the target mobile object while the target mobile object is in motion in the monitoring mode, and presents a confirmation notification to the operator to prevent erroneous operation. (14) The information processing method according to (12) or (13) above, wherein "while the target mobile object is in motion" means that the speed of the target mobile object is not zero, or that the automatic driving system holds the right to operate the target mobile object. (15) The information processing method according to any one of (1) to (14) above, wherein the target mobile object side of the information processing system cancels the monitoring mode when the driver directly operates the target mobile object and / or the automatic driving system for the target mobile object. (16) The information processing method according to any one of (1) to (15) above, wherein the operator side of the information processing system prohibits the input of an operation instruction to terminate the remote monitoring of the target mobile object and the input of an operation instruction for the remote operation in the monitoring mode. (17) The information processing method according to any one of (1) to (16) above, wherein at least one operator side of the information processing system presents operator information indicating the operator whose operation is acquired by the operator side that has transitioned to the monitoring mode, together with the target mobile object on the operator side that has transitioned to the monitoring mode. (18) The information processing method according to any one of (1) to (17) above, wherein each of the at least one operator side of the information processing system waits to receive an auxiliary request from the target mobile object, and the target mobile object transmits the auxiliary request to the operator side if it is unable to continue moving or if instructions from the operator are required.(19) The information processing method according to any one of the above paragraphs (1) to (18), wherein the operator side changes the number of at least one remote control buttons for inputting remote control instructions in the monitoring mode, the assignment of the operation instructions to be acquired, and the enabled or disabled state, according to the state of the target mobile body and / or the type of support provided to the target mobile body. (20) An information processing device that realizes the operator side in an information processing system that supports remote control of at least one autonomous mobile body by an operator, wherein when an operation instruction to start remote monitoring is acquired, the device transitions from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility to a monitoring mode, and enables the input of a remote control instruction to remotely control the target mobile body among the at least one autonomous mobile body that is the subject of the remote monitoring, and when an operation instruction to end remote monitoring is acquired, the device transitions from the monitoring mode to the viewing mode, and disables the input of the remote control instruction. (21) A program that, in an information processing system that supports remote control by an operator over at least one autonomous mobile body, causes a computer representing the operator to, upon receiving an operation instruction to start remote monitoring, transition from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, enable the input of an operation instruction for remote control of the target mobile body among the at least one autonomous mobile body that is the subject of the remote monitoring, and upon receiving an operation instruction to end the remote monitoring, transition from the monitoring mode to the viewing mode and disable the input of an operation instruction for the remote control. (22) An information processing device comprising at least one processor and at least one memory, wherein the at least one processor executes the information processing method described in any one of the above items (1) to (19) by executing a program stored in the at least one memory.(23) A program that causes a computer to execute the information processing method described in any one of the above paragraphs (1) to (19), or a computer-readable non-transient storage medium in which such program is stored.

[0134] 1 Remote control system 2 Front end 20 Operator terminal 201 Communication unit 202 Calculation unit 203 Storage unit 204 Operation input unit 25 Controller 3 Back end 30 Server (information processing device) 301 Communication unit 302 Calculation unit 303 Storage unit 4 Edge 40 Edge device 401 Communication unit 402 Calculation unit 403 Storage unit 404 Vehicle information input unit 405 Vehicle control unit 45 Camera 46 Vehicle (mobile object) 610 Display screen 611 Image 612a-612d Vehicle operation instruction buttons 613 Map 614 Operator information display 615 Vehicle list display 616 Monitoring start / end button 617 Vehicle status display 619a, 619b Monitoring level display 621 Simultaneous monitoring start / end button 620 Display screen 624 Operator information display 625 Vehicle list display 640 Display screen 8 Information processing unit 81 Processor 82 ROM 83 RAM 84 Equipment I / F section

Claims

1. An information processing method performed by an information processing system that supports remote control by an operator over at least one autonomous mobile object, wherein the operator side of the information processing system, upon receiving an operation instruction to start remote monitoring, transitions from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, and enables the input of an operation instruction for remote control of the target mobile object among the at least one autonomous mobile object that is the subject of remote monitoring; and upon receiving an operation instruction to end remote monitoring, transitions from the monitoring mode to the viewing mode and disables the input of an operation instruction for remote control.

2. The information processing method according to claim 1, wherein the viewing mode is a mode of remote control that remotely presents mobile information relating to at least one autonomous mobile body to the operator for viewing.

3. The information processing method according to claim 1, wherein the monitoring mode is a mode of remote control that includes a first transmission mode in which a target mobile object is remotely controlled by transmitting a single control signal to the target mobile object in response to an operation instruction for the remote operation.

4. The information processing method according to claim 1, wherein the monitoring mode is a mode of remote control that includes a second transmission mode for remotely controlling the target mobile object by transmitting a series of control signals to the target mobile object in response to the remote operation instructions.

5. The information processing method according to claim 1, wherein the remote operation instruction includes an operation instruction by the operator for at least one of starting or stopping the movement of the target moving object.

6. The information processing method according to claim 1, wherein the information processing system's side of the target mobile body transmits transmission information to at least one operator's side at a predetermined interval, the operator's side, when it receives an operation instruction for the remote operation in the monitoring mode, transmits the transmission information that was attached to the mobile body information from the target mobile body side to a control signal corresponding to the acquired remote operation instruction, and returns it to the target mobile body side, the target mobile body's side calculates a round-trip delay time, including the operator's response time from when the mobile body information is presented to the operator until the operator's remote operation instruction based on the mobile body information is executed, using the transmission and reception times of the transmission information, and executes control of the target mobile body in accordance with the control signal from the operator's side if the round-trip delay time is within a predetermined time.

7. The information processing method according to claim 6, wherein if the round-trip delay time exceeds a predetermined time, the side of the target moving object does not permit control of the target moving object in accordance with the control signal from the operator, and performs a pre-configured measure to automatically transition the target moving object to a safe state.

8. The information processing method according to claim 6, wherein the target moving object rejects control signals from other operators on the side of the operator that has transitioned to the monitoring mode, in response to remote operation instructions.

9. The information processing method according to claim 1, wherein the information processing system's side of the target mobile body transmits transmission information to at least one operator's side at predetermined intervals, the operator's side periodically returns the transmission information that was attached to the mobile body information from the target mobile body side in the monitoring mode to the target mobile body side, and the target mobile body calculates an estimated round-trip delay time, including the estimated response time from when the mobile body information is presented to the operator until the operator's remote operation instruction based on the mobile body information is executed, using the transmission and reception times of the transmission information, and if the estimated round-trip delay time exceeds a predetermined time, it executes a pre-configured measure to automatically transition the target mobile body to a safe state.

10. The information processing method according to claim 9, wherein the target moving object does not perform the pre-configured measures based on the estimated round-trip delay time for the other operator's side of the operator's side that has transitioned to the monitoring mode.

11. The information processing method according to claim 1, wherein the server that relays the transmission and reception of information between the target mobile body side of the information processing system, or between each of the at least one autonomous mobile body side and at least one operator side, restricts the operator side that causes the target mobile body to transition to the monitoring mode to any one of the at least one operator side, and the other operator side of the operator side that has transitioned to the monitoring mode disables the input of an operation instruction to start the remote monitoring of the target mobile body.

12. The information processing method according to claim 1, wherein the operator of the information processing system disables the input of an operation instruction to terminate remote monitoring of the target mobile object when the target mobile object is in motion in the monitoring mode.

13. The information processing method according to claim 1, wherein, in the monitoring mode, the operator of the information processing system receives an instruction to terminate remote monitoring of the target mobile object while the target mobile object is in motion, and presents the operator with a confirmation notification to prevent erroneous operation.

14. The information processing method according to claim 12 or 13, wherein "the target moving object is in motion" means that the speed of the target moving object is not zero, or that the automatic driving system holds the right to operate the target moving object.

15. The information processing method according to claim 1, wherein the information processing system on the side of the target mobile body releases the monitoring mode when the driver directly operates on the target mobile body and / or the automatic driving system of the target mobile body.

16. The information processing method according to claim 1, wherein the operator of the information processing system prohibits input of an operation instruction to terminate the remote monitoring of the target moving object and input of an operation instruction for the remote operation in the monitoring mode.

17. The information processing method according to claim 1, wherein at least one operator side of the information processing system presents operator information indicating the operator whose operation is acquired by the operator side that has transitioned to the monitoring mode, together with the target moving object on the operator side that has transitioned to the monitoring mode.

18. The information processing method according to claim 1, wherein each of the at least one operator side of the information processing system waits to receive an assistance request from the target mobile body, and the target mobile body transmits the assistance request to the operator side if it is unable to continue moving or if instructions from the operator are required.

19. The information processing method according to claim 1, wherein the operator changes one of the following in the monitoring mode: the number of remote control buttons for inputting remote control operation instructions, the assignment of operation instructions to be acquired, and the enabled or disabled state, according to the state of the target mobile object and / or the type of support provided to the target mobile object.

20. An information processing device that represents the operator side in an information processing system that supports remote control by an operator over at least one autonomous mobile object, wherein when an operation instruction to start remote monitoring is received, the device transitions from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, and enables the input of an operation instruction for remote control of the target mobile object among the at least one autonomous mobile object that is the subject of the remote monitoring, and when an operation instruction to end the remote monitoring is received, the device transitions from the monitoring mode to the viewing mode and disables the input of an operation instruction for remote control.

21. A program that, in an information processing system supporting remote control by an operator over at least one autonomous mobile object, causes a computer representing the operator to, upon receiving an operation instruction to start remote monitoring, transition from a viewing mode in which the operator has no control responsibility to a monitoring mode in which the operator has control responsibility, and enable the input of a remote operation instruction to remotely control the target mobile object among the at least one autonomous mobile object that is the subject of the remote monitoring; and upon receiving an operation instruction to end the remote monitoring, transition from the monitoring mode to the viewing mode and disable the input of the remote operation instruction.

Images