Information processing method, information processing device, and program

Abstract

An information processing method according to the present disclosure supports remote control of a plurality of autonomous mobile bodies by a plurality of operators. The information processing method includes, when a target mobile body requiring remote intervention by an operator arises from among two or more monitored mobile bodies of which a first operator holds authority for remote control: determining a remote intervention scene on the basis of at least one of the situation of the target mobile body, the situation of surrounding traffic participants, and a relationship between the target mobile body and the surrounding traffic participants; determining, in accordance with the scene, whether it is necessary to issue an authority transfer alert proposing that the first operator's authority for remote control of the target mobile body be transferred to a second operator; and, if the scene requires an authority transfer alert, issuing a notification of the authority transfer alert via an operator terminal of the first operator that is displaying video from each of the two or more monitored mobile bodies.

Description

Information Processing Method, Information Processing Apparatus, and Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program.

[0002] In recent years, the introduction of moving bodies such as self-driving vehicles and remotely operated vehicles that can travel without the driver having to be on board has been promoted in various places such as roads, factories, airports, and ports. In the remote control of such moving bodies, a controller (operator) remotely monitors, operates, and drives while checking the video and audio obtained for the controlled object.

[0003] For example, in the remote control of moving bodies, in order to reduce service downtime, there have been cases where multiple operators manage the operation of multiple moving bodies. For example, Patent Document 1 discloses a technique for assigning a remote operator to a vehicle that is the object of remote monitoring.

[0004] Japanese Patent Application Laid-Open No. 2023-156078

[0005] However, when the monitored moving body is in a situation where remote intervention by the operator such as attention, operation, or driving is required, the operator needs to determine whether it is a situation where they can handle it themselves or a situation where they delegate authority to another operator to handle it. If this determination takes time, it may lead to a delay in the response to remote intervention and may affect traffic participants around the moving body.

[0006] The present disclosure has been made in view of the above, and one of its purposes is to support the determination related to the delegation of authority of an operator who monitors multiple moving bodies.

[0007] The information processing method relating to this disclosure is an information processing method performed by an information processing system that supports the remote control of multiple operators to multiple mobile bodies, each configured to move autonomously and perform predetermined tasks. In the information processing method, the multiple operators include a first operator and a second operator. When a target mobile body requiring remote intervention by an operator arises from among two or more mobile bodies being monitored by the first operator, the information processing method determines the scene of the remote intervention based on at least one of the following: the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants. The information processing method determines, according to the determined scene, whether an authority transfer alert is necessary to propose the transfer of authority from the first operator to the second operator. If an authority transfer alert is necessary, the authority transfer alert is notified via the operator terminal of the first operator, which is displaying video from each of the two or more mobile bodies being monitored.

[0008] Figure 1 is a diagram showing an example of the configuration of a remote control system according to the embodiment. Figure 2 is a diagram showing an example of the functional configuration of a flight management server according to the embodiment. Figure 3 is a diagram showing an example of the functional configuration of an operator terminal according to the embodiment. Figure 4 is a diagram showing an example of the functional configuration of a mobile body according to the embodiment. Figure 5 is a diagram showing an example of the hardware configuration of an information processing device that realizes each device included in the remote control system according to the embodiment. Figure 6 is a flowchart showing an example of the processing flow related to authority delegation according to the embodiment. Figure 7 is a diagram for explaining the processing related to authority delegation according to the embodiment. Figure 8 is a sequence diagram showing an example of the processing flow related to authority delegation according to the embodiment. Figure 9 is a diagram showing an example of the display of an authority delegation alert in an operator terminal according to the embodiment. Figure 10 is a diagram showing another example of the configuration of a remote control system according to the embodiment. Figure 11 is a flowchart showing another example of the processing flow related to authority delegation according to the embodiment.

[0009] Hereinafter, embodiments of the information processing method, information processing system, information processing device, mobile device, program, and recording medium related to this disclosure will be described in detail with reference to the attached drawings.

[0010] In this disclosure, components having the same or substantially the same function as those described above in previously shown drawings are denoted by the same reference numerals, and explanations may be omitted as appropriate. Furthermore, even when representing the same or substantially the same parts, the dimensions and proportions may be shown differently in different drawings. In addition, for example, from the viewpoint of ensuring the readability of the drawings, reference numerals may be assigned only to the main components in the explanation of each drawing, and reference numerals may not be assigned to components having the same or substantially the same function as those described above in previously shown drawings.

[0011] The information processing system relating to this disclosure is a remote control system that provides remote control, including remote monitoring and / or control, of at least one controlled mobile object, which includes at least one autonomously navigable autonomous mobile object, by an operator. The remote control system provides various services through its remote control, such as delivery, security, cleaning, childcare, nursing care, sales, agricultural work, manufacturing, cargo handling, transportation, and construction. In this remote control system, mobile object information, including video and audio of the controlled mobile object, is presented to the operator, and the operator performs remote monitoring of the mobile object based on the mobile object information. In addition, in the remote control system, an operator is assigned in response to a request for support (remote request) from a mobile object that requires remote control, and the assigned operator intervenes in the control of the mobile object, such as remote operation or remote driving.

[0012] For example, the controlled object transmits images (video) captured by cameras or other devices mounted on the controlled object to the remote control side. For example, the remote control side displays a screen containing images from the controlled object on the operator's terminal. The display screen contains at least one image (video) relating to at least one controlled object that the operator is responsible for. The operator monitors the status of each of the at least one controlled object that they are responsible for while viewing the display screen on their terminal. Here, the images used for remote control may be still images or moving images (video).

[0013] For example, if the controlled vehicle becomes unable to autonomously travel, for instance, if an obstacle such as a fallen object or a parked vehicle is detected in its path, the controlled vehicle sends a request for remote control assistance (remote request) to the remote control side. This assistance request may be an operation request requesting assistance through remote control based on the operator's operation instructions (remote instructions), or a driving request requesting assistance through remote driving based on the operator's driving instructions (remote instructions). For example, when the remote control side receives an assistance request from the controlled vehicle, it presents the operator with a remote instruction request requesting remote instructions for remote operation or remote driving. The operator performs remote control (assistance) of the controlled vehicle by inputting remote instructions while viewing the display screen.

[0014] In this disclosure, “remote control” is classified into three modes (functions) – “remote monitoring,” “remote operation,” and “remote driving” – which differ in the responsibilities of the controller (controller, operator), the available control, and the conditions for activating safety functions, depending on the form of control over the target moving object. Here, “remote control” is at least one of “remote monitoring,” “remote operation,” and “remote driving” over at least one target moving object, each of which is configured to move autonomously and perform a predetermined task.

[0015] In this disclosure, "remote monitoring" refers to a mode of remote control in which mobile information relating to at least one controlled mobile object and its surrounding environment is presented to the operator in a manner that allows for remote monitoring (confirmation), or in which the operator monitors (confirms) such information. This mobile information includes camera images and sensing information from cameras and various sensors installed around the controlled mobile object, information generated by processing such information, and / or information representing the state of the mobile object itself, such as speed. Cameras and various sensors installed around the controlled mobile object may be fixed infrastructure equipment or devices held or worn by an operator. An operator is a person who remotely monitors the controlled mobile object and remotely controls the mobile object as needed. Here, "remote monitoring" is an example of a mode of remote control in which a mobile object is remotely monitored by transmitting mobile information, such as images, relating to at least one controlled mobile object.

[0016] Furthermore, in this disclosure, "remote control" refers to a mode of remote control in which operational instructions (remote instructions) such as starting and / or stopping are given to a controlled mobile object from a remote location. As an example, "remote control" is a mode of remote control in which, in addition to "remote monitoring," control instructions are given to the controlled mobile object by transmitting a control signal (driving instruction signal) in response to an operational instruction to the controlled mobile object. These operational instructions are given by an operator remotely to control the actions of an automatically driving mobile object, such as starting or stopping the automatic driving of the mobile object, but may also include instructions related to emergency stops or actions unrelated to driving. Here, "remote control" is an example of a transmission mode in which a controlled mobile object is remotely controlled by a control signal transmitted as a single control instruction (remote instruction).

[0017] Furthermore, in this disclosure, "remote operation" refers to a mode of remote control in which the operation (such as steering and / or braking) of a controlled mobile object is performed from a remote location. In other words, "remote operation" is a mode of remote control in which driving instructions (remote instructions) such as steering and / or braking are given to a controlled mobile object from a remote location. As an example, "remote operation" is a mode of remote control in which, in addition to "remote monitoring" or "remote operation," control signals (driving signals) corresponding to driving instructions are transmitted to the controlled mobile object among at least one of the controlled mobile objects, thereby remotely giving control instructions to the said mobile object. Here, "remote operation" is an example of a transmission mode in which a controlled mobile object is remotely controlled by control signals transmitted as a series of control instructions (driving instructions).

[0018] In this disclosure, the modes of remote control, including "remote operation" and / or "remote driving," may be combined and referred to as "remote control." Therefore, "remote control" means that an operator remotely monitors the status of the controlled mobile object based on mobile object information (video and various other information) and remotely controls the controlled mobile object by remote instructions (operation instructions and / or driving instructions) as needed.

[0019] The controlled mobile entity relating to this disclosure performs various tasks, including autonomous driving, related to various services provided by remote control systems, such as delivery, advertising, security, cleaning, mobile sales, childcare, caregiving, agricultural work, manufacturing, cargo handling, transportation, and construction. For example, the controlled mobile entity is configured to perform predetermined tasks through autonomous control, control in accordance with the operations of an onboard driver, and / or remote control in accordance with remote instructions from an operator.

[0020] The controlled mobile object is, for example, a vehicle, but is not limited to that. The controlled mobile object may be any type of mobile object configured to perform predetermined tasks, such as movement, by remote control, at least in accordance with remote instructions from an operator. This mobile object may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Also, for example, the mobile object may be an Automatic Guided Vehicle (AGV), or various robots such as construction machinery, agricultural machinery, or drones. Furthermore, these mobile objects are not limited to transporting people, but may also transport objects other than people, and may provide specific services in addition to transportation. In addition, the mobile object may be configured to be controllable not only by autonomous control or remote control, but also by direct operation by a driver.

[0021] Thus, the remote control system according to this disclosure is configured to execute an information processing method (remote control support method) that supports an operator's remote control of at least one moving object of control, based on images (e.g., video) obtained on the controlled object side.

[0022] (First Embodiment) Figure 1 is a diagram showing an example of the configuration of a remote control system 1 according to an embodiment. The remote control system 1a (remote control system 1) illustrated in Figure 1 includes an operation management server 3, a database 4, an operator terminal 5, and a mobile unit 6.

[0023] Here, the operation management server 3 in this embodiment is an example of a remote control support device (information processing device) that manages remote control (operation) of the mobile object 6 and supports remote control by operator P. The operator terminal 5 is an example of a remote control terminal (information processing device) used by operator P for remote control of the mobile object 6. The mobile object 6 is an example of an object controlled by operator P.

[0024] Figure 1 illustrates a remote control system 1 that includes at least an operator terminal 5-1 used by operator P1 and an operator terminal 5-2 used by operator P2. Furthermore, the example in Figure 1 illustrates a remote control system 1 that includes a group of mobile units consisting of n mobile units 6-1 to 6-n. The number of operator terminals 5 and mobile units 6 included in the remote control system 1 can be arbitrarily changed according to design conditions, etc.

[0025] In the remote control system 1, the operation management server 3 is located on a cloud built on any network such as the internet. The operation management server 3 and the mobile unit 6 are connected via network N so as to be able to communicate. The database 4 is configured to be accessible from the operation management server 3. The operation management server 3 and the database 4 may be connected via any network such as the internet so as to be able to communicate, or they may be directly connected via a dedicated line so as to be able to communicate. The operation management server 3 and the operator terminal 5 may be connected via any network such as the internet so as to be able to communicate, or they may be directly connected via a dedicated line so as to be able to communicate. In addition, various communications in the remote control system 1 may be wired, wireless, or a combination of these.

[0026] Network N may be any cellular V2X public communication line, such as a mobile communication system of each generation in accordance with the IMT2020 provisions and the 3GPP (registered trademark) specifications, such as 3G, LTE (Long Term Evolution), 4G, and 5G, or a next-generation mobile communication system such as Beyond 5G (6G). Network N may also be a communication line using the LPWA (Low Power Wide Area) method, for example. Network N may also be a communication line using other communication methods, such as the IEEE-compliant DSRC method or the Vehicle-to-Cellular Network (V2N) method. Furthermore, Network N may be a communication line connected via short-range wireless communication of any communication method, such as Wi-Fi®, Bluetooth® Low Energy (BLE), or Ultra-Wide Band (UWB). This short-range wireless communication may be performed using an Electronic Toll Collection System (ETC).

[0027] Furthermore, the mobile device 6 may be connected to any communication line via an internal mobile network such as CAN (Controller Area Network), Ethernet (registered trademark), or USB (Universal Serial Bus (registered trademark)) (for example, an in-vehicle network in the case of a vehicle).

[0028] Furthermore, the connection of the mobile unit 6 and / or the operation management server 3 to the network N, and the connection of the operation management server 3 and / or the operator terminal 5 to any network such as the Internet, may be performed by establishing a VPN (Virtual Private Network).

[0029] The operation management server 3 may be configured as an integral part of the database 4. Furthermore, the operation management server 3 and the database 4 may each be configured as an integral part of one of the operator terminals 5, or they may be implemented by two or more operator terminals 5.

[0030] The operation management server 3 may be, for example, a server device built on the cloud. Figure 2 is a diagram showing an example of the functional configuration of the operation management server 3 according to the embodiment. As shown in Figure 2, the operation management server 3 has a communication control unit 31, a management unit 32, and a decision unit 33.

[0031] The communication control unit 31 controls communication with the database 4, the operator terminal 5, and the mobile device 6.

[0032] The control unit 32 assists operator P in the remote control of the mobile unit 6. This remote control assistance includes displaying images, videos, and various notifications about the mobile unit 6 on the operator terminal 5. It also includes executing remote control of the mobile unit 6 in response to various operations and instructions from operator P on the operator terminal 5.

[0033] The judgment unit 33 determines the scene of the mobile body 6 that issued the remote request, i.e., the remote intervention scene, and determines whether an authority delegation alert is necessary and the priority of the response based on the scene. The necessity of an authority delegation alert and the priority of the response are each pre-associated with the scene and stored in a database 4 or the like. This association may also be stored in the internal memory of the operation management server 3 or the like.

[0034] For example, the scene of a moving object 6 is determined based on at least one of the following: the status of the target moving object 6, the status of the surrounding traffic participants, and the relationship between the target moving object 6 and the surrounding traffic participants. Here, the status of the target moving object 6 includes at least one of the following: information on its state, such as whether it is moving or stopped; information on its location and travel path; and information on its location, such as whether it is on a road, in a parking lot, or on a road. The status of the surrounding traffic participants includes at least one of the following: the location of the surrounding traffic participants, their travel path, or their class identification result. The relationship between the target moving object 6 and the surrounding traffic participants includes at least one of the relationship between their locations and their travel paths. Details of scene determination will be described later.

[0035] In this disclosure, "delegation of authority" refers to the transfer of authority to perform remote control functions such as monitoring, operation, and driving from one operator P1 to another operator P2. In other words, authority includes at least one of legal monitoring authority and remote control authority. The necessity of a delegation of authority alert in this disclosure is different from the priority of response regarding the importance and immediacy of remote requests from the mobile device 6.

[0036] Database 4 is a storage device that stores various types of information related to remote control. For example, Database 4 stores various types of information such as the operator, the operating area (operating base) of the mobile unit 6, the mobile unit 6, the services provided by the mobile unit 6, and the operating pattern. Database 4 also stores information indicating the correspondence between pre-configured authorization delegation alerts, their respective response priorities, and the scenes.

[0037] The operator terminal 5 may be, for example, a computer installed at a control center where operator P performs remote control duties. Figure 3 is a diagram showing an example of the functional configuration of the operator terminal 5 according to this embodiment. As shown in Figure 3, the operator terminal 5 has a communication control unit 51, an input control unit 52, and an output control unit 53.

[0038] The communication control unit 51 controls communication with the operation management server 3.

[0039] The input control unit 52 acquires the operations of operator P on an input device mounted on or connected to the operator terminal 5.

[0040] The output control unit 53 displays various information related to remote control using an output device mounted on or connected to the operator terminal 5, such as a liquid crystal display. The output control unit 53 may also output various notification sounds and voices related to remote control using, for example, a speaker.

[0041] The mobile unit 6 may be configured to provide a predetermined service, including driving, at a predetermined driving base, for example. Figure 4 is a diagram showing an example of the functional configuration of the mobile unit 6 according to the embodiment. As shown in Figure 4, the mobile unit 6 has a communication control unit 61, a detection control unit 62, a driving control unit 63, and a speech control unit 64.

[0042] The communication control unit 61 controls communication with the operation management server 3.

[0043] The detection control unit 62 acquires information regarding the moving body 6 and its surroundings by using sensors (input devices) mounted on or connected to the moving body 6.

[0044] Note that one or more cameras (imaging devices) for capturing the surroundings to obtain images or videos (information regarding the surroundings of the moving body 6) may be mounted on or connected to the moving body 6 as sensors (input devices). Further, a LiDAR (Light Detection And Ranging), radar, or sonar for detecting surrounding objects (information regarding the surroundings of the moving body 6) may be mounted on or connected to the moving body 6 as sensors (input devices). Also, a microphone for picking up notification sounds or voices from surrounding objects may be mounted on or connected to the moving body 6.

[0045] Note that the information regarding the moving body 6 and its surroundings may be moving body information indicating the actual operation amount and state of the moving body 6, such as the remaining battery level, temperature, speed, acceleration, and current position, information on the surrounding situation of the moving body 6, such as the presence, number, type, and distance of surrounding objects, or various information used for obtaining these. Surrounding objects are traffic participants (surrounding traffic participants) existing around the moving body 6, such as surrounding obstacles like falling objects and walls, or various moving bodies such as pedestrians, bicycles, motorcycles, and automobiles. Note that traffic participants in a stopped state, such as parked vehicles, can be treated as obstacles.

[0046] Note that the information acquired by various sensors of the moving body 6 may be output to the travel control unit 63 of the moving body 6 and may be used for controlling operations related to various tasks of the moving body 6, including travel such as steering and acceleration / deceleration of the moving body 6.

[0047] The travel control unit 63 controls the travel of the mobile body 6 by controlling the travel mechanism based on at least one of the information acquired by various sensors of the mobile body 6 and the remote control instruction from the operation management server 3 based on the instruction (operation input) of the operator P at the operator terminal 5. Here, the remote control instruction includes information for remotely controlling the mobile body 6. For example, the remote control instruction includes various control commands for the mobile body 6 and the like.

[0048] The travel mechanism of the mobile body 6 includes a mechanism capable of performing various operations for executing various tasks including the movement of the mobile body 6, such as a mechanism capable of traveling the mobile body 6 in a predetermined direction for a predetermined distance at a predetermined speed. For example, under the control of the travel control unit 63, the travel mechanism of the mobile body 6 performs steering, driving, braking, etc. of the mobile body 6.

[0049] The speech control unit 64 controls the speech by the mobile body 6 by controlling the speaker mounted or connected to the mobile body 6 based on at least one of the information acquired by various sensors of the mobile body 6 (for example, the approach of surrounding traffic participants) and the speech instruction or call instruction from the operation management server 3 according to the instruction (operation input) of the operator P at the operator terminal 5.

[0050] FIG. 5 is a diagram showing an example of the hardware configuration of the information processing device 8 that realizes each device included in the remote control system 1 according to the embodiment. The information processing device 8 may be various computers such as a dedicated device, a PC, or a smartphone. The information processing device 8 may be a computer mounted or connected to each device included in the remote control system 1.

[0051] The information processing device 8 that realizes each function of the mobile body 6 may be a DCU (Domain Control Unit) such as an ECU (Electronic Control Unit) provided inside the mobile body 6 or a CDC (Cockpit Domain Controller) integrating a plurality of ECUs, or a computer such as an OBU (On Board Unit).

[0052] As shown in Figure 5, the information processing device 8 includes a processor 81, a main memory 82, an auxiliary storage device 83, and an I / F (interface) circuit 84. The processor 81, the main memory 82, the auxiliary storage device 83, and the I / F circuit 84 are connected to each other so as to be able to communicate with one another, for example, via a bus 89. Note that each component of the information processing device 8 may be realized by a combination of two or more components.

[0053] The information processing device 8 has at least one processor 81 and at least one main memory 82 (memory), and has a hardware configuration that utilizes a normal computer. For example, at least one processor 81 loads a program stored in the auxiliary storage device 83 into the main memory 82 and executes the loaded program to realize each function of each device. Note that at least one processor 81 of the information processing device 8 may be configured as a dedicated hardware circuit.

[0054] For example, in the operation management server 3, at least one processor 81 implements the functions of the operation management server 3, including a communication control unit 31, a management unit 32, and a decision unit 33. For example, in the operator terminal 5, at least one processor 81 implements the functions of the operator terminal 5, including a communication control unit 51, an input control unit 52, and an output control unit 53. For example, in the mobile unit 6, at least one processor 81 implements the functions of the mobile unit 6, including a communication control unit 61, a detection control unit 62, a driving control unit 63, and a speech control unit 64.

[0055] As at least one processor 81, various types of processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array) can be used as appropriate.

[0056] Various types of memory, such as RAM (Random Access Memory), can be used as the main memory 82 as appropriate.

[0057] As the auxiliary storage device 83, various recording media and recording devices such as ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), and Flash memory can be used as appropriate.

[0058] For example, in the operation management server 3, the operator terminal 5, and the mobile unit 6, at least one main memory device 82 and one auxiliary memory device 83 each realize their respective storage units (not shown).

[0059] The I / F circuit 84 is an interface for realizing input / output functions, connecting external devices, and / or communicating with the outside world. The I / F circuit 84 may also be an output interface for connecting or realizing output devices that output audio, images, or video; an input interface for connecting or realizing input devices that acquire user input, sensor detection results, audio, images, or video; or an interface that functions as one of these devices. Various displays such as LCD displays, organic EL displays, and projectors, as well as speakers, can be used as output devices. Similarly, keyboards, mice, touch panels, microphones, cameras, and sensors can be used as input devices.

[0060] For example, in the operation management server 3, the I / F circuit 84 implements a communication interface (communication unit) for communicating with the outside. For example, in the operator terminal 5, the I / F circuit 84 implements an output interface (output unit, display unit, speech unit) for connecting or implementing output devices, an input interface (input unit, operation unit, sound pickup unit) for connecting or implementing input devices, and a communication interface (communication unit) for communicating with the outside. For example, in the mobile unit 6, the I / F circuit 84 implements an output interface (output unit, display unit, speech unit) for connecting or implementing output devices, an input interface (input unit, imaging unit, sound pickup unit) for connecting or implementing input devices, and a communication interface (communication unit) for communicating with the outside.

[0061] With regard to the devices included in the remote control system 1 according to the above embodiment, only the configurations necessary for explaining the main parts of this embodiment are shown as examples, but the configurations of these devices are not limited to these.

[0062] Next, with reference to the drawings, an example of the operation of the remote control system 1 according to the embodiment will be described. Note that the operation procedure and processing flow described below are just examples, and it is possible to change the order of the steps, delete some steps, or add other steps as desired.

[0063] Figure 6 is a flowchart showing an example of the processing flow related to the delegation of authority according to the embodiment. Figure 7 is a diagram illustrating the processing related to the delegation of authority according to the embodiment.

[0064] As an example, the flow shown in Figure 6 is executed when operator P1 is remotely controlling (monitoring) at least one mobile object 6 assigned to him while viewing the screen display on operator terminal 5-1. This display screen includes at least one remote control image (video) based on video data received from each of the at least one mobile object 6 being controlled (see Figure 9).

[0065] As an example, at the start of the flow shown in Figure 6, operator P2 is monitoring the status of multiple mobile objects 6, as illustrated in "Automatic Driving" in Figure 7. Furthermore, each of the multiple mobile objects 6 being monitored by operator P1 is assumed to be in an automatic driving state.

[0066] As an example, at the start of the flow shown in Figure 6, operator P2 is not performing remote control such as monitoring, as illustrated in "Automatic Driving" in Figure 7.

[0067] First, the operation management server 3 performs the surrounding traffic participant detection and notification process (S101). Subsequently, the communication control unit 31 in the operation management server 3 obtains the result of operator P1's decision on whether or not to take action (operation input for action decision) from operator terminal 5-1 via the I / F circuit 84. Then, the decision unit 33 in the operation management server 3 determines, based on the obtained decision result from operator P1 on whether or not to take action (remote intervention) such as monitoring, operation, or driving by the operator (S102), or whether to transfer authority to another operator (S103).

[0068] The following explanation will illustrate a case where an arbitrary moving object 6 being monitored by operator P1 transitions to a stopped state and issues a remote request upon detection of surrounding traffic participants, as illustrated in Figure 7, "When a remote control request is issued."

[0069] Here, we will explain the processing of the operation management server 3 in S101 to S103, and an example of the operation of the remote control system 1 associated with said processing. Figure 8 is a sequence diagram showing an example of the processing flow related to the delegation of authority according to the embodiment.

[0070] In the mobile unit 6, the detection control unit 62 detects the approach of a pedestrian (participant in surrounding traffic) to the mobile unit 6 via the I / F circuit 84 (S1011). Here, the approach of a pedestrian (participant in surrounding traffic) to the mobile unit 6 is a situation that issues a remote request for response (remote intervention) from operator P. In the mobile unit 6, the driving control unit 63 stops the mobile unit 6 if it is in motion. Also, in the mobile unit 6, the speech control unit 64 makes a speech to inform the detected participant in the surrounding traffic that the mobile unit 6 is nearby (S1012). The content of this speech is, for example, pre-set and stored in the internal memory of the mobile unit 6.

[0071] In the mobile unit 6, the communication control unit 61 outputs detection information (remote request) to the operation management server 3 via the I / F circuit 84. In the operation management server 3, the communication control unit 31 acquires detection information (remote request) from the mobile unit 6 via the I / F circuit 84 (S1013).

[0072] In the operation management server 3, the decision unit 33 determines whether or not an authority transfer alert is necessary according to the scene of the mobile body 6 that issued the remote request (S1014).

[0073] For example, the decision unit 33 may determine that an authority transfer alert is unnecessary in a scene that can be handled by a speech from the mobile device 6. In this case, it is expected that speech control will be performed after the alert notification in a scene that can be handled by a speech from the mobile device 6. For this reason, a scene that can be handled by a speech from the mobile device 6 can be described as a scene in which speech control is expected to be performed.

[0074] For example, the decision unit 33 may determine that an authority transfer alert is necessary in situations where the situation cannot be handled solely by the speech from the mobile device 6.

[0075] For example, the decision unit 33 may determine that an authorization delegation alert is necessary in a situation where attention, including monitoring, is required.

[0076] For example, the decision unit 33 may determine that an authority delegation alert is necessary in a scenario where action, including driving, is required.

[0077] For example, consider a scenario where the mobile vehicle 6 is stopped in front of a garage, obstructing surrounding traffic. In such a scenario, promptly restarting the mobile vehicle 6 and moving it will eliminate the obstruction. Furthermore, restarting the mobile vehicle 6 without requiring constant observation can be easily instructed via the UI of the operator terminal 5 (see Figure 9). For this reason, scenarios where the stopped mobile vehicle 6 is obstructing surrounding traffic may be pre-configured as having "high priority" and "no authority delegation alert required."

[0078] For example, consider a scenario where mobile object 6 crosses a busy road. In such a scenario, it is not necessary to immediately resume movement of mobile object 6, but it is necessary to monitor it closely and ensure safety before resuming movement. For this reason, the scenario of mobile object 6 crossing a busy road may be pre-configured as "low response priority" and "authority delegation alert required."

[0079] For example, consider a scenario where there is no space to pass, or where passing is difficult, and a specific traffic participant is approaching. Here, a specific traffic participant may be a person who may make unpredictable movements, i.e., a person whose behavior is unpredictable, such as a child. Alternatively, a specific traffic participant may be a person in a wheelchair, a person with a white cane, or a person using a cane, who should be given priority in passing. In the remote control system 1 according to this embodiment, the moving object 6 reduces the risk of accidents by notifying the surroundings of its presence and approach by sending sound information (for example, S1012 in Figure 8), and ensures safe and secure movement of the moving object 6 for pedestrians (surrounding traffic participants) who are walking while operating a smartphone or walking while looking away. In order to achieve such safe and secure movement, the moving object 6 is further required to adjust its speed to account for children suddenly running out into the road, and to yield the way so as not to obstruct the passage of people with disabilities. Therefore, in situations where there is no space to pass or passing is difficult, and a specific traffic participant is approaching, it may be pre-set as "high priority for response" and "authority delegation alert required."

[0080] For example, consider a scenario where there is ample space for passing, such as on a wide road, or where passing is easy, and there are other traffic participants approaching the moving object 6, and these participants are aware of the moving object 6. Such scenarios may be pre-configured as having "low response priority" and "no authority delegation alert required."

[0081] Furthermore, the determination of whether or not an authority delegation alert is necessary is not limited to a determination based on the location information of the mobile body 6, but may also be determined based on the service status of the mobile body 6, such as being in transit or being moved. For example, the determination unit 33 may determine that an authority delegation alert is necessary if the service status of the mobile body 6 is in transit. For example, the determination unit 33 may determine that an authority delegation alert is unnecessary if the service status of the mobile body 6 is being moved.

[0082] Furthermore, the determination of whether or not an authority delegation alert is necessary may be based on the operator status of the target mobile object 6, such as proficiency level, past response record, and fatigue level. For example, the determination unit 33 may determine that an authority delegation alert is necessary if the proficiency level is lower than a predetermined threshold, there is no past response record or the record is less than a predetermined threshold, or the fatigue level is higher than a predetermined threshold. For example, the determination unit 33 may determine that an authority delegation alert is unnecessary if the proficiency level is higher than a predetermined threshold, there is no past response record or the record is more than a predetermined threshold, or the fatigue level is lower than a predetermined threshold.

[0083] Furthermore, the determination of whether or not an authority delegation alert is necessary may be based on at least one of the following: the urgency of the response, whether or not the operator P2 of the transfer destination or candidate for the transfer is currently responding to a mobile object 6, and, if operator P2 is currently responding, the estimated time when that response will be completed. For example, the determination unit 33 may determine that an authority delegation alert is necessary if the urgency of the response is higher than a predetermined threshold, there is no mobile object 6 that operator P2 of the transfer destination or candidate for the transfer is currently responding to, or, if operator P2 is currently responding, the estimated time when that response will be completed is shorter than the predetermined threshold. For example, the determination unit 33 may determine that an authority delegation alert is unnecessary if the urgency of the response is lower than a predetermined threshold, there is a mobile object 6 that operator P2 of the transfer destination or candidate for the transfer is currently responding to, or, if operator P2 is currently responding, the estimated time when that response will be completed is longer than the predetermined threshold.

[0084] Now, referring again to Figures 6 and 7, we will continue to explain the process flow related to the delegation of authority according to the embodiment.

[0085] If operator intervention (remote intervention) is required (S102: Yes) and authority is not to be delegated (S103: No), the operation management server 3 executes the response process based on the instructions of operator P1 (first operator) (S104). Subsequently, the process shown in Figure 6 proceeds to the process in S108.

[0086] As an example, in response processing based on instructions from operator P1 (first operator), the management unit 32 acquires the response instructions (speech instructions) from operator P1 via the I / F circuit 84 from the operator terminal 5-1. The communication control unit 31 then outputs a command (instruction) indicating the acquired response instructions from operator P1 to the mobile device 6 via the I / F circuit 84. When the mobile device 6 receives these response instructions, it repeats the speech from its speaker. The content of this repeated speech may be the same as the previous speech (for example, S1012 in Figure 8) or it may be different.

[0087] If operator intervention (remote intervention) is required (S102: Yes) and authority is to be transferred (S103: Yes), the operation management server 3 executes the authority transfer process from operator P1 (first operator) to operator P2 (second operator) (S105).

[0088] As an example, in the process of transferring authority from operator P1 to operator P2, the decision unit 33, upon receiving an authority transfer request command from operator terminal 5-1 via the I / F circuit 84 in response to an authority transfer alert from operator P1, for example, an authority transfer request (authority transfer instruction), determines which operator P2 will receive the authority. The communication control unit 31 also outputs a notification requesting acceptance of the authority transfer to operator terminal 5-2 used by the determined recipient operator P2. At this time, operator terminal 5-2 may display the notification requesting acceptance of the authority transfer. Then, when the decision unit 33 receives an authority acceptance command from operator terminal 5-2 via the I / F circuit 84 in response to an operation by operator P2 who has accepted the authority transfer, it transfers the authority of the target mobile object 6 from operator P1 to operator P2, as illustrated in "Before remote operation starts" in Figure 7. Furthermore, the communication control unit 31 outputs a power transfer completion notification to operator terminal 5-1 to notify that the power transfer has been completed, and outputs a power acceptance completion notification to operator terminal 5-2.

[0089] Thus, the authority delegation process according to this embodiment is configured such that the authority remains with the requesting operator P1 until the delegation is approved by the receiving operator P2. This configuration makes it possible to suppress the occurrence of a situation where no one is monitoring during the authority delegation, i.e., a situation where a response requiring immediate action cannot be taken.

[0090] The decision unit 33 may, for example, determine an operator P2 who does not have a mobile object 6 currently being handled as the transfer destination. Alternatively, the decision unit 33 may, for example, determine the operator P2 with the shortest estimated time to complete the handling of a mobile object 6 among the operators P2 who are currently handling a mobile object 6 as the transfer destination. Alternatively, the decision unit 33 may, for example, determine a transfer destination from operators P2 whose estimated time to complete the handling is shorter than a predetermined threshold. Alternatively, the decision unit 33 may, for example, determine an operator P2 with high skill level, past handling experience or a lot of experience, and low fatigue level as the transfer destination, similar to the determination of whether or not an authority transfer alert is necessary. Alternatively, the decision unit 33 may, for example, notify multiple operators P2 of the authority transfer request and determine the operator P2 who performs the operation (instruction) to accept the authority transfer first as the transfer destination.

[0091] Furthermore, after authority is transferred to operator P2 (second operator), the operation management server 3 executes the corresponding processing based on the instructions of operator P2 (second operator) (S106).

[0092] As an example, in response processing based on instructions from operator P2 (second operator), the management unit 32, as illustrated in "When remote operation is performed" in Figure 7, receives the response instructions from operator P2 via the I / F circuit 84 from operator terminal 5-2, and outputs those response instructions to the mobile unit 6 via the I / F circuit 84. These response instructions are the instructions that operator P2 inputs to operator terminal 5-2 based on the result of their judgment after determining the content of the response. For example, if operator P2 instructs a response by voice, a voice command (voice instruction) is output from operator terminal 5-2 to mobile unit 6 via the operation management server 3. For example, if operator P2 instructs remote driving for a right-of-way response, a driving command (driving instruction) is output from operator terminal 5-2 to mobile unit 6 via the operation management server 3. For example, if operator P2 instructs a response by making a call with a participant in surrounding traffic, a call command (call instruction) is output from operator terminal 5-2 to mobile unit 6 via the operation management server 3.

[0093] Furthermore, after completing the response processing based on the instructions of operator P2 (second operator), the operation management server 3 executes the authority transfer process from operator P2 (second operator) to operator P1 (first operator) (S107). Subsequently, the flow shown in Figure 6 proceeds to the process in S108.

[0094] As an example, in the process of transferring authority from operator P2 to operator P1, the operation management server 3 transfers the authority of the target mobile object 6 from operator P2 to operator P1 in the same manner as the process of transferring authority from operator P1 to operator P2, as illustrated in "Automatic Driving Restoration" in Figure 7.

[0095] If operator intervention (remote intervention) is not required (S102: No), or if the response process is based on instructions from operator P1 (first operator) (S104), or if the response process is based on instructions from operator P2 (second operator) with delegated authority (S105-S107), the operation management server 3 executes the automatic driving restart process (S108).

[0096] As an example, in the automatic driving restart process, the operation management server 3 outputs a command (instruction) indicating the driving restart instruction from operator P1 to the mobile body 6 via the I / F circuit 84, in the same manner as the response process based on the instructions of operator P1, as illustrated in "Automatic Driving Restoration" in Figure 7. Note that the automatic driving restart process may be performed prior to the authority transfer process from operator P2 to operator P1, i.e., it may be performed by operator P2.

[0097] Figure 9 shows an example of the display of a delegation alert in the operator terminal 5 according to this embodiment. Figure 9 illustrates the screen display on the operator terminal 5-1 used by operator P1 (first operator) for remote control of the mobile device 6.

[0098] The screen display 900 in Figure 9 includes multiple areas 901 for remote control of multiple mobile units 6. Each area 901 displays the status 905 of the mobile unit 6 and multiple video feeds of the area surrounding the mobile unit 6. These video feeds may also be configured to function as operation buttons for instructing remote control. For example, in the example in Figure 9, the "Video (Front)" in area 901 is an operation button for instructing the "Start Driving" operation. For example, in the example in Figure 9, the "Video (Left Rear)" in area 901 is an operation button for instructing the "Excuse me" utterance. For example, in the example in Figure 9, the "Video (Right Rear)" in area 901 is an operation button for instructing the "Please go ahead" utterance.

[0099] On the screen display of the operator terminal 5-1 for remote control of the mobile object 6, a delegation alert proposing the delegation of authority may be displayed. For example, in the area 901 of the mobile object 6 with "ID: 051" on the screen display 900 in Figure 9, a delegation alert 911a labeled "Traffic Limiting" is displayed. As an example, this "Traffic Limiting" delegation alert 911a is a notification that the mobile object 6 is obstructing surrounding traffic participants and proposes delegating authority to address the issue.

[0100] Note that the delegation of authority alert 911a may differ in its display from other notifications. For example, the delegation of authority alert 911a may differ from other notifications in terms of the color, wording, icon, and whether or not it flashes.

[0101] Furthermore, the screen display on the operator terminal 5-1 for remote control of the mobile unit 6 may display information according to the response priority. For example, in the screen display 900 of Figure 9, the area 901 for each mobile unit 6 with "ID: 071, 076, 079" displays a notification 911b in response to a remote request for "safety confirmation," that is, a normal alert without a recommendation for delegation of authority.

[0102] Furthermore, in the screen display on the operator terminal 5-1 for remote control of the mobile object 6, alerts may be displayed for items that can be addressed immediately. Also, if, for example, operator P2 is already dealing with another mobile object 6 and is unable to address the target mobile object 6, the estimated time until operator P2 can complete the response, or a notification prompting operator P1 to pay attention to the situation, may be displayed.

[0103] Furthermore, it is conceivable that after operator P1 has handled the situation, authority may be transferred to operator P2 for further action. In such scenarios, operator P2 may need time to grasp the situation. Also, operator P2 may perform actions that overlap with those taken by operator P1, potentially resulting in a loss of time for essential actions. For example, consider a scenario where a pedestrian wearing headphones and unable to hear the spoken voice from the mobile device 6 approaches the device without noticing it. In such a scenario, where operator P1 has issued a verbal warning but the pedestrian continues to approach without noticing the mobile device 6, operator P1 may request that authority be transferred to operator P2. However, in such a scenario, there is a high possibility that operator P2's verbal warning after the transfer of authority will also go unnoticed, potentially delaying the start of the essential action of remotely controlling the vehicle to yield the right of way. Therefore, in scenarios where operator P1 handles a request and then delegates authority to operator P2, the operator terminal 5-2 screen on operator P2's side may display information indicating the content of the request already handled by operator P1, such as a log of the operation performed, at the time of the authority delegation. Alternatively, the display of information indicating the content of the request already handled by operator P1 may be performed at the time of the authority delegation request.

[0104] If it is not possible to transfer authority over the target mobile object 6 to operator P2, operator P1 may stop other mobile objects 6 that are under surveillance, and operator P1 may perform monitoring, remote control with monitoring, or remote operation of the target mobile object 6.

[0105] Thus, the remote control system 1 relating to this disclosure is configured to display an authority delegation alert proposing the delegation of authority to the mobile body 6 that issued the remote request, depending on whether an alert associated with that scene is necessary.

[0106] This configuration supports the decision-making process regarding the delegation of authority by operator P1, who is monitoring multiple mobile devices 6. More specifically, operator P1, who is monitoring multiple mobile devices 6, can easily determine whether delegation of authority is necessary based on the alert display, thereby reducing the time required for that decision. Reducing the time required for the decision on delegation of authority contributes to suppressing response delays.

[0107] The following describes other embodiments of the remote control system 1 relating to this disclosure. The following description will mainly explain the differences from the first embodiment, and redundant explanations will be omitted as appropriate.

[0108] (Second Embodiment) Figure 10 shows another example of the configuration of the remote control system 1 according to the embodiment. The remote control system 1b (remote control system 1) illustrated in Figure 10 includes an operation management server 3, a database 4, an operator terminal 5, and a mobile unit 6.

[0109] Database 4 stores, for example, operator information, including the information of the operations manager and the information of the remote driver.

[0110] Here, the operations manager is operator P1, who primarily monitors and, when necessary, performs remote operations without direct observation. For example, confirmation requests and control requests from the operations management server 3 or mobile unit 6 are first notified to the operations manager. For example, if the operations manager determines that it is necessary for operator P2 (remote driver) to take action, such as remote operation, they will request operation (request for delegation of authority) from the remote driver.

[0111] As an example, the operations manager monitors the video feed from the 6 mobile units and the confirmation and control requests from the mobile units 6 and the operations management server 3, and performs tasks such as issuing stop orders, start (resume) orders, and requesting operation from remote drivers (requests for delegation of authority) as needed.

[0112] For example, if a mobile vehicle 6 becomes immobile due to a jam, construction, demonstration, etc., the operations manager will perform operational decision tasks as appropriate. For example, operational decisions include contacting the service provider (servicer) of the mobile vehicle 6 in the event of immobility. For example, operational decisions include instructions regarding the mobile vehicle 6 in the event of continued operation, such as micro-level instructions like rerouting or remote intervention, and macro-level instructions such as changes to the charging plan or decisions regarding such changes.

[0113] For example, a train operations manager is tasked with understanding the operational status and service quality based on delays to the schedule and related information, adjusting the operational volume, and communicating with various stakeholders, such as service providers and recipients, as needed.

[0114] The tasks assigned to the operations manager may be any one of the three tasks mentioned above, or a combination of two or more tasks.

[0115] In the example shown in Figure 10, there is an operations manager for each operating base of the mobile unit 6. Figure 10 illustrates multiple operations managers, including operator P1a, who monitors multiple mobile units 6 operating at operating base A, and operator P1b, who monitors multiple mobile units 6 operating at operating base B. Figure 10 also illustrates operator terminals 5-1a used by operator P1a and operator terminals 5-1b used by operator P1b.

[0116] Furthermore, a remote driver is an operator P2 who primarily performs remote control and remote driving, often involving constant observation. For example, a remote driver is an operator P2 who performs remote driving such as avoidance maneuvers, gives instructions to resume driving at crosswalks, and handles phone calls such as conversations with traffic controllers.

[0117] In the example shown in Figure 10, the remote operator can remotely control the mobile unit 6 regardless of which base station they are at. Figure 10 illustrates operators P2a and P2b as multiple remote operators. Figure 10 also illustrates operator terminal 5-2a used by operator P2a and operator terminal 5-2b used by operator P2b.

[0118] As described above, the remote control system 1 according to this embodiment is configured to display an authority transfer alert regarding the transfer of authority from operator P1, who is the operations manager, to operator P2, who is the remotely driven vehicle.

[0119] Even with this configuration, the same effects as the embodiment described above can be obtained. Furthermore, since operator P1 is primarily a flight manager who performs monitoring, the number of mobile units 6 that are monitored can be significantly increased, improving the efficiency of remote control of multiple units.

[0120] (Third Embodiment) In order to realize the smooth movement of autonomous vehicles, appropriate behavior of the autonomous vehicle toward surrounding traffic participants is expected. In this context, when the surrounding traffic participants are children (specific traffic participants), they may make movements that are particularly difficult to predict. In addition, the surrounding traffic participants may be people with disabilities (specific traffic participants) who require consideration when passing. In order to realize the safe and secure movement of autonomous vehicles toward these specific traffic participants, the autonomous vehicle is required to behave in ways such as adjusting its speed to account for sudden movements and yielding the right of way so as not to obstruct the passage of people with disabilities.

[0121] For example, for pedestrians walking while operating a smartphone (participants in surrounding traffic) or pedestrians walking while looking away (participants in surrounding traffic), the autonomous mobile device can notify those around it of their presence and approach, thereby reducing the risk of accidents and ensuring safe and secure travel. On the other hand, if the autonomous mobile device may obstruct a specific participant in traffic, it must actively behave in a way that prioritizes the passage of that participant.

[0122] However, conventionally, due to the driving performance of autonomous vehicles, it has sometimes been impossible to drive in a way that takes into account specific traffic participants whose passage should be prioritized by society.

[0123] For example, when passing children whose behavior is unpredictable or people in wheelchairs on narrow roads, it is necessary to adjust the autonomous vehicle's speed and make adjustments to decisions such as driving, stopping, and rerouting, depending on the surrounding traffic participants and driving environment.

[0124] For example, if an event occurs that an autonomous mobile device cannot handle automatically, the autonomous mobile device or the remote control system needs to detect the event and notify the operator in order to request an override.

[0125] For example, an autonomous vehicle that does not classify surrounding traffic participants may stop or adjust its speed to ensure the same clearance for all traffic participant classes as for specific traffic participants in order to mitigate accident risk. Alternatively, such an autonomous vehicle may stop or adjust its speed with a uniform stopping clearance for all traffic participants without ensuring the clearance necessary to mitigate accident risk. In these cases, there is a risk of excessive stopping or deceleration, or that the risk of accidents is not adequately mitigated.

[0126] For example, an autonomous vehicle that does not classify surrounding traffic participants, or an autonomous vehicle that does not perform semantic recognition using camera images, cannot recognize situations where passage is impossible due to an event being held or children playing within the driving area. Furthermore, children playing (specific traffic participants) or people participating in an event (specific traffic participants) may have a lower level of awareness of their surroundings than usual. For this reason, if the vehicle does not stop with a greater clearance than usual to mitigate the risk of accidents involving specific traffic participants, the risk of accidents may not be sufficiently reduced.

[0127] For example, a stationary autonomous vehicle could potentially obstruct traffic for specific traffic participants that should have priority in society, or for vehicles exiting a parking lot (specific traffic participants).

[0128] Therefore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile unit 6 are configured to detect specific traffic participants through image recognition of video footage from a camera mounted on or connected to the mobile unit 6.

[0129] As an example, the decision unit 33 of the operation management server 3 and / or the detection control unit 62 of the mobile body 6 are configured to input camera images to a machine learning model whose parameters have been learned to classify surrounding traffic participants, and to acquire the class identification results output from the machine learning model in response to the input camera images.

[0130] The recognition process for surrounding traffic participants using a pre-trained class identification model may be performed by the mobile device 6 or by the operation management server 3 that acquires the camera images. The execution of the recognition process means inputting the camera images into the model and obtaining identification results corresponding to the input. The class identification (inference) by the model may be performed internally or externally. Similarly, the pre-trained class identification model may be stored in the internal memory of the mobile device 6, in the internal memory of the operation management server 3, or in an external information processing device such as a cloud server.

[0131] Furthermore, "specific traffic participants" may also include surrounding traffic participants who may make unpredictable movements, that is, children and other people whose behavior is unpredictable, and / or surrounding traffic participants who should be given priority in passage, such as people in wheelchairs, people with white canes, and people using canes.

[0132] Furthermore, in the remote control system 1 according to this embodiment, when the positional relationship between a specific traffic participant and the target moving object 6 approaches, the operation management server 3 notifies the operator P1 with a high-priority normal alert or authority delegation alert, thereby prompting a response through remote intervention and reducing the risk of a collision.

[0133] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile body 6 may detect a specific traffic participant, and if the travel paths of the specific traffic participant and the target mobile body 6 intersect, the stopping clearance (margin) may be changed to be larger than usual, i.e., larger than that of other surrounding traffic participants of the specific traffic participant.

[0134] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile body 6 may detect a specific traffic participant, and if the travel paths of the specific traffic participant and the target mobile body 6 intersect, they may change the travel path.

[0135] In particular, when vehicles are traveling on narrow roads and passing oncoming vehicles, there is a high possibility that their paths will overlap, increasing the risk of accidents. For this reason, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile unit 6 may stop, move to the side, or reroute with a larger margin than usual.

[0136] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile vehicle 6 may be configured to perform voice control to notify the surroundings of at least one of the following intentions: "notification of the presence of the target mobile vehicle," "notification of the intention to yield the right of way," "notification of the intention to change the direction of travel," and "the status of the remote operator's response," along with the stopping clearance and / or change of the route. With this configuration, it is possible to stop or drive with sufficient clearance from the travel route of the specific traffic participant, thereby reducing the risk of collisions even for specific traffic participants.

[0137] Furthermore, in the remote control system 1 according to this embodiment, if there are surrounding traffic participants whose driving paths intersect with the target mobile vehicle 6 due to the mobile vehicle 6 and / or the surrounding traffic participants turning right or left, the operation management server 3 and / or the mobile vehicle 6 may detect an obstacle between the target mobile vehicle 6 and the surrounding traffic participants. If an obstacle is found, the operator may be notified of the presence of the target mobile vehicle 6 by notifying the surroundings, using voice control, and adjusting the lights, and the target mobile vehicle 6 may be stopped as appropriate. This voice control may include, for example, a statement such as, "A remote operator will respond shortly. In case of emergency, you can operate the brake bar to manually move the vehicle."

[0138] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or the mobile body 6 may perform at least one of the following actions depending on the driving environment information, which includes at least one of the traffic volume and noise level of the driving environment of the target mobile body 6, and the class of the surrounding traffic participants: control of the speed of the target mobile body 6, control of driving and stopping, adjustment of notification sounds including volume, and adjustment of the illumination of mounted lights. With this configuration, the risk of noise from the speech sounds of the target mobile body 6 can be reduced while notifying the surroundings of the presence of the target mobile body 6, thereby reducing the risk of contact.

[0139] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or the mobile body 6 may recognize scenes where passage is prohibited, such as when children are playing or when an event is being held, using at least one of the following: the class of traffic participants around the target mobile body 6, the number of traffic participants, the width of the road in the driving area, and recognition information processed by camera sensors and distance measuring sensors. In addition, in the remote control system 1 according to this embodiment, the operation management server 3 and / or the mobile body 6 may perform autonomous movement stop control and speed adjustment based on the recognition results.

[0140] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile unit 6 may use sound information such as surrounding voices and video information from cameras as input data for a learned class identification model to recognize scenes such as approaching specific traffic participants who should be given priority in social traffic, or interacting with traffic controllers.

[0141] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or mobile unit 6 may, in a situation where the target mobile unit 6 is unable to drive autonomously, make a request for approval to the operator P1 to propose an alternative route, and if the operator P1 approves, autonomous driving using the alternative route may be performed.

[0142] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or the mobile body 6 may estimate the travel paths of vehicles (specific traffic participants) that are within a predetermined range around the target mobile body 6 and / or on the travel path. Also, in the remote control system 1 according to this embodiment, if the predicted travel point of the vehicle and / or the current location of the vehicle is a parking lot, and the target mobile body 6 is on the travel path of the vehicle, the operation management server 3 and / or the mobile body 6 may perform evacuation control while performing voice control to indicate the intention to quickly move the target mobile body 6 and / or light control to signal with lights. This configuration makes it possible to reduce the risk of traffic being limited by the target mobile body 6.

[0143] Furthermore, in the remote control system 1 according to this embodiment, the operation management server 3 and / or the mobile body 6 may use at least one of the following to determine whether to adjust the speed of the target mobile body 6, stop it, or continue driving: information on the number of children (specific traffic participants) present within a predetermined range around the target mobile body 6 and / or along its path; information on the children's paths; information on the children's speeds; and behavioral status recognition information such as "blocking the road while playing" or "touching the target mobile body." This configuration can reduce the risk of accidents involving children.

[0144] Here, an example of the operation of the remote control system 1 according to this embodiment will be described. Figure 11 is a flowchart showing another example of the processing flow related to the delegation of authority according to this embodiment. Here, the processing mainly performed by the operation management server 3 is illustrated. In the same manner, processing may be performed by the mobile unit 6, or by their cooperation.

[0145] The communication control unit 31 acquires camera images of the surrounding area of ​​the target mobile object 6 from the target mobile object 6 via the I / F circuit 84. The determination unit 33 inputs the acquired camera images into a pre-learned class identification model and obtains the class identification results of surrounding traffic participants within a predetermined range. The determination unit 33 also acquires travel path information for each of the surrounding traffic participant groups within the predetermined range (S201). The determination unit 33 also determines whether the stopping position or travel path of the target mobile object 6 intersects with the travel path of a specific traffic participant (S202), or whether the target mobile object 6 is stopped (S203).

[0146] If the stopping position or path of the target moving object 6 does not intersect with the path of the designated traffic participant (S202: No), the flow in Figure 11 returns to the process in S201. On the other hand, if the stopping position or path of the target moving object 6 intersects with the path of the designated traffic participant (S202: Yes), and the target moving object 6 is in motion (S203: No), the management unit 32 stops the target moving object 6 (S204). After that, or if the target moving object 6 was already stopped (S203: Yes), the judgment unit 33 determines whether there is sufficient road width for the designated traffic participant to avoid the stopped target moving object 6 (S205).

[0147] If there is sufficient road width for the designated traffic participant to avoid the stationary target vehicle 6 (S205: Yes), the management unit 32 causes the target vehicle 6 to perform at least one of the following verbal control actions: "intention to stop" and "intention to yield the right of way," thereby notifying those around it of its presence (S206). Subsequently, the judgment unit 33 determines whether the paths of the target vehicle 6 and the designated traffic participant intersect (S207). If the paths of the target vehicle 6 and the designated traffic participant do not intersect (S207: No), the process in Figure 11 proceeds to S111.

[0148] If there is not enough road width for a designated traffic participant to avoid the stopped target vehicle 6 (S205: No), or if the paths of the target vehicle 6 and the designated traffic participant intersect (S207: Yes), the management unit 32 causes the target vehicle 6 to perform a speech control indicating at least one of the following: "the intention to stop" and "the intention to open a path for you to pass" (S2086). The judgment unit 33 also causes the operator terminal 5-1 to notify operator P1 of a traffic bottleneck alert (authority delegation alert) and requests remote intervention from operator P2 (S209). When operator P2 approves the intervention request, remote intervention begins (S210). Subsequently, the process in Figure 11 proceeds to S111, where remote intervention is performed by the authorized operator P, and the process ends when the traffic bottleneck is resolved (S211).

[0149] Thus, the remote control system 1 according to this embodiment can classify surrounding traffic participants and enable the movement of the mobile vehicle 6 while actively considering the presence of specific traffic participants. This makes it possible to realize safe and secure movement of the autonomous mobile vehicle.

[0150] The authority transfer process according to this embodiment may be implemented by the remote control system 1a according to the first embodiment, or by the remote control system 1b according to the second embodiment.

[0151] In each of the embodiments described above, the determination of "whether or not it is A" may be achieved by determining only whether it is "A", by determining only whether it is "not A", or by determining both.

[0152] In each of the embodiments described above, "any of A" means "at least one of A".

[0153] Furthermore, the programs executed by each device of the remote control system 1 according to each embodiment described above may be provided as files in an installable or executable format, recorded on a computer-readable recording medium (Computer Program Product) such as a CD-ROM, floppy disk, CD-R, or DVD.

[0154] Furthermore, the programs executed by each device of the remote control system 1 according to each embodiment described above may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs executed by each device of the remote control system 1 according to each embodiment described above may be provided or distributed via a network such as the Internet.

[0155] Furthermore, the programs executed by each device of the remote control system 1 according to each of the above embodiments may be pre-installed and provided in ROM or the like.

[0156] According to at least one embodiment described above, it is possible to support decisions regarding the delegation of authority by an operator monitoring multiple mobile objects.

[0157] While embodiments of this disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0158] (Note) The above description of embodiments discloses the following technology: (1) An information processing method performed by an information processing system that supports remote control by multiple operators of multiple mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the multiple operators include a first operator and a second operator, and when a target mobile body requiring remote intervention by an operator occurs from among two or more mobile bodies under monitoring for which the first operator holds the authority to remotely control, the system determines the scene of the remote intervention based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants, and determines whether an authority transfer alert is necessary to propose the transfer of authority to the second operator from the first operator's authority to remotely control the target mobile body, and in the case of a scene requiring an authority transfer alert, the system notifies the authority transfer alert via the operator terminal of the first operator, which is displaying video from each of the two or more mobile bodies under monitoring. (2) The information processing method described in (1) above, which stores in advance information indicating the correspondence between the remote intervention scene, the priority of the response to the remote intervention, and whether or not the authority transfer alert is necessary. (3) The information processing method described in (1) or (2) above, which determines that the authority transfer alert is unnecessary in the case of the remote intervention scene in which it is assumed that a response to speech control to cause the target mobile body to speak is performed. (4) The information processing method described in any one of (1) to (3) above, which determines that the authority transfer alert is necessary in the case of the remote intervention scene in which a response including at least one of the operator watching the video from the target mobile body and driving is performed. (5) The information processing method described in any one of (1) to (4) above, which determines that the authority transfer alert is necessary when a specific traffic participant present around the target mobile body is detected by image recognition processing of the video from the target mobile body.(6) The information processing method according to (5) above, which, along with the authority transfer alert, stops, moves to the side or reroutes with a larger margin than in the case of other surrounding traffic participants of the specified traffic participant. (7) The information processing method according to any one of (1) to (6) above, which, if there is no second operator to whom the authority of the first operator to remotely control the target traffic object can be transferred, stops all other traffic objects among the two or more traffic objects being monitored, and the remote intervention includes at least one of the first operator watching the video from the target traffic object and driving. (8) The information processing method according to any one of (1) to (6) above, which, the first operator is a traffic manager who monitors the video from the two or more traffic objects and performs operations without watching, and the second operator is a remote driver who performs operations with watching the video from the target traffic object to which the authority of remote control has been transferred from the first operator, and driving. (9) An information processing device that supports remote control by multiple operators of multiple mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the multiple operators include a first operator and a second operator, and the information processing device displays video from each of two or more mobile bodies under monitoring for which the first operator has the authority to remotely control, and when a target mobile body requiring remote intervention by an operator occurs among the two or more mobile bodies under monitoring, the information processing device displays an alert, along with video from the target mobile body, which is determined to be necessary according to the scene of the remote intervention, determined based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants, and which proposes a transfer of authority to transfer the authority of the first operator to remotely control the target mobile body to the second operator.(10) A program to be executed by a computer that supports the remote control of a plurality of operators to a plurality of mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the plurality of operators include a first operator and a second operator, and when a target mobile body requiring remote intervention by an operator occurs from among two or more mobile bodies under monitoring for which the first operator holds the authority to remotely control, the program determines the scene of the remote intervention based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants, the program determines whether an authority delegation alert is necessary to propose a delegation of authority to transfer the authority of the first operator to the second operator for remote control of the target mobile body, and if an authority delegation alert is necessary in the scene, the program notifies the authority delegation alert via the operator terminal of the first operator, which is displaying video from each of the two or more mobile bodies under monitoring. (11) An information processing device comprising at least one processor and at least one memory, wherein the at least one processor executes the information processing method described in any one of (1) to (8) above by executing a program stored in the at least one memory. (12) A program that causes a computer to execute the information processing method described in any one of (1) to (8) above, or a computer-readable non-transient recording medium in which the program is stored.

[0159] 1 Remote control system 3 Operation management server 31 Communication control unit 32 Management unit 33 Decision unit 4 Database 5 Operator terminal 51 Communication control unit 52 Input control unit 53 Output control unit 6 Mobile unit 61 Communication control unit 62 Detection control unit 63 Driving control unit 64 Speech control unit 8 Information processing unit 81 Processor 82 Main memory 83 Auxiliary memory 84 I / F circuit

Claims

1. An information processing method performed by an information processing system that supports remote control by multiple operators of multiple mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the multiple operators include a first operator and a second operator, and when a target mobile body requiring remote intervention by an operator occurs from among two or more mobile bodies under monitoring for which the first operator holds the authority to remotely control, the system determines the scene of the remote intervention based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants; determines, according to the determined scene, whether an authority delegation alert is necessary to propose a delegation of authority to transfer the authority of the first operator to remotely control the target mobile body to the second operator; and, if an authority delegation alert is necessary, the system notifies the authority delegation alert via the operator terminal of the first operator, which is displaying video from each of the two or more mobile bodies under monitoring.

2. The information processing method according to claim 1, wherein information indicating the correspondence between the remote intervention scene, the priority of the response to the remote intervention, and whether or not the authority delegation alert is necessary is stored in advance.

3. The information processing method according to claim 1, wherein in the remote intervention scenario in which it is assumed that speech control to cause the target mobile body to speak is performed, the authority delegation alert is deemed unnecessary.

4. The information processing method according to claim 1, wherein in the case of the remote intervention scene in which the operator observes the video from the target moving object and takes action including at least one of the following, the method determines that the authority delegation alert is necessary.

5. The information processing method according to claim 1, wherein if specific traffic participants present around the target moving object are detected by image recognition processing of the video from the target moving object, the method determines that the authority transfer alert is necessary.

6. The information processing method according to claim 5, which, along with the authority delegation alert, stops, moves to the side, or reroutes the specific traffic participant with a larger margin than in the case of other surrounding traffic participants.

7. If there is no second operator to whom the authority of the first operator to remotely control the target mobile body can be transferred, the information processing method according to claim 1, wherein the remote intervention includes at least one of the first operator watching the video feed from the target mobile body and driving it.

8. The information processing method according to claim 1, wherein the first operator is a traffic manager who monitors and operates without looking at the video from the two or more moving objects, and the second operator is a remote driver who operates and drives the video from the target moving object to which the authority for remote control has been transferred from the first operator.

9. An information processing device that supports the remote control of multiple operators to multiple mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the multiple operators include a first operator and a second operator, and the device displays video from each of two or more monitored mobile bodies for which the first operator has the authority to remotely control, and when a target mobile body requiring remote intervention by an operator occurs among the two or more monitored mobile bodies, the device displays an authorization delegation alert, along with video from the target mobile body, which proposes a delegation of authority to transfer the authority of the first operator to remotely control the target mobile body to the second operator, with the necessity of the remote intervention determined based on the scene of the remote intervention determined based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants.

10. A program to be executed by a computer that supports the remote control of multiple operators of multiple mobile bodies, each configured to move autonomously and perform predetermined tasks, wherein the multiple operators include a first operator and a second operator, and when a target mobile body requiring remote intervention by an operator occurs from among two or more mobile bodies under monitoring for which the first operator holds the authority to remotely control, the program determines the scene of the remote intervention based on at least one of the status of the target mobile body, the status of traffic participants around the target mobile body, and the relationship between the target mobile body and the traffic participants, determines whether an authority delegation alert is necessary to propose a delegation of authority to transfer the authority of the first operator to the second operator for remote control of the target mobile body, and if an authority delegation alert is necessary, the program notifies the authority delegation alert via the operator terminal of the first operator, which is displaying video from each of the two or more mobile bodies under monitoring.

Images