Information processing method, information processing device, and program

Abstract

An information processing method according to the present disclosure is executed by an information processing system that provides assistance for remote control by an operator of least one autonomous mobile body. The information processing method comprises: outputting, from a terminal of an operator, a plurality of videos of at least one autonomous mobile body and audio associated with the plurality of videos; identifying, as a video of interest, a video corresponding to a piece of audio of interest that satisfies a prescribed condition from among the plurality of videos if the audio of interest thereof is detected from among the audio associated with the plurality of videos; and changing a display mode of a display area of the video of interest within a display area of the terminal of the operator.

Description

Information processing method, information processing apparatus and program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program.

[0002] In recent years, the introduction of autonomous driving vehicles and remotely operated vehicles that can travel without requiring a driver to board the vehicle has been promoted in various locations such as public roads, factories, airports, and ports. In remote control targeting such autonomous driving vehicles, remotely operated vehicles, and remotely driven vehicles, a controller (operator) on the remote control side performs remote monitoring, operation, and driving while checking video and audio obtained for each of a plurality of controlled objects.

[0003] For example, Patent Document 1 discloses a technology that enables audio transmission between a specific participant among a plurality of participants even during a video conference held by the plurality of participants.

[0004] Japanese Unexamined Patent Application Publication No. 2015-046822

[0005] However, in remote control where a plurality of controlled objects are checked simultaneously, the large amount of information makes it difficult to recognize information on the controlled object side such as people, vehicles, and obstacles, and to identify and judge events requiring remote operation or remote driving, resulting in a problem that the work burden on the remote control side is heavy. Furthermore, even when there is a single controlled object, in remote control where a plurality of pieces of video and audio from the controlled object side captured or collected in different directions are checked, the work burden on the remote control side is similarly heavy.

[0006] The present disclosure has been made in view of the above, and one of the objects thereof is to reduce the work burden on the remote control side with respect to remote control using video and audio from the controlled object side.

[0007] The information processing method relating to this disclosure is an information processing method performed by an information processing system that supports remote control by an operator of at least one autonomous mobile object. The information processing method outputs a plurality of images of at least one autonomous mobile object and audio associated with the plurality of images from the operator's terminal, and when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of images, the image corresponding to the audio of interest among the plurality of images is identified as the image of interest, and the display mode of the display area of ​​the image of interest in the display area of ​​the operator's terminal is changed.

[0008] Figure 1 is a diagram showing an example of the configuration of a remote control system according to the embodiment. Figure 2 is a diagram showing an example of the hardware configuration of an information processing device that realizes each function of each device included in the remote control system according to the embodiment. Figure 3 is a flowchart showing an example of the flow of information processing performed by the remote control system according to the embodiment. Figure 4 is a diagram showing an example of screen display in remote control according to the embodiment. Figure 5 is a flowchart showing an example of the flow of remote control processing performed by the remote control system according to the embodiment. Figure 6 is a diagram showing an example of display layout information for the screen display in remote control according to the embodiment. Figure 7 is a diagram showing an example of split layout information for the screen display in remote control according to the embodiment.

[0009] Hereinafter, embodiments of the information processing method, information processing apparatus, information processing system, mobile device, program, and recording medium related to this disclosure will be described in detail with reference to the attached drawings.

[0010] In this disclosure, components having the same or substantially the same function as those described above in previously shown drawings are denoted by the same reference numerals, and explanations may be omitted as appropriate. Furthermore, even when representing the same or substantially the same parts, the dimensions and proportions may be shown differently in different drawings. In addition, for example, from the viewpoint of ensuring the readability of the drawings, reference numerals may be assigned only to the main components in the explanation of each drawing, and reference numerals may not be assigned to components having the same or substantially the same function as those described above in previously shown drawings.

[0011] In addition, in the descriptions of this disclosure, components having the same or substantially the same function may be distinguished by adding alphanumeric characters and / or symbols to the end of the reference numeral. Alternatively, if multiple components having the same or substantially the same function are not to be distinguished, they may be described together by omitting the alphanumeric characters and / or symbols to the end of the reference numeral.

[0012] The information processing system relating to this disclosure is a remote control system that remotely controls at least one autonomously drivable vehicle (autonomous mobile object) on the controlled side, including remote monitoring and / or control, by an operator on the remote control side. The remote control system provides various services through its remote control, such as delivery, security, cleaning, childcare, nursing care, sales, agricultural work, manufacturing, cargo handling, transportation, and construction. In this remote control system, vehicle information (mobile object information), including video and audio of the target vehicle (target mobile object), is presented to the operator on the remote control side, and the operator performs remote monitoring of the controlled vehicle based on the vehicle information. In addition, in the remote control system, an operator is assigned in response to a request for assistance (remote request) from a vehicle that requires remote control, and the assigned operator intervenes in the control of the vehicle, such as remote operation or remote driving.

[0013] For example, the controlled side transmits images (videos) captured by cameras or other devices mounted on the autonomous mobile unit to the remote control side. For example, the remote control side displays a screen containing images from the controlled side on the operator's terminal. The display screen contains at least one image (video) relating to at least one autonomous mobile unit that the operator is responsible for. The operator monitors the status of each of the at least one autonomous mobile unit they are responsible for while viewing the display screen on their terminal.

[0014] For example, if the controlled vehicle becomes unable to move autonomously, such as when an obstacle like a fallen object or a parked vehicle is detected in its path, the controlled vehicle sends a request for remote control assistance (remote request) to the remote control side. This assistance request may be an operation request requesting assistance through remote control based on the operator's operation instructions (remote instructions), or a driving request requesting assistance through remote driving based on the operator's driving instructions (remote instructions). For example, when the remote control side receives an assistance request from the controlled vehicle, it presents the operator with a remote instruction request requesting remote instructions for remote operation or remote driving. The operator performs remote control (assistance) of the autonomous vehicle by inputting remote instructions while viewing the display screen.

[0015] In this disclosure, "remote control" is classified into three modes (functions)—"remote monitoring," "remote operation," and "remote driving"—which differ in the responsibilities of the controller (controller, operator), the control capabilities, and the conditions for activating safety functions, depending on the form of control over the target moving object. Here, "remote control" refers to at least one of "remote monitoring," "remote operation," and "remote driving" over at least one moving object, each configured to move autonomously and perform predetermined tasks.

[0016] In this disclosure, "remote monitoring" refers to a mode of remote control in which mobile information relating to at least one target mobile object and its surrounding environment is presented to an operator in a manner that allows for remote monitoring (confirmation), or in which the operator monitors (confirms) such information. This mobile information includes camera images and sensing information from cameras and various sensors installed on or around the mobile object, information generated by processing such information, and / or information representing the state of the mobile object itself, such as vehicle speed. Cameras and various sensors installed around the mobile object may be fixed infrastructure equipment or devices held or worn by an operator. An operator is a person who remotely monitors a mobile object such as a vehicle and remotely controls the mobile object as needed. Here, "remote monitoring" is an example of a mode of remote control in which a mobile object is remotely monitored by transmitting mobile information such as images relating to at least one target mobile object.

[0017] Furthermore, in this disclosure, "remote control" refers to a mode of remote control in which operational instructions (remote instructions) such as starting and / or stopping are given to a target mobile object from a remote location. As an example, "remote control" is a mode of remote control in which, in addition to "remote monitoring," control instructions are given to the target mobile object by transmitting a control signal (driving instruction signal) in response to an operational instruction to at least one of the target mobile objects. These operational instructions are given by an operator to remotely instruct the operation of an automatically driving mobile object, such as starting or stopping the automatic driving of the mobile object, but may also include instructions related to emergency stops or operations unrelated to driving. Here, "remote control" is an example of a transmission mode in which a target mobile object is remotely controlled by a control signal transmitted as a single control instruction (remote instruction).

[0018] Furthermore, in this disclosure, "remote operation" refers to a mode of remote control in which the operation (such as steering and / or braking) of a target mobile object is performed from a remote location. In other words, "remote operation" refers to a mode of remote control in which driving instructions (remote instructions) such as steering and / or braking are given to a target mobile object from a remote location. As an example, "remote operation" is a mode of remote control in which, in addition to "remote monitoring" or "remote operation," control signals (driving signals) corresponding to driving instructions are transmitted to at least one target mobile object, thereby remotely giving control instructions to the target mobile object. Here, "remote operation" is an example of a transmission mode in which a target mobile object is remotely controlled by control signals transmitted as a series of control instructions (driving instructions).

[0019] In this disclosure, the modes of remote control, including "remote operation" and / or "remote driving," may be combined and referred to as "remote control." Therefore, "remote control" means that an operator remotely monitors the status of a mobile object using mobile object information (video and various other information) and remotely controls the target mobile object by remote instructions (operation instructions and / or driving instructions) as needed.

[0020] The mobile entities relating to this disclosure perform various tasks, including autonomous driving, related to various services provided by remote control systems, such as delivery, security, cleaning, childcare, nursing care, sales, agricultural work, manufacturing, cargo handling, transportation, and construction. For example, the mobile entities are configured to perform predetermined tasks by autonomous control and / or remote control following remote instructions from an operator monitoring multiple mobile entities.

[0021] Furthermore, the mobile entity is not limited to a vehicle, but may be any type of mobile entity configured to move in response to remote operation or driving by an operator. The mobile entity may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Also, for example, the mobile entity may be an Automated Guided Vehicle (AGV), or various robots such as construction machinery, agricultural machinery, or drones. Moreover, these mobile entities are not limited to transporting people, but may transport objects other than people, and may provide specific services in addition to transportation. Furthermore, the mobile entity is not limited to autonomous movement, but may also be configured to move in response to direct operation and / or driving by a driver.

[0022] Thus, the remote control system relating to this disclosure is configured to be able to execute an information processing method (remote control support method) that supports an operator's remote control of at least one autonomous mobile object based on remote control images captured by a camera.

[0023] (Example of Remote Control System Configuration) Figure 1 is a diagram showing an example of the configuration of a remote control system 1 according to an embodiment. As shown in Figure 1, the remote control system 1 includes a target system 2, a control system 4, and an information aggregation system 5.

[0024] In the remote control system 1, the target system 2, the control system 4, and the information aggregation system 5 are each connected to the network N, either directly or via other telecommunications lines. This connection to the network N may be made by establishing a VPN (Virtual Private Network).

[0025] Network N may be any cellular V2X public communication network, such as a mobile communication system of each generation in accordance with the IMT2020 provisions or the 3GPP (registered trademark) specifications, such as 3G, LTE, 4G, or 5G, or a next-generation mobile communication system such as Beyond 5G (6G). In addition, each target system 2 may be connected to Network N by other communication methods, such as the IEEE compliant DSRC method or the Vehicle-to-Cellular Network (V2N) method. Furthermore, each target system 2 may be connected to Network N via short-range wireless communication of any communication method, such as Wi-Fi (registered trademark), Bluetooth (registered trademark) Low Energy (BLE), or Ultra-Wide Band (UWB). Furthermore, the short-range wireless communication may be performed using an Electronic Toll Collection System (ETC). Also, when each target system 2 is mounted on a moving object such as a vehicle, it may be connected to network N via an in-vehicle network such as CAN (Controller Area Network), Ethernet (registered trademark), or USB (Universal Serial Bus (registered trademark)).

[0026] (Regarding the target system) The target system 2 is at least one information processing system on the remotely controlled target side. The target system 2 realizes or supports the provision of various services by autonomous mobile units. The target system 2 may be implemented as a single device or through the cooperation of multiple devices.

[0027] As an example, the remote control system 1 according to this embodiment can be constructed by applying edge computing. In this case, for example, the target system 2 is used as the network edge, but other devices may be used as the edge.

[0028] In the example shown in Figure 1, the target systems 2 of the remote control system 1 are exemplified as the first target system 2-1, the second target system 2-2, the third target system 2-3, and the fourth target system 2-4, but are not limited to these. The target systems 2 included in the remote control system 1 may be one, two, three, or five or more. Furthermore, any two target systems 2 included in the remote control system 1 may have a common configuration or may have different configurations.

[0029] As an example, at least one of the target system 2 may be an on-board system mounted on the vehicle (target mobile object) that is the target of remote control. This on-board system may be configured to control the vehicle's movement, for example. This on-board system may also cooperate with other on-board systems configured to control the vehicle's movement, for example.

[0030] As an example, at least one of the target system 2 may be a fixed or portable information processing system installed outside the autonomous mobile object being remotely controlled. This information processing system may be a system that takes photographs, records sound, and / or amplifies sound of the autonomous mobile object and its surroundings using cameras and microphones on infrastructure equipment installed around the autonomous mobile object performing the task, such as around the autonomous mobile object's travel path. Alternatively, this information processing system may be a system that takes photographs, records sound, and / or amplifies sound of the autonomous mobile object and its surroundings using a wearable device worn or held by a worker around the autonomous mobile object performing the task.

[0031] Each of at least one target system 2 has, as shown in Figure 1, a target processing unit 21, a communication unit 22, an imaging unit 23, a sound amplification unit 24, and a sound collection unit 25.

[0032] The target-side processing unit 21 comprehensively controls the operation of its own system.

[0033] For example, the target-side processing unit 21 may acquire at least one image of the target autonomous mobile object using the imaging unit 23. Alternatively, the target-side processing unit 21 may transmit the acquired at least one image to the information aggregation system 5 using the communication unit 22.

[0034] For example, the target-side processing unit 21 may acquire at least one audio recording of the target autonomous mobile object using the sound collection unit 25. Alternatively, the target-side processing unit 21 may transmit the acquired at least one audio recording to the information aggregation system 5 using the communication unit 22.

[0035] For example, the target processing unit 21 may output the audio received from the control system 4 by the communication unit 22 from the target amplification unit 24.

[0036] For example, the target-side processing unit 21 may control the operation of the autonomous mobile unit in response to a control signal from the control-side system 4.

[0037] The communication unit 22 is a communication interface that communicates with the outside of the system via the network N.

[0038] The imaging unit 23 is a camera (imaging device) that captures images of the autonomous mobile object targeted by the system and acquires video. This video is an example of an image for remote control. This image for remote control may be a still image or a moving image (video). Furthermore, this image for remote control may be the same as or different from the image used for autonomous driving on the target mobile object.

[0039] The public address unit 24 is a speaker (public address device, audio output device) that outputs audio such as voice from the control system 4.

[0040] The sound-collecting unit 25 is a microphone (sound-collecting device) that collects sound from the autonomous mobile object targeted by the system and acquires its voice.

[0041] Furthermore, each of the imaging unit 23, the loudspeaker unit 24, and the sound collection unit 25 may have directivity with respect to at least one specific direction, or may have omnidirectional directivity. Further, each of the imaging unit 23, the loudspeaker unit 24, and the sound collection unit 25 may be provided for each of at least one specific direction, or may be configured such that the direction of its directivity is variable. For example, in the target-side system 2 mounted on an autonomous mobile body, each of the imaging unit 23, the loudspeaker unit 24, and the sound collection unit 25 may be provided for each of four directions: front, rear, left, and right of the autonomous mobile body.

[0042] Further, the imaging unit 23 and the sound collection unit 25 may each acquire video and audio including two or more directions, and the video and audio including the two or more directions may be subjected to software processing and divided into video and audio for each direction. Note that this division may be executed by the target-side processing unit 21, or may be executed by the control-side system 4 or the information aggregation system 5.

[0043] Further, each of the at least one target-side system 2 may include only any one of the imaging unit 23, the loudspeaker unit 24, and the sound collection unit 25, or may include a combination of two or more thereof.

[0044] (Regarding Control-side System) The control-side system 4 is at least one information processing system on the remote control side. The control-side system 4 implements an operator terminal that supports remote control of an autonomous mobile body by an operator. The control-side system 4 may be implemented as a single device, or may be implemented through cooperation of a plurality of devices.

[0045] As an example, the control-side system 4 may be, for example, a system that supports remote control on a browser of an operator's terminal. For example, the control-side system 4 may inquire of the information aggregation system 5 about access right when the browser is started or in response to an operator's operation acquired on the browser. Further, for example, the control-side system 4 may perform various screen displays related to remote control on the browser.

[0046] As an example, the control-side system 4 may be a system that supports remote control on an application (application program) installed in an operator's terminal, for example. For example, the control-side system 4 may inquire the information aggregation system 5 about access right when the application is started or in response to an operator's operation acquired on the application. Further, for example, the control-side system 4 may perform various screen displays related to remote control on the application.

[0047] Note that the control-side system 4 may be configured as a system that supports remote control through cooperation between a browser and an application. Further, the operator's terminal may be configured as a dedicated terminal, or a browser operating on the operator's terminal may be configured as a dedicated browser. Further, for example, a configuration may be adopted in which only a part of remote control support such as access right inquiry is performed on a browser or an application.

[0048] In the example of FIG. 1, one control-side system 4 is illustrated as the control-side system 4 of the remote control system 1, but the present invention is not limited thereto. Two or more control-side systems 4 may be included in the remote control system 1.

[0049] As shown in FIG. 1, each of the at least one control-side system 4 includes a remote control processing unit 41, a communication unit 42, a display unit 43, a loudspeaker unit 44, and a sound collection unit 45.

[0050] The remote control processing unit 41 integrally controls the operation of the own system.

[0051] As an example, the remote control processing unit 41 may acquire at least one video of an autonomous mobile object that is a target of remote control from the information aggregation system 5 via the communication unit 22. Further, the remote control processing unit 41 may display the acquired at least one video on the display unit 43.

[0052] As an example, the remote control processing unit 41 may acquire at least one audio message about the autonomous mobile object being remotely controlled from the information aggregation system 5 via the communication unit 22. Alternatively, the remote control processing unit 41 may output the acquired at least one audio message from the amplification unit 44.

[0053] For example, if one of the multiple images displayed by the display unit 43 is selected, the remote control processing unit 41 may transmit information indicating the selected image to the information aggregation system 5.

[0054] For example, the remote control processing unit 41 may acquire the voice of the remote control operator using the sound pickup unit 45. Alternatively, the remote control processing unit 41 may transmit the acquired operator's voice to at least one target system 2 and output it from its amplification unit 24. This voice output may be described as the control of the autonomous mobile's operation in response to a control signal from the control system 4, for example, if the amplification unit 24 is installed on the autonomous mobile.

[0055] For example, the remote control processing unit 41 may acquire remote instructions for controlling the operation of the operator's autonomous mobile device, for example, via the touch panel of the display unit 43 or a controller (not shown). The remote control processing unit 41 may also transmit control signals corresponding to the acquired remote instructions to the target autonomous mobile device or the target system 2 via the communication unit 42.

[0056] The communication unit 42 is a communication interface that communicates with the outside of the system via the network N.

[0057] The display unit 43 is a display interface, such as a liquid crystal display, that displays information on the screen based on the display information from the information aggregation system 5. This display interface may be configured as a touch panel display. Alternatively, the display interface may be various display devices connected to the control system 4.

[0058] The display information from the information aggregation system 5 may include at least one video to be displayed on the screen and display layout information for that at least one video. The display information may also include information indicating the display mode for each video. The display layout information may include information including a display pattern, or it may be information indicating which display pattern to use, for example, if each display pattern is pre-stored within the control system 4. In other words, the display information from the information aggregation system 5 may be an image of the display screen of the display unit 43, or various types of information for generating the display screen. Furthermore, the screen display by the display unit 43 may be realized by displaying at least one image showing the state in which each video is incorporated into the display pattern. Alternatively, for example, the screen display by the display unit 43 may be realized by displaying at least one image that shows the display area defined by the display pattern with a frame or the like, and each video (image) incorporated into that area.

[0059] The sound amplification unit 44 is a speaker (sound amplification device, sound output device) that outputs sound such as voice from the target system 2. The sound amplification unit 44 may be configured to output sound from a direction corresponding to the direction in which the output sound is picked up.

[0060] The sound pickup unit 45 is a microphone (sound pickup device) that picks up sound from an operator performing remote control using its own system and acquires their voice.

[0061] (Regarding the Information Aggregation System) The Information Aggregation System 5 is at least one information processing system on the remote control side. The Information Aggregation System 5 supports the operator's remote control of the autonomous mobile object. The Information Aggregation System 5 may be implemented as a single device or through the cooperation of multiple devices.

[0062] The information aggregation system 5 is built using, for example, a cloud server, but it may also be built using servers in an on-premises environment, or a combination of both.

[0063] In the example shown in Figure 1, one information aggregation system 5 is exemplified as the information aggregation system 5 of the remote control system 1, but it is not limited to this. There may be two or more information aggregation systems 5 included in the remote control system 1.

[0064] Each of the information aggregation systems 5, as shown in Figure 1, includes an information storage unit 51, an access rights management unit 52, a display processing unit 53, and a video and audio control unit 54.

[0065] The information storage unit 51 stores various types of information related to remote control.

[0066] As an example, the information storage unit 51 may store multiple video images of at least one autonomous mobile object acquired from each target system 2, and audio associated with those multiple video images.

[0067] As an example, the information storage unit 51 may store various types of information for determining the display mode of the screen display in the control system 4. This information for determining the display mode may include, for example, display layout information for the screen display. This display layout information may include division layout information for each area of ​​the screen display. The display layout information and division layout information may be stored as independent information or as integrated information. Furthermore, the information for determining the display mode may include, for example, various thresholds, conditions, and information on changes related to determining or changing the display mode.

[0068] For example, the information storage unit 51 may store access rights management information for the control system 4. This access rights management information may include, for example, information for uniquely identifying each control system 4. Alternatively, for example, this access rights management information may include information indicating at least one target system 2 to which each control system 4 has access rights.

[0069] The access rights management unit 52 manages the access rights of the control system 4.

[0070] As an example, the access rights management unit 52 determines whether the control system 4 that made the access request has access rights. This means that the control system 4 that made the access request is a control system 4 registered in the information storage unit 51. Alternatively, this means that the control system 4 that made the access request has the authority (control rights) to remotely control the autonomous mobile object or target system 2 included in the access request. Alternatively, this means that the control system 4 has access rights if no other control system 4 is remotely controlling the autonomous mobile object or target system 2 included in the access request.

[0071] The display processing unit 53 determines and / or changes the display mode of the video for each target system 2 in the control system 4.

[0072] As an example, the display processing unit 53 may determine the display mode of the video for each target system 2 based on the display layout information and the divided layout information.

[0073] As an example, the display processing unit 53 may detect an audio of interest that satisfies predetermined conditions from among multiple audio sources for at least one target system 2. For example, when the display processing unit 53 detects an audio of interest that satisfies predetermined conditions, it may identify the video corresponding to the audio of interest from among multiple videos for at least one target system 2 as the video of interest.

[0074] For example, the display processing unit 53 may identify the selected video as the video of interest when an operator selects one of the multiple videos displayed by the display unit 43 in the control system 4.

[0075] As an example, the display processing unit 53 may change the display mode of the display area of ​​the image of interest within the display area of ​​the display unit 43.

[0076] Furthermore, the audio of interest that meets the predetermined conditions may be audio that exceeds the first volume threshold. Also, changing the display mode of the display area of ​​the video of interest may be a superimposed display in which information indicating the volume level of the audio of interest is superimposed on the display area. The information indicating the volume level may be, for example, a volume level indicator.

[0077] Furthermore, the "sound of interest" that meets the specified conditions may be of a predetermined type. The type may be at least one of the following: sounds emitted by emergency vehicles, sounds related to human actions, and sounds spoken by specific workers. Also, changing the display manner of the display area of ​​the "sound of interest" may mean changing the display manner of the information indicating the volume level in that display area. Changing the display manner of this information indicating the volume level may mean changing the display color, background color, line type, icon, or display position.

[0078] Furthermore, any changes to the display mode of the display area for the video of interest only need to be made visible to the operator as it is the video of interest. This is not limited to the superimposition of information indicating volume levels, such as indicators, but may also include highlighting the outer perimeter of the display area (e.g., the frame).

[0079] As an example, the display processing unit 53 may detect whether any of the multiple audio sources for at least one target system 2 exceeds a second volume threshold for a predetermined period of time. For example, if the display processing unit 53 detects an audio source that exceeds the second volume threshold for a predetermined period of time, it may reduce the volume output from the sound amplification unit 44 to a predetermined value. This predetermined value may be zero. Alternatively, instead of a predetermined value, the volume may be reduced by a predetermined amount or a predetermined percentage from the current volume. For example, the display processing unit 53 may change the display manner of the information indicating the volume level in accordance with the decrease in the volume level output from the sound amplification unit 44.

[0080] Furthermore, the detection of whether an audio signal exceeds the second threshold in volume for a predetermined period of time may be limited to specific types of audio signals, such as construction noise or aircraft blast noise.

[0081] The second volume threshold is, for example, greater than the first volume threshold, but is not limited to this. For example, different second thresholds may be set for each type of sound, or the first and second thresholds may be common for specific types of sound.

[0082] For example, the display processing unit 53 may detect whether any of the multiple voices from at least one target system 2, or the voice of interest, is a human voice. For example, if the display processing unit 53 detects a human voice, it may increase the volume output from the sound amplification unit 44 to a predetermined value. This predetermined value may be a value set for each operator. Alternatively, instead of a predetermined value, the volume may be increased by a predetermined amount or a predetermined percentage from the current volume. For example, the display processing unit 53 may change the display manner of the information indicating the volume level as the volume level output from the sound amplification unit 44 increases.

[0083] Furthermore, the change in the display manner of the information indicating the volume level associated with the increase and / or decrease in volume output from the sound amplification unit 44 may include changes to whether or not the information is displayed, the display color and background color, line type, icon, and display position. Also, the change in the display manner of the information indicating the volume level may include changes to whether or not a specific icon is displayed.

[0084] Furthermore, information indicating the volume level of the sound of interest may be superimposed on the display area of ​​the image of interest at a position corresponding to the sound of interest. For example, the image of interest may be a wide-angle image of the surrounding environment obtained with a fisheye camera, and the information indicating the volume level may be at a position on the image of interest corresponding to the direction of the sound of interest.

[0085] The image of interest may be, for example, a wide-angle image of the surrounding environment obtained with a fisheye camera, and the sound amplification unit 44 in the direction corresponding to the selected position within the display area may be identified as the control target. This selection of the position on the image may be achieved by a selection operation on information indicating the volume level superimposed on the position on the image corresponding to the direction of the sound.

[0086] For example, the display processing unit 53 may determine whether the autonomous mobile object to be controlled is moving based on the mobile object information, and if it is moving, it may identify the image in the direction of travel from among the multiple images displayed by the display unit 43 as the image of interest.

[0087] For example, the display processing unit 53 may identify multiple images containing the same object from among the multiple images displayed by the display unit 43 in the control system 4 as images of interest. Here, the same object may be at least one of the same person, moving object, and obstacle.

[0088] As an example, the display processing unit 53 may detect images that satisfy predetermined conditions from image recognition processing of multiple images for at least one target system 2. For example, if the display processing unit 53 detects an image that satisfies predetermined conditions, it may identify that image as an image of interest. Here, an image that satisfies predetermined conditions may be an image in which a specific moving object such as a person or an emergency vehicle, or an obstacle, is detected.

[0089] As an example, the display processing unit 53 may select an autonomous mobile body corresponding to the image of interest, or its direction, as the control target. Specifically, the display processing unit 53 may select an autonomous mobile body that acquired the image of interest, or an autonomous mobile body that captures the image of interest, or the target system 2 in the direction of acquisition, as the control target.

[0090] The display processing unit 53 may identify an image of interest depending on the state of the autonomous mobile device equipped with the target system 2 or the worker wearing or gripping it. Furthermore, the output direction of the operator's voice in the target system 2 selected as the control target, or the volume level in each direction, may be changed depending on the state of the autonomous mobile device equipped with the target system 2 or the worker wearing or gripping it. For example, when the autonomous mobile device or worker is moving, the direction of the voice output may be set to the direction of travel, the volume level in the direction of travel may be made higher than in other directions, or the volume levels in other directions may be made lower. For example, when the autonomous mobile device or worker is moving, the volume level of the voice output in each direction may be made lower, set to zero, or made higher. These changes can be appropriately varied depending on, for example, the work (task) content, the control target, the speed of movement, etc.

[0091] As an example, the display processing unit 53 may change the display mode of the display area of ​​the video of interest that has been identified as the control target within the display area of ​​the display unit 43. This change in display mode may relate to the display indicating whether or not there is audio output from the sound amplifier 24 of the target system 2. For example, if the control system 4 is outputting operator voice from the sound amplifier 24, the display processing unit 53 may change the display mode so that it can be visually distinguished from other videos, including other videos of interest. This change in display mode may also refer to a change in the display mode of information indicating the volume level in the display area, similar to a change in the display mode of the video of interest. Furthermore, this change in the display mode of information indicating the volume level may also refer to a change in the display color, background color, line type, icon, or display position.

[0092] For example, the display processing unit 53 may deselect an image as an image of interest if the image it has identified as an image of interest no longer meets the predetermined conditions or if the operator issues a deselection command. In this case, along with the deselection of the image of interest, the selection as a control target may also be deselected.

[0093] The display processing unit 53 may, depending on the state of the autonomous mobile device equipped with the target system 2 or the operator wearing or gripping it, deselect the image of interest, or restore the output direction of the operator voice or the volume levels of each direction to their original settings. For example, if the forward image was selected as the image of interest while driving, the selection of the forward image may be deselected when the device stops from driving.

[0094] The display processing unit 53 may, at once, deselect multiple images containing the same object as images of interest. Here, the same object may be at least one of the same person, moving object, and obstacle.

[0095] The video and audio control unit 54 controls the transmission of video and / or audio from each target system 2 to the control system 4.

[0096] As an example, the video and audio control unit 54 transmits multiple video images of at least one autonomous mobile object, acquired from each target system 2, along with the audio associated with those multiple video images, to the control system 4, where they are output by the display unit 43 and sound amplification unit 44.

[0097] As an example, the video and / or audio control unit 54 may identify which direction the video and / or audio acquired from each target system 2 was obtained from. In other words, the video and / or audio control unit 54 may associate each video and / or audio acquired from each target system 2 with the direction from which the video and / or audio was obtained from the autonomous mobile body.

[0098] Furthermore, the video and audio control unit 54 may control the transmission of audio picked up by the control system 4 and control signals corresponding to acquired operation instructions to the autonomous mobile object being controlled or to each target system 2 targeting said autonomous mobile object.

[0099] The control system 4 and the information aggregation system 5 may be configured as a single unit.

[0100] Furthermore, the multiple images of at least one autonomous mobile body acquired from each target system 2, or displayed in the control system 4, may, for example, be at least one image of each different autonomous mobile body. Also, for example, the multiple images may be multiple images of a single autonomous mobile body. Also, for example, the multiple images may be images with different shooting directions. Also, for example, the multiple images may be images with different field of view. Also, the multiple images may be images obtained by dividing an omnidirectional image through image processing, with different directions from the autonomous mobile body. This direction from the autonomous mobile body is, for example, the shooting direction by the imaging unit 23 mounted on the autonomous mobile body, but it may also be, for example, the direction from the autonomous mobile body to an area within the field of view of the imaging unit 23 outside the autonomous mobile body. Also, for example, the multiple images may include images that are not associated with audio. Also, for example, the multiple images may include audio that is not associated with any of the images.

[0101] Figure 2 shows an example of the hardware configuration of an information processing device 8 that realizes each function of each device included in the remote control system 1 according to the embodiment. The information processing device 8 is a computer that comprehensively controls the operation of the entire device in each device included in the remote control system 1.

[0102] Furthermore, the information processing device 8 that realizes each function of the target system 2 mounted on the vehicle (autonomous mobile device) may be a computer such as an ECU (Electronic Control Unit) or a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit).

[0103] As shown in Figure 2, the information processing device 8 includes a processor 81, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 83, and a device I / F (interface) unit 84.

[0104] The processor 81 is, for example, a CPU (Central Processing Unit), but in addition to or instead of a CPU, at least one of various processors such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array) can be used as appropriate. Here, the processor 81 according to this embodiment is an example of at least one processor in the information processing device 8.

[0105] For example, the processor 81 of the target system 2 realizes the functions of the target processing unit 21, communication unit 22, imaging unit 23, sound amplification unit 24, and sound collection unit 25 as illustrated in Figure 1 by executing a program stored in ROM 82, for example. For example, the processor 81 of the control system 4 realizes the functions of the remote control processing unit 41, communication unit 42, display unit 43, sound amplification unit 44, and sound collection unit 45 as illustrated in Figure 1 by executing a program stored in ROM 82, for example. For example, the processor 81 of the information aggregation system 5 realizes the functions of the information storage unit 51, access rights management unit 52, display processing unit 53, and video and audio control unit 54 as illustrated in Figure 1 by executing a program stored in ROM 82, for example.

[0106] In the example shown in Figure 1, only the functions necessary for explaining the main parts of the embodiment are illustrated, but the functions of each device included in the remote control system 1 are not limited to these. In the embodiment, the processor 81 executes a program stored in the ROM 82 to realize each function of each device, including the functions of each part described above. However, it is not limited to this, and some or all of these functions may be realized by dedicated hardware circuits. In addition, each device of the remote control system 1 may integrate two or more functions to realize as a single function. Similarly, each device of the remote control system 1 may divide one function to realize two or more functions. In addition, in the remote control system 1, the functions of two or more devices may be integrated to realize at least one function of any of the devices. Similarly, in the remote control system 1, the functions of one device may be divided to realize two or more functions of two or more devices.

[0107] ROM 82 is a non-volatile memory and is an auxiliary storage device that stores various types of information, including programs executed by the processor 81. The information processing device 8 is not limited to ROM 82; various recording media and devices such as HDD (Hard Disk Drive), SSD (Solid State Drive), and Flash memory can be used as appropriate. RAM 83 is a volatile memory that has a working area for the processor 81 and is the main memory. Here, ROM 82 and RAM 83 in this embodiment are examples of at least one memory in the information processing device 8. The device I / F unit 84 is an interface for implementing communication devices, display devices, output devices, input devices, etc., in each device included in the remote control system 1, and for connecting to external devices such as communication devices, display devices, output devices, and input devices.

[0108] Next, examples of the operation of each embodiment of the remote control system 1 configured as described above will be explained with reference to the drawings. Note that the operation procedures and processing flows described below are examples only, and the order of steps can be changed, some steps can be deleted, or other steps can be added as desired.

[0109] Figure 3 is a flowchart showing an example of the information processing flow performed by the remote control system 1 according to this embodiment. Figure 4 is a diagram showing an example of the screen display in the remote control according to this embodiment.

[0110] First, the control system 4 accesses the information aggregation system 5, for example, when an application is launched or in response to an operator's operation (S101). The information aggregation system 5 determines whether the control system 4 that accessed it has access rights (S102). If the control system 4 does not have access rights (S102: No), remote control is not started, and the flow in Figure 3 returns to S101.

[0111] On the other hand, if the control system 4 has access rights (S102: Yes), the information aggregation system 5 starts transmitting video and audio of the controlled autonomous mobile object to the control system 4, and the control system 4 displays the video (S103). Figure 4 illustrates a case where the video of four controlled autonomous mobile objects, "A" to "D", is assigned to and displayed in the four display areas 603 of the display area 601 of the screen display. In addition, each display area 603 is assigned multiple videos with different directions from the autonomous mobile object, such as the shooting direction. For example, in the example in Figure 4, in a certain display area 603, "A-1", which is the video of the first direction of the autonomous mobile object "A", "A-2", which is the video of the second direction of the autonomous mobile object "A", "A-3", which is the video of the third direction of the autonomous mobile object "A", and "A-4", which is the video of the fourth direction of the autonomous mobile object "A", are assigned and displayed.

[0112] Furthermore, the information aggregation system 5 identifies the audio of interest and visualizes the audio level of the audio of interest on the video of interest corresponding to the identified audio of interest (S104). Figure 4 illustrates the case when "D-1," which is the video of the first direction for the autonomous mobile body "D," is identified as the video of interest. The audio level display 611 of the audio of interest is superimposed on the video of "D-1," which is the video of interest. Subsequently, if no operator instruction to terminate access is given (S105: No), the processes from S103 to S104 are repeated. Then, if an operator instruction to terminate access is given (S105: Yes), access from the control system 4 to the information aggregation system 5 is disconnected (S106), and the flow in Figure 3 ends.

[0113] Figure 5 is a flowchart showing an example of the flow of remote control processing performed by the remote control system 1 according to the embodiment. The flow in Figure 5 is performed, for example, from the start to the end of display in Figure 3 (S102: Yes to S105: No), in parallel with or instead of visualization (S103).

[0114] The information aggregation system 5 determines whether a specific video has been selected (S201). If it has not been selected (S201: No), the flow in Figure 5 returns to the process in S201. On the other hand, if any video is selected (S201: Yes), the information aggregation system 5 identifies the control target associated with the selected video as the control target (S202) and starts outputting operator voice from the identified control target (S203). At this time, an indicator or the like may be displayed for the video corresponding to the direction from which the voice output is coming, as in Figure 4. If it is still selected (S204: No), the process in S203 is repeated. Then, if it is deselected (S204: Yes), the information aggregation system 5 terminates the output of operator voice from that control target (S205). After that, the flow in Figure 5 ends.

[0115] Here, we will describe the screen display according to the embodiment described above. In the example in Figure 4, we illustrate a case where four images are displayed for each of the four target systems 2, but we are not limited to this. The screen display in the remote control according to the embodiment can be appropriately changed depending on the number of controlled objects, the number of directions for acquiring images and sounds for each object, or the operator's selection results.

[0116] Figure 6 shows an example of display layout information 701 for screen display in remote control according to the embodiment. The display layout information 701 is, for example, predetermined and stored in the information storage unit 51 of the information aggregation system 5. Each display pattern of the display layout information 701 defines an arrangement pattern of at least one display area 603 to which each image is assigned in the display area 601 of the display unit 43. For example, the display processing unit 53 may select a display pattern according to the number of controlled objects (e.g., autonomous mobile objects) and display each image in each display area 603 defined by the display pattern. The number of display areas 603 may be greater than the number of controlled objects. If the number of display areas 603 is greater than the number of controlled objects, the display areas 603 that do not display images may be indicated as blank, for example, to indicate that they are not connected. This indication may be given by text or icons. Map information may also be displayed in all or part of at least one display area 603. When assigning images to the display areas 603, letterboxing or side-cutting may be used.

[0117] Figure 7 shows an example of screen display division layout information 703 in remote control according to the embodiment. The division layout information 703 is, for example, predetermined and stored in the information storage unit 51 of the information aggregation system 5. Each division pattern of the division layout information 703 defines the pattern of video allocation in each display area 603. For example, the display processing unit 53 may select a division pattern for the corresponding display area 603 according to the number of videos from an arbitrary controlled object (e.g., an autonomous mobile object), and display each video in each display area within the display area 603 defined by the division pattern. The number of divisions may be greater than the number of videos (shooting direction) from the controlled object, and display areas that do not display videos may be left blank or display a message to indicate that they are not connected. Also, as exemplified by division pattern 1, a display area for error messages etc. may be provided along with the video. Furthermore, different division patterns may be used for each display area 603.

[0118] As described above, the remote control according to this embodiment is configured to change the display mode of interest images associated with interest audio that meets predetermined conditions, among multiple images of at least one autonomous mobile object on the controlled side. With this configuration, the operator can intuitively grasp the image to be recognized from the multiple images. In other words, the remote control according to this embodiment can reduce the workload of the remote control operator in relation to remote control using the video and audio of the controlled object. Furthermore, the operator can perform remote control based on the interest image intuitively grasped, such as by outputting operator voice from the direction corresponding to the identified interest image.

[0119] In the embodiments described above, the determination of "whether or not it is A" may be achieved by determining only whether it is "A", by determining only whether it is "not A", or by determining both.

[0120] In the embodiments described above, "any of A" means "at least one of A".

[0121] Furthermore, the programs executed by each device of the remote control system 1 according to the above embodiment may be provided as files in an installable or executable format, recorded on a computer-readable, non-transient recording medium such as a CD-ROM, floppy disk, CD-R, or DVD.

[0122] Furthermore, the programs executed by each device of the remote control system 1 according to the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. Alternatively, the programs executed by each device of the remote control system 1 according to the above embodiment may be provided or distributed via a network such as the Internet.

[0123] Furthermore, the program executed by each device of the remote control system 1 according to the above embodiment may be pre-installed and provided in ROM or the like.

[0124] According to at least one embodiment described above, the workload on the remote control side can be reduced in relation to remote control using video and audio from the controlled side.

[0125] While embodiments of this disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0126] (Note) The above description of embodiments discloses the following technologies: (1) An information processing method performed by an information processing system that supports remote control of at least one autonomous mobile body by an operator, comprising: outputting a plurality of images of at least one autonomous mobile body and audio associated with the plurality of images from the operator's terminal; when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of images, the image of interest among the plurality of images corresponding to the audio of interest is identified as the image of interest, and the display mode of the display area of ​​the image of interest in the display area of ​​the operator's terminal is changed. (2) The information processing method according to (1) above, wherein the plurality of images include images of different autonomous mobile bodies and / or a plurality of images of a single autonomous mobile body. (3) The information processing method according to (1) above or (2) above, wherein the plurality of images include images with different shooting directions and / or images based on omnidirectional images with different directions from the autonomous mobile body. (4) The information processing method according to any one of (1) to (3) above, wherein the sound of interest that satisfies the predetermined conditions is a sound that exceeds a first threshold of volume, and the change in the display manner of the display area of ​​the image of interest includes the superimposition of information indicating the volume level of the sound of interest onto the display area. (5) The information processing method according to (4) above, wherein the sound of interest that satisfies the predetermined conditions is a sound of a predetermined type, and the display manner of the information indicating the volume level of the sound of interest differs depending on the type. (6) The information processing method according to (5) above, wherein the type includes at least one of the sounds emitted by emergency vehicles, sounds related to human actions, and sounds spoken by a specific worker. (7) The information processing method according to (5) or (6) above, wherein if the sound associated with the plurality of images exceeds a second threshold of volume for a predetermined period of time, the volume output from the operator's terminal is reduced to a predetermined value, and the display manner of the information indicating the volume level of the sound of interest is changed.(8) The information processing method according to any one of (5) to (7) above, wherein if the sound of interest is a sound uttered by a person, the volume output from the operator's terminal is increased to a predetermined value, and the display manner of the information indicating the volume level of the sound of interest is changed. (9) The information processing method according to any one of (5) to (8) above, wherein the information indicating the volume level of the sound of interest is superimposed and displayed at a position in the display area of ​​the image of interest that corresponds to the sound of interest. (10) The information processing method according to any one of (1) to (9) above, wherein if the image of interest that satisfies predetermined conditions is detected from a plurality of images of the autonomous mobile body, the display manner of the display area of ​​the image of interest in the display area of ​​the operator's terminal is changed. (11) The information processing method according to any one of (1) to (10) above, wherein the autonomous mobile body corresponding to the image of interest is identified as a control target by the operator's terminal. (12) The information processing method according to any one of the above paragraphs (1) to (11), wherein when one of the multiple videos output from the operator's terminal is selected, the autonomous mobile body corresponding to the selected video is identified as a control target by the operator's terminal. (13) An information processing device that supports remote control by an operator for at least one autonomous mobile body, which outputs a plurality of videos of at least one autonomous mobile body and audio associated with the plurality of videos, and when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of videos, the display mode of the display area of ​​the video of interest corresponding to the audio of interest among the display areas of the plurality of videos. (14) A program that causes a computer in an information processing system that supports remote control of at least one autonomous mobile object by an operator to output a plurality of images of at least one autonomous mobile object and audio associated with the plurality of images from the operator's terminal, and when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of images, to identify the image of interest among the plurality of images that corresponds to the audio of interest, and to change the display mode of the display area of ​​the video of interest in the display area of ​​the operator's terminal.(15) An information processing apparatus comprising at least one processor and at least one memory, wherein the at least one processor executes the information processing method described in any one of (1) to (12) above by executing a program stored in the at least one memory. (16) A program that causes a computer to execute the information processing method described in any one of (1) to (12) above, or a computer-readable non-transient storage medium in which the program is stored.

[0127] 1 Remote Control System 2 Target System 2-1 First Target System 2-2 Second Target System 2-3 Third Target System 2-4 Fourth Target System 21 Target Processing Unit 22 Communication Unit 23 Imaging Unit 24 Amplification Unit 25 Sound Collection Unit 4 Control System 41 Remote Control Processing Unit 42 Communication Unit 43 Display Unit 44 Amplification Unit 45 Sound Collection Unit 5 Information Aggregation System 51 Information Storage Unit 52 Access Rights Management Unit 53 Display Processing Unit 54 Video and Audio Control Unit 601 Display Area 603 Display Area 611 Audio Level Display 701 Display Layout Information 703 Split Layout Information 8 Information Processing Device 81 Processor 82 ROM 83 RAM 84 Device I / F Unit N Network

Claims

1. An information processing method performed by an information processing system that supports remote control by an operator of at least one autonomous mobile object, comprising: outputting a plurality of images of at least one autonomous mobile object and audio associated with the plurality of images from the operator's terminal; and when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of images, identifying the image of interest among the plurality of images that corresponds to the audio of interest, and changing the display mode of the display area of ​​the video of interest in the display area of ​​the operator's terminal.

2. The information processing method according to claim 1, wherein the plurality of images include images of different autonomous mobile bodies and / or a plurality of images of a single autonomous mobile body.

3. The information processing method according to claim 1, wherein the plurality of images include images with different shooting directions and / or images based on omnidirectional images with different directions from the autonomous mobile body.

4. The information processing method according to claim 1, wherein the audio of interest that satisfies the predetermined conditions is an audio that exceeds a first threshold of volume, and the change in the display mode of the display area of ​​the video of interest includes the superimposition of information indicating the volume level of the audio of interest onto the display area.

5. The information processing method according to claim 4, wherein the audio of interest that satisfies the predetermined conditions is of a predetermined type, and the display manner of information indicating the volume level of the audio of interest differs depending on the type.

6. The information processing method according to claim 5, wherein the type includes at least one of the following: sounds emitted by emergency vehicles, sounds related to human actions, and sounds spoken by specific workers.

7. The information processing method according to claim 4, wherein if the audio associated with the plurality of videos exceeds a second volume threshold for a predetermined period of time, the volume output from the operator's terminal is reduced to a predetermined value, and the display mode of the information indicating the volume level of the audio of interest is changed.

8. The information processing method according to claim 4, wherein, if the voice of interest is a sound uttered by a person, the volume output from the operator's terminal is increased to a predetermined value, and the display mode of the information indicating the volume level of the voice of interest is changed.

9. The information processing method according to claim 4, wherein information indicating the volume level of the audio of interest is superimposed and displayed in a position corresponding to the audio of interest within the display area of ​​the video of interest.

10. The information processing method according to claim 1, wherein when a video of interest that satisfies predetermined conditions is detected from a plurality of videos of the autonomous mobile body, the display mode of the display area of ​​the video of interest in the display area of ​​the operator's terminal is changed.

11. The information processing method according to any one of claims 1 to 10, wherein the autonomous mobile body corresponding to the image of interest is identified as a control target by the operator's terminal.

12. The information processing method according to any one of claims 1 to 10, wherein when one of the multiple videos output from the operator's terminal is selected, the autonomous mobile body corresponding to the selected video is identified as a control target by the operator's terminal.

13. An information processing device that supports remote control by an operator of at least one autonomous mobile object, which outputs a plurality of images of at least one autonomous mobile object and audio associated with the plurality of images, and when an audio of interest that satisfies predetermined conditions is detected from the audio associated with the plurality of images, changes the display mode of the display area of ​​the image of interest that corresponds to the audio of interest among the display areas of the plurality of images.

14. A program that causes a computer in an information processing system supporting remote control by an operator of at least one autonomous mobile object to output multiple images of at least one autonomous mobile object and audio associated with those multiple images from the operator's terminal, and when it detects an audio of interest that satisfies predetermined conditions from the audio associated with the multiple images, it identifies the image of interest among the multiple images that corresponds to the audio of interest, and changes the display mode of the display area of ​​the video of interest in the display area of ​​the operator's terminal.

Images