Information processing method and information processing device

Abstract

The present disclosure relates to an information processing method and an information processing device with which remote control of a mobile object can be effectively assisted. In the present invention, operation input is received from a mobile object, environmental body information pertaining to the environment of the mobile object and the body of the mobile object are collected, the operation input and environmental body information are managed as log information, current operation input and environmental body information are compared with the log information, and notification information is presented on the basis of the comparison result. The present invention can be applied to drones.

Description

Information Processing Method and Information Processing Apparatus

[0001] The present disclosure relates to an information processing method and an information processing apparatus, and more particularly to an information processing method and an information processing apparatus that can effectively assist the remote operation of a moving object.

[0002] There is an increasing need for technology that assists remote operation of a moving object such as a drone and realizes comfortable remote operation.

[0003] Since some moving objects have achieved autonomous driving, it is conceivable to use the technology related to the autonomous driving of this moving object to assist remote operation.

[0004] More specifically, as a technology related to the autonomous driving of a moving object, a technology has been proposed to control the current autonomous driving of the moving object based on the history of sensor detection values of various sensors related to the past movement of the moving object (see Patent Document 1).

[0005] By applying the technology related to the autonomous driving of this moving object to remote operation, it is conceivable to realize the control of driving in a state where there is no intervention of user operation related to remote operation and to assist remote operation.

[0006] Japanese Patent Application Laid-Open No. 2017-228111

[0007] However, since the technology of Patent Document 1 is the control of autonomous driving using the history of sensor detection values of various sensors related to the past movement of the moving object, cooperation in control is not considered in a state where there is intervention of user operation related to remote operation.

[0008] Therefore, in a state where there is intervention of user operation related to remote operation, there is a risk that remote operation cannot be effectively assisted.

[0009] The present disclosure has been made in view of such a situation, and particularly enables effective assistance for remote operation of a moving object.

[0010] One aspect of the information processing method of this disclosure is an information processing method that includes receiving operation input on a mobile body, collecting environmental and aircraft information relating to the environment of the mobile body and the aircraft of the mobile body, managing the operation input and environmental and aircraft information as log information, comparing the current operation input and environmental and aircraft information with the log information, and presenting notification information based on the comparison results obtained from the comparison.

[0011] An information processing device in one aspect of this disclosure includes: an operation input receiving unit that receives operation inputs from a mobile device; an environmental body information collection processing unit that collects environmental body information relating to the environment of the mobile device and the body of the mobile device; a log management processing unit that manages the operation inputs and environmental body information as log information; a comparison unit that compares the current operation inputs and environmental body information with the log information; and a notification information presentation unit that presents notification information based on the comparison results of the comparison unit.

[0012] In one aspect of this disclosure, an operation input is received from a mobile device, environmental and aircraft information relating to the environment of the mobile device and the aircraft of the mobile device is collected, the operation input and environmental and aircraft information are managed as log information, the current operation input and environmental and aircraft information are compared with the log information, and notification information is presented based on the comparison result.

[0013] This is a diagram illustrating an overview of the present disclosure. This is a diagram illustrating an overview of the present disclosure. This is a diagram illustrating an example configuration of the mobile control system of the present disclosure. This is a block diagram illustrating the hardware configuration of the mobile body in Figure 3. This is a functional block diagram illustrating the functions realized by the mobile body in Figure 4. This is a diagram illustrating the functions realized by the abnormality suggestion processing unit. This is a flowchart illustrating the mobile control processing (part 1) in the first embodiment. This is a diagram illustrating an example of the controller display due to the processing in Figure 7. This is a flowchart illustrating the mobile control processing (part 2) in the second embodiment. This is a diagram illustrating an example of the controller display due to the processing in Figure 9. This is a flowchart illustrating the mobile control processing (part 3) in the third embodiment. This is a diagram illustrating an example of the controller display due to the processing in Figure 11. This is a flowchart illustrating the mobile control processing (part 4) in the fourth embodiment. This is a diagram illustrating an example of the controller display due to the processing in Figure 13. This is a diagram illustrating an example configuration of a computer.

[0014] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0015] The following describes the configurations for implementing this technology. The explanation will proceed in the following order.

[0016] 1. Outline of this Disclosure 2. First Embodiment 3. Second Embodiment 4. Third Embodiment 5. Fourth Embodiment 6. Description of a Computer Applying this Technology

[0017] <<1. Overview of this Disclosure>> This disclosure is particularly intended to enable effective assistance in the remote control of mobile devices. Therefore, we will first explain the overview of this disclosure.

[0018] In this embodiment, a drone (flying object) that flies (soars) by remote control will be used as an example of a mobile object for the explanation. However, the description is not limited to drones, as long as it is a mobile object that can be remotely controlled, it may also be a vehicle that travels on land, a ship that travels on the sea, a submarine that travels underwater, a spacecraft that travels in outer space, etc.

[0019] The control inputs required for remotely operating mobile devices such as drones are diverse, and the settings vary depending on the operating environment.

[0020] For example, if the same operation input is repeated each time a mobile object is remotely controlled, the user's operational burden increases due to the repetition of the same operations. Furthermore, the existence of a wide variety of operation inputs can lead to errors in operation input and subsequent errors in operation settings. In such cases, proper flight of the drone or imaging using the camera mounted on the drone may not be achieved, potentially requiring the flight or imaging to be restarted. Moreover, in the worst-case scenario of a setting error, the drone may not be able to fly properly, potentially colliding with obstacles or crashing.

[0021] Thus, the input for remote control of mobile devices must balance reducing the user's input burden with avoiding operational errors. Achieving these two goals will enable effective assistance for remote control of mobile devices.

[0022] Therefore, in this disclosure, in addition to the information currently acquired by the drone, past log information is used and compared with current user operation information and sensor detection values ​​to detect similar patterns and anomaly patterns. Then, based on the detected patterns, future user operations are estimated and suggested, or the drone's operation settings are automatically changed.

[0023] As a more concrete example, consider the case where a user P remotely controls a drone D by operating a controller C, as shown in Figure 1.

[0024] User P operates controller C to move drone D to the destination and perform imaging at the destination. At this time, the four operations shown in Figure 1 (operations 1 through 4) are considered a routine.

[0025] Specifically, as shown in the upper left of Figure 1, the first operation is to perform the startup operation; next, as shown in the upper right of Figure 1, the second operation is to perform the takeoff operation; and then, as shown in the lower left of Figure 1, the third operation is to perform a preliminary flight, which involves swaying up and down and left and right to check for any abnormalities; and finally, as shown in the lower right of Figure 1, the fourth operation is to move to the destination.

[0026] In this disclosure, first, the four operations, from the first to the fourth, are recorded as log information.

[0027] Then, when user P operates controller C to move drone D to the imaging location and perform imaging at the imaging location, similar to the case in Figure 1, the user's input is detected, recorded as log information, and compared with past log information.

[0028] Here, when user P performs an operation to move drone D to a destination by operating controller C and to take images at the destination, for example, as shown in Figure 2, user P forgets the third operation, the preliminary flight is not performed, and there is no operation to check for abnormalities, and the fourth operation, moving to the imaging point, is performed.

[0029] In this disclosure, by comparing past log information consisting of the first to fourth operations shown in Figure 1 with the current operation input, as shown in Figure 2, if the third operation, a preliminary flight, is not performed and the fourth operation, movement to the imaging point, is performed, the absence of the third operation and the fact that an operation pattern different from past patterns are detected as an anomaly. Then, as shown in the lower right of Figure 2, a warning is issued when the fourth operation, movement to the destination, begins, indicating that the third operation is missing.

[0030] Through this process, user P can recognize that they have forgotten the third operation out of the first four operations that they were performing as a routine. For example, by repeating the third operation, they can re-check whether there is any abnormality in drone D before proceeding with the fourth operation, which is moving to the destination.

[0031] As a result, it becomes possible to effectively assist users in remotely controlling mobile devices.

[0032] <<2. First Embodiment>> Next, with reference to Figure 3, an example of the configuration of the first embodiment of the mobile control system of the present disclosure will be described.

[0033] The mobile control system 31 shown in Figure 3 consists of a mobile unit 51, a controller 52, and a server computer 53.

[0034] The mobile body 51 is, for example, a drone, and is controlled based on operation signals corresponding to operation input from the user 50 to the controller 52. In response to the operation signals, it moves to a predetermined destination by driving a drive unit 51a such as a propeller. The mobile body 51 is equipped with an imaging unit such as a camera 51b, and when it moves to the predetermined destination, it can capture images of the area around the predetermined destination.

[0035] The mobile unit 51 stores operation signals supplied by the controller 52 and sensor detection values ​​from various built-in sensors as log information. The mobile unit 51 then compares the current operation signals and sensor detection values ​​with past operation signals and sensor detection values ​​stored as log information, generates information according to the comparison result, and supplies it to the controller 52 for presentation to the user 50. Furthermore, based on the information according to the comparison result presented to the controller 52, when the mobile unit 51 receives an operation signal corresponding to the operation input entered by the user operating the controller 52, it reflects the received operation signal in its own movement.

[0036] <Example of Mobile Unit Configuration> Next, with reference to Figure 4, an example of the hardware configuration of the mobile unit 51 in Figure 3 will be described.

[0037] The mobile unit 51 consists of a processing circuit 71, an input unit 72, an output unit 73, a storage unit 74, a communication unit 75, a drive 76, a removable storage medium 77, a camera 81 (51b), a sensor 82, and a drive unit 83 (51a), which are connected to each other via a bus 78, and can send and receive data and programs.

[0038] The processing circuit 71 consists of a processor and memory and controls the overall operation of the mobile body 51. The processing circuit 71 also includes an environmental body information collection unit 91, a log information management unit 92, an abnormality suggestion processing unit 93, a drive control unit 94, and a camera control unit 95.

[0039] Furthermore, the environmental device information collection unit 91, log information management unit 92, abnormality suggestion processing unit 93, drive control unit 94, and camera control unit 95 will be described later in conjunction with the explanation of the functions realized by the mobile body 51 shown in Figure 4, which will be explained with reference to Figure 5.

[0040] The input unit 72 consists of input devices such as a keyboard, mouse, and touch panel for inputting various types of information, and supplies the input information and corresponding signals to the processing circuit 71. The input unit 72 may also be implemented by a microphone that accepts voice input.

[0041] The output unit 73 is controlled by the processing circuit 71 and includes, for example, a display unit such as a display for displaying various processing results and input contents, and an audio output unit such as a speaker for generating processing results and various warning sounds.

[0042] The storage unit 74 consists of an HDD (Hard Disk Drive), SSD (Solid State Drive), or semiconductor memory, etc., is controlled by the processing circuit 71, and writes or reads various data and programs. Further, the storage unit 74 stores log information 74a in which operation signals supplied from the controller 52, sensor detection values of the sensor 82, etc. are recorded, and is written or read as necessary under the control of the processing circuit 71.

[0043] The communication unit 75 is controlled by the processing circuit 71, realizes communication represented by, for example, wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), etc., communicates with the controller 52, receives operation signals, and transmits information such as notification information and proposals for changing operation settings as necessary. Further, the communication unit 75 transmits and receives various data and programs to and from the server computer 53 and other information processing devices via a network as necessary.

[0044] Various data and programs may be stored in the server computer 53. For example, map data related to the movement of the moving body 51 may be stored. Here, the map data related to the movement of the moving body 51 may include, in addition to position information such as a set destination and address, information for identifying whether it is an urban area with a relatively high population density on the map or a suburban area with a relatively low population density outside the urban area.

[0045] The drive 76 reads and writes data to a removable storage medium 77 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including a MD (Mini Disc)), or a semiconductor memory.

[0046] The camera 81 (51b) consists of, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, is controlled by the camera control unit 95 of the processing circuit 71, captures an image within a predetermined visual field, and stores the imaging result, for example, in the storage unit 74.

[0047] The sensor 82 detects various information such as the temperature, humidity, air pressure, brightness, and moving speed of the surrounding environment during the movement of the moving body 51, as well as various information such as the temperature and rotation speed of a motor that drives a propeller inside the moving body 51, etc., as sensor detection values, and supplies them to the processing circuit 71. Further, the sensor 82 may include a GPS (Global Positioning System) or an IMU (Inertial Measurement Unit) that detects the position and attitude of the moving body 51. Furthermore, the sensor 82 may be a sensor that detects information about the body of the moving body 51, and may include, for example, an obstacle sensor, an image sensor, a position sensor, a wind speed sensor, etc. Note that the information detected by the sensor 82 is also referred to as environmental body information.

[0048] The drive unit 83 (51a) is, for example, a drive mechanism for moving the body of the moving body 51 such as a motor that rotates a propeller and a tilt mechanism that tilts the rotation axis of the propeller, and is driven based on a drive control signal supplied from the drive control unit 94 of the processing circuit 71.

[0049] <Function realized by the moving body in FIG. 4> Next, referring to FIG. 5, the function realized by the moving body 51 in FIG. 4 will be described.

[0050] As explained with reference to Figure 4, the processing circuit 71 consists of a processor and memory and controls the overall operation of the mobile body 51. The processing circuit 71 also includes an environmental body information collection unit 91, a log information management unit 92, an abnormality suggestion processing unit 93, a drive control unit 94, and a camera control unit 95.

[0051] The environmental device information collection unit 91 controls the sensor 82 to acquire various sensor detection values ​​in the surrounding environment and inside the mobile device 51 while the mobile device 51 is moving (acquires environmental device information), and supplies it to the log information management unit 92 and the abnormality suggestion processing unit 93.

[0052] When the log information management unit 92 acquires sensor detection values ​​supplied from the environmental device information collection unit 91 and operation signals supplied from the operation input unit 52b of the controller 52, it stores them in chronological order as log information 74a in the storage unit 74, associating them with time information, and supplies them to the abnormality suggestion processing unit 93 as necessary. The operation signals supplied from the operation input unit 52b of the controller 52 may include not only operation signals for controlling the movement of the mobile body 51, but also general operation signals related to the operation of the mobile body 51, for example, operation signals for setting the exposure mode and focus mode of the mounted camera 81.

[0053] The abnormality suggestion processing unit 93 compares real-time sensor detection values ​​supplied from the environmental aircraft information collection unit 91 and real-time operation signals supplied from the operation input unit 52b with their log information 74a, detects an abnormality according to the comparison result, supplies it to the controller 52, and causes the UI display unit 52a to display information indicating that an abnormality has been detected. If an abnormality is detected, the abnormality suggestion processing unit 93 may control the drive control unit 94 to stop the flight.

[0054] Furthermore, if the abnormality suggestion processing unit 93 detects an abnormality by comparing real-time sensor detection values ​​and operation signals with their log information 74a, it may generate suggestion information that proposes past operation settings as recommended settings for changing the operation settings, supply it to the controller 52, and have the suggestion information displayed on the UI display unit 52a along with information indicating that an abnormality has been detected.

[0055] If the abnormality detection processing unit 93 does not detect an abnormality by comparing the real-time sensor detection value and operation signal with the log information 74a, it supplies a drive control signal to the drive control unit 94 to drive the drive unit 83 based on the current operation settings.

[0056] When the abnormality detection processing unit 93 detects an abnormality by comparing real-time sensor detection values ​​and operation signals with their log information 74a, information indicating that an abnormality has been detected is presented to the controller 52. When the operation input unit 52b is operated and an operation signal indicating a change in operation settings is supplied, the corresponding operation settings are changed and a drive control signal is supplied to the drive control unit 94 to drive the drive unit 83.

[0057] The drive control unit 94 drives the drive unit 83 based on the drive control signal supplied by the abnormality suggestion processing unit 93.

[0058] If the anomaly detection processing unit 93 does not detect an anomaly by comparing the real-time sensor detection value and operation signal with the log information 74a, it supplies a camera control signal to the camera control unit 95 to operate the camera 81 based on the current operation settings. If the anomaly detection processing unit 93 detects an anomaly by comparing the real-time sensor detection value and operation signal with the log information 74a, the controller 52 displays information indicating that an anomaly has been detected. When the operation input unit 52b is operated and an operation signal indicating a change in the operation settings related to the camera 81 is supplied, the anomaly detection processing unit 93 changes the corresponding operation settings for the camera 81 and supplies a camera control signal to the camera control unit 95 to operate the camera 81.

[0059] The camera control unit 95 operates the camera 81 to take an image based on the camera control signal supplied by the abnormality suggestion processing unit 93.

[0060] More specifically, the information accumulated in log information 74a includes real-time input of operation information, saved (i.e., previously entered) setting information and various past registration information, and aircraft information.

[0061] The information on operation inputs that is input in real time may include, for example, the touch position if the operation input unit 52b is a touch panel, the input voice if the operation input unit 52b supports voice input, the operation input of the stick buttons if the operation input unit 52b is configured to be operated by stick buttons, and screen transitions presented on the UI display unit 52a supplied by the error suggestion processing unit 93.

[0062] Furthermore, the saved (i.e., entered via operation input) setting information and various past registration information may include, for example, a mission database, geofences, characteristics of the target to be recognized, aircraft user settings (RTH altitude, obstacle brake settings, maximum flight speed), controller 52 user settings (operation mode assignment, button function assignment), camera settings (exposure mode and focus mode settings, etc.), and user information (ID, physical condition, flight experience).

[0063] Furthermore, the aircraft information may include, for example, information from external sensors (including sensor 82) (obstacle sensor, image, position, wind speed), device status (battery level, fault information), and information obtained from the server computer 53 (weather data during movement, map data of the location).

[0064] <Details of the operation of the error suggestion processing unit> Next, with reference to Figure 6, the details of the operation of the error suggestion processing unit 93 will be explained.

[0065] Basically, the abnormality notification processing unit 93 compares the sensor detection values ​​and operation inputs, which are input in real time, with past information stored in the log information 74a, and supplies the controller 52 with notifications of abnormalities based on the comparison results and suggested information to resolve the abnormality, which are then displayed on the UI display unit 52a.

[0066] More specifically, the anomaly detection processing unit 93 determines from the comparison results of the two whether there is no difference and they (almost) match. If there is no difference and they (almost) match when movement was achieved without anomalies in the past, it determines that there is no anomaly. The anomaly detection processing unit 93 also determines from the comparison results of the two whether an anomaly has occurred if they do not match. When the anomaly detection processing unit 93 determines that an anomaly has occurred, it generates a notification image warning that an anomaly has occurred, supplies it to the controller 52, and has it displayed on the UI display unit 52a.

[0067] For example, when a difference is detected by comparing past sensor detection values ​​stored as log information 74a with real-time sensor detection values, the abnormality suggestion processing unit 93 may determine that an abnormality has occurred, generate a warning notification, supply it to the controller 52, and have it displayed on the UI display unit 52a.

[0068] A difference in sensor readings is detected in situations such as when the wind speed is stronger than the highest wind speed recorded during past flights, when the flight environment differs from past flights (for example, when it was daytime in the past but nighttime in the present, or when it was sunny in the past but stormy in the present), or when the obstacle detection setting, which stops the aircraft when an obstacle is detected, was ON in the past but is OFF in the present when the flight has started.

[0069] Furthermore, for example, if the past sensor detection values ​​stored as log information 74a are compared with real-time sensor detection values ​​and match an abnormal condition in the past, the abnormality suggestion processing unit 93 may determine that an abnormality has occurred, generate a warning notification, supply it to the controller 52, and have it displayed on the UI display unit 52a.

[0070] Here, when the sensor detected value is almost identical to an abnormal condition with no significant difference, it means, for example, when it is almost identical (or even greater) to a wind speed that has caused a crash in a past flight, or when it is almost identical to a time of day or weather conditions that have caused a crash in a past flight, with no significant difference.

[0071] Furthermore, when a difference is detected by comparing past operation inputs stored as log information 74a with real-time operation inputs, the abnormality suggestion processing unit 93 determines that an abnormality has occurred, generates a warning notification, supplies it to the controller 52, and has it displayed on the UI display unit 52a.

[0072] Here, the difference from past operation inputs includes cases where there are no specific operations that are expected to be routine, such as clearing the way or compass calibration, or when a real-time sequence of operation patterns consisting of multiple operation inputs does not match a sequence of operation patterns consisting of multiple operations performed in the past.

[0073] Furthermore, if, by comparing past operation inputs that can be considered abnormal and stored as log information 74a, and real-time operation inputs, there is no difference and they are considered to be nearly identical, the abnormality suggestion processing unit 93 may determine that an abnormality has occurred, generate a warning notification, supply it to the controller 52, and have it displayed on the UI display unit 52a.

[0074] In this case, "no difference from past operation inputs and nearly identical" includes, for example, cases where an operation input is made that is considered to be nearly identical to a series of operation inputs that can be considered a specific pattern of operation inputs that were recorded in the past when an anomaly such as a crash is suspected to have occurred.

[0075] When the abnormality detection processing unit 93 receives a notification on the UI display unit 52a of the controller 52 indicating that these abnormalities have been detected, and the operation input is changed, it supplies drive control information based on the newly changed operation input to the drive control unit 94, causing the drive unit 83 to be controlled.

[0076] Furthermore, when the operation input unit 52b is operated separately and a request is made to display log information 74a, the abnormality suggestion processing unit 93 may, in response to the request, read the log information 74a and supply it to the controller 52, causing it to be displayed on the UI display unit 52a or the like.

[0077] Furthermore, the comparison between the real-time sensor detection values ​​and operation inputs and the past information stored in the log information 74a, as described above, compares the real-time sensor detection values ​​and operation inputs with the sensor detection values ​​and operation inputs stored in the past, but also compares the setting information set based on the sensor detection values ​​and operation inputs (setting information set in conjunction with the sensor detection values ​​and operation inputs).

[0078] In other words, when various setting information is changed due to sensor detection values ​​or operation input, the changed setting information is also compared with the setting information stored in log information 74a (i.e., previously entered via operation input). For example, if an operation setting such as RTH altitude setting is changed via operation input, it is also compared with the past RTH altitude setting or other operation setting stored in log information 74a (i.e., previously entered via operation input).

[0079] In other words, the abnormality notification processing unit 93 compares the current sensor detection value and operation input with past sensor detection value and operation input stored in the log information 74a, and also compares the various setting information set in accordance with the current sensor detection value and operation input with the various setting information set (or pre-set) in accordance with past sensor detection value and operation input stored in the log information 74a. Based on the comparison results, the abnormality notification processing unit 93 determines whether or not there is an abnormality, and if it determines that there is an abnormality, it generates a notification image indicating a warning, supplies it to the controller 52, and has it displayed on the UI display unit 52a.

[0080] <Mobile Body Control Processing (Part 1)> Next, the mobile body control processing by the mobile body 51 will be explained with reference to the flowchart in Figure 7.

[0081] In step S31, the abnormality suggestion processing unit 93 controls the communication unit 75 to determine whether the operation input unit 52b of the controller 52 has been operated and whether an operation signal corresponding to the operation content has been supplied.

[0082] If, in step S31, the operation input unit 52b of the controller 52 is operated and it is determined that an operation signal corresponding to the operation content has been supplied, the process proceeds to step S32.

[0083] In step S32, the abnormality detection processing unit 93 controls the communication unit 75 to receive the operation input and corresponding operation signal supplied from the controller 52. If it is determined in step S31 that no operation signal corresponding to the operation content has been supplied, the processing in step S32 is skipped.

[0084] In step S33, the environmental device information collection unit 91 controls the sensor 82 to acquire a predetermined sensor detection value and outputs it to the log information management unit 92 and the abnormality suggestion processing unit 93.

[0085] In step S34, the log information management unit 92 stores the operation input corresponding to the operation signal and the sensor detection value in chronological order in the log information 74a of the storage unit 74, associating them with time information. Note that if there is no operation input, only the sensor detection value may be stored in the log information 74a.

[0086] In step S35, the abnormality suggestion processing unit 93 compares the sensor detection value supplied from the environmental unit information collection unit 91 and the operation input supplied from the controller 52 with the corresponding log information 74a.

[0087] In step S36, the abnormality suggestion processing unit 93 determines whether or not an abnormality has occurred based on the comparison result between the sensor detection value supplied from the environmental unit information collection unit 91 and the operation input supplied from the controller 52 and the corresponding log information 74a.

[0088] If it is determined in step S36 that an abnormality has occurred, the process proceeds to step S37.

[0089] In step S37, the abnormality detection processing unit 93 generates a notification image, such as the one shown in Figure 8, indicating that an abnormality has occurred, and supplies it to the controller 52 to display it on the UI display unit 52a. If it is not determined in step S36 that an abnormality has occurred, the processing in step S37 is skipped.

[0090] In step S38, it is determined whether or not the termination of the mobile body control process has been instructed. If termination is not instructed, the process returns to step S31, and the subsequent processes are repeated.

[0091] Then, in step S38, if the termination of the mobile body control process is instructed, the process ends.

[0092] In Figure 8, the UI display unit 52a displays a notification image 111 in which the word "Warning" is written at the top, "Unusual flight behavior detected" is written below it, and "No vanguard flight" is written below that, in order to make the user aware that an abnormality has occurred.

[0093] In other words, as explained with reference to Figure 2, the notification image 111 in Figure 8 warns the user that the flight has commenced without any preliminary flight being conducted to check for any abnormalities in the mobile unit 51.

[0094] In addition, while Figure 8 has described an example in which a warning is presented as text information via a notification image 111 on the UI display unit 52a, the warning may be presented not only by display but also by sound, or the controller 52 may be equipped with a vibration function to present the warning by vibration along with the text and sound information.

[0095] This allows the user to recognize if they have forgotten an operation they perform as part of their routine to check for any abnormalities in the mobile unit 51, such as a preliminary flight, and to repeat the forgotten routine if necessary, thereby enabling safe flight.

[0096] As a result, it becomes possible to effectively assist in the remote control of mobile objects.

[0097] In addition, although the above has described an example in which the abnormality suggestion processing unit 93 is provided in the processing circuit 71 of the mobile unit 51, it may also be provided in a location other than the mobile unit 51, for example, in the controller 52 or the server computer 53.

[0098] In other words, in the above, the abnormality suggestion processing unit 93 provided on the mobile body 51 detects the presence or absence of an abnormality based on the operation input supplied from the controller 52 and the sensor detection value supplied from the environmental body information collection unit 91. When an abnormality is detected, it generates a notification image 111, supplies it to the controller 52, and displays it on the UI display unit 52a.

[0099] However, the controller 52 and server computer 53 may be equipped with an abnormality suggestion processing unit 93, which detects the presence or absence of an abnormality based on the operation input to the controller 52 and the sensor detection values ​​supplied from the environmental body information collection unit 91 of the mobile body 51. When an abnormality is detected, a notification image 111 is generated and displayed on the UI display unit 52a.

[0100] Furthermore, while the above has described an example in which log information 74a is accumulated for each mobile unit 51, log information 74a accumulated in multiple other mobile units 51 may be shared among the multiple mobile units 51.

[0101] Furthermore, log information 74a may not only be stored within the mobile unit 51, but log information 74a from multiple other mobile units 51 may also be uploaded and shared to a server computer 53 or a cloud server on a network (not shown).

[0102] Furthermore, by uploading and sharing log information 74a from the time of a crash or collision to a cloud server or the like, anomalies may be detected when matching operation inputs or sensor detection values ​​are found, thereby enabling warnings against crashes and collisions.

[0103] Furthermore, if multiple users use the same mobile device 51, log information may be registered separately for each user within the same log information 74a, and the user can switch between them depending on who is using the device. In addition, if multiple users use the same mobile device 51, the log information of an experienced user within the same log information 74a may be made available to users who are not familiar with remotely operating the mobile device 51.

[0104] Alternatively, a learning model that realizes the processing of the anomaly suggestion processing unit 93 described above may be generated by learning using log information 74a from multiple users, and the learned learning model may be used as the anomaly suggestion processing unit 93.

[0105] Furthermore, if the anomaly suggestion processing unit 93 is implemented using a trained learning model based on log information 74a, the anomaly suggestion processing unit 93 may be configured to estimate and recognize the current state of the mobile body 51 and the user 50 based on real-time operation input and sensor detection values.

[0106] Furthermore, the controller 52 may be used at a distance where the user 50 using the controller 52 can visually observe the mobile object 51, or it may be possible to achieve complete remote control at a distance where the user using the controller 52 cannot visually observe the mobile object 51, using satellite communication or the like.

[0107] <<3. Second Embodiment>> In the above, we have described an example in which an abnormality is notified when it is determined that an abnormality has occurred based on the comparison result of the sensor detection value supplied from the environmental device information collection unit 91, the operation input supplied from the controller 52, and the corresponding log information 74a.

[0108] However, when an abnormality is detected, the system may notify the system of the abnormality and offer a suggestion for changing the settings. If the system accepts the suggestion, the settings may be changed to avoid the abnormality.

[0109] More specifically, when the comparison results between real-time sensor detection values ​​and operation inputs and past operation inputs and past operation settings in log information 74a do not match and it is determined that an abnormality has occurred, the past operation inputs and past operation settings stored in log information 74a may be presented as recommended settings, and a suggestion to change the settings may be made.

[0110] For example, consider a case where, as recorded in log information 74a, the operation input unit 52b was operated before the start of automatic driving to set the exposure mode, focus mode, WB (White Balance), and image quality settings of the camera 81.

[0111] In this case, if automatic operation is started without the setting operations registered as past log information 74a described above being performed based on real-time sensor detection values ​​and operation inputs, the abnormality suggestion processing unit 93 may detect the abnormality, stop the automatic operation, and present the exposure mode setting, focus mode setting, WB (White Balance) setting, and image quality setting recorded in the log information 74a as recommended settings that can be changed, thereby suggesting setting information.

[0112] Furthermore, if the real-time sensor detection values ​​and operation inputs are compared with past sensor detection values ​​and operation inputs that constitute log information 74a, and the results of the comparison with past sensor detection values ​​or past operation settings that are deemed dangerous, such as accident reports, match and it is determined that an abnormality has occurred, then past operation inputs and past operation settings stored in log information 74a that do not have accident reports and have a history of safe flight may be presented as recommended settings that can be changed, and a suggestion for changing the settings may be made.

[0113] Furthermore, when it is determined that an abnormality has occurred, processing may be performed to restrict the movement and settings of the mobile unit 51. For example, this may include stopping the flight, preventing operation input and causing it to hover in the same position, or immediately performing RTH (Return to Home).

[0114] The basic configuration is the same as that of the mobile body 51 in Figure 5, so its explanation will be omitted.

[0115] <Mobile Unit Control Processing (Part 2)> Next, referring to the flowchart in Figure 9, we will explain mobile unit control processing that, when it is determined that an abnormality has occurred, notifies the system of the abnormality and presents a suggestion for changing the settings. If the system accepts the suggestion for changing the settings, it changes the settings to avoid the abnormal state.

[0116] Furthermore, the processes in steps S51 to S55 and S60 in the flowchart of Figure 9 are the same as the processes in steps S31 to S35 and S38 in the flowchart of Figure 7, so their explanation will be omitted.

[0117] In other words, through the processing in steps S51 to S55, the operation input corresponding to the operation signal and the sensor detected value are stored in the log information 74a of the storage unit 74 in a time series, corresponding to the time information, and the sensor detected value and the operation input are compared with the corresponding log information 74a.

[0118] In step S56, the abnormality suggestion processing unit 93 determines whether an abnormality has occurred and whether there are any setting candidates that can be presented, based on the comparison result between the sensor detection value supplied from the environmental device information collection unit 91 and the operation input supplied from the controller 52 and the corresponding log information 74a.

[0119] For example, if log information 74a records that the operation input unit 52b was operated before the start of automatic driving to set the exposure mode and focus mode of the camera 81, and if automatic driving is started without the setting operation registered as past log information 74a being performed based on real-time sensor detection values ​​and operation input, the abnormality suggestion processing unit 93 will detect an abnormality.

[0120] Furthermore, at this time, the log information 74a records that the operation input unit 52b was operated before the start of automatic driving to set the exposure mode and focus mode of the camera 81. Therefore, the abnormality suggestion processing unit 93 determines that there are setting candidates that can be presented, and the process proceeds to step S57.

[0121] In step S57, the abnormality suggestion processing unit 93 generates a notification image 121, such as the one shown in Figure 10, which notifies the controller 52 that an abnormality has occurred and suggests recommended settings that are candidates for setting changes. This notification image is then supplied to the controller 52 and displayed on the UI display unit 52a.

[0122] In Figure 10, the UI display unit 52a displays "Warning" at the top to indicate that an abnormality has occurred, followed by "Unusual settings. Automatic flight has been stopped." below that, from top to bottom, "Recommended settings," "Exposure mode: Center-weighted," and "Focus mode: Auto." Below this, in the lower left corner, a notification image 121 is displayed, consisting of a button 121a labeled "Apply settings" and a button 121b labeled "Cancel."

[0123] In other words, the notification image 121 in Figure 10 shows the user that the settings are different from usual and that automatic flight has been stopped. Furthermore, the setting information registered in the log information 74a is presented as a "recommended setting" that can be changed, and specifically, the user is suggested to set the exposure mode to center-weighted and the focus mode to auto.

[0124] If the user accepts this suggestion and agrees to set the exposure mode to center-weighted and the focus mode to auto, they press the button 121a labeled "Apply Settings". If they do not accept, they press the button 121b labeled "Cancel".

[0125] Furthermore, in Figure 10, the warning may be presented not only by displaying the notification image 121, but also by sound, or the controller 52 may be equipped with a vibration function so that the warning is presented by vibration in addition to display and sound.

[0126] In step S58, the abnormality suggestion processing unit 93 determines whether the operation input unit 52b has been operated by the user to accept the suggestion. That is, in the case of Figure 10, it is determined whether or not the operation to accept the suggestion has been performed by whether or not the button 121a labeled "Apply Settings" is pressed.

[0127] In step S58, if the operation input unit 52b is operated by the user and it is determined that an operation to accept the proposal has been performed, the process proceeds to step S59.

[0128] In step S59, the abnormality suggestion processing unit 93 changes the operation settings to the suggested content, generates an image indicating that the operation settings have been changed, outputs it to the controller 52, and displays it on the UI display unit 52a.

[0129] Furthermore, if it is determined in step S58 that the operation to accept the proposal has not been performed, the process in step S59 is skipped.

[0130] Then, in step S60, it is determined whether or not the termination of the mobile body control process has been instructed. If termination is not instructed, the process returns to step S51, and the subsequent processes are repeated. If termination of the mobile body control process is instructed, the process ends.

[0131] Through the above process, the user will be able to recognize any abnormalities in input operations or operating settings, and will also be presented with recommended settings for necessary changes, enabling them to quickly change the operating settings and resume flight promptly.

[0132] Furthermore, even if the user intentionally performs a new input operation or operation setting that differs from the past, the operation settings will not be unilaterally changed. By not accepting candidates for changing the operation settings, it becomes possible to remotely control the mobile unit 51 with new operation settings that differ from those previously made.

[0133] At this time, the new operation settings will also be stored in log information 74a, and from the next time onward, they will be treated as operation settings performed in the past.

[0134] Furthermore, when determining whether or not there is an abnormality by comparing the current sensor detection value and operation input with the corresponding log information 74a, the current sensor detection value and operation input may be compared with sensor detection value and operation input stored in the log information 74a a predetermined number of times or more, so that an abnormality is detected only when they are different.

[0135] Furthermore, by ensuring that only the current sensor detection value and operation input are compared with the sensor detection value and operation input that have been repeatedly stored in the log information 74a a predetermined number of times or more, if the user intentionally changes to a new operation input or operation setting midway through, it will be detected as an abnormality until the new operation input or operation setting is repeatedly stored a predetermined number of times. However, once it has been repeated more than the predetermined number of times, it will be compared with the new operation input or operation setting, and thereafter it will no longer be detected as an abnormality.

[0136] In either case, the result is that it becomes possible to effectively assist in the remote control of mobile objects.

[0137] <<4. Third Embodiment>> In the above, we have described an example in which, when it is determined that an abnormality has occurred, the system notifies the system of the abnormality and presents a suggestion for changing the settings, and when an operation input is made to accept the suggestion for changing the settings, the settings are changed to avoid the state in which the abnormality has occurred.

[0138] However, if an anomaly is detected, and the type of anomaly requires immediate action to avoid it, the system may change the operating settings to allow for avoidance of the anomaly before notifying the system of the anomaly or suggesting configuration changes, and then notify the system of the change in operating settings retrospectively.

[0139] More specifically, for example, the RTH (Return to Town) altitude is often set to different heights depending on whether it's an area where interference with tall buildings is unlikely, such as a riverbank or suburban area, or an area with many tall buildings where interference is likely to occur, such as an urban area with many skyscrapers.

[0140] For example, consider a case where, in log information 74a, past flights often had the RTH altitude setting set to 70m in urban areas and 30m in suburban areas.

[0141] Note that while the RTH altitude setting is generally set to around 30m by default, in urban areas with many tall buildings, it is necessary to fly at an altitude higher than the height of the buildings. Therefore, it is necessary to set the altitude higher than 30m depending on the height of the buildings, and here we will use an example where the RTH altitude setting is often set to 70m.

[0142] In this case, when the abnormality suggestion processing unit 93 obtains information about the current location, including the flight altitude, from the GPS of the sensor 82, it controls the communication unit 75 to access the server computer 53 that provides map information, obtains the map information, and determines whether the current location is in an urban area or a suburban area.

[0143] For example, if the current sensor detection value indicates a flight altitude of 40m, and the current location is in an urban area, the environmental aircraft information collection unit 91 sets the RTH altitude setting to 70m and continues flying at a flight altitude of 40m, based on map information.

[0144] Furthermore, when the current location is in the suburbs, if the flight altitude is 40m, the RTH altitude setting is set to 30m. The abnormality notification processing unit 93 then generates information indicating that the RTH altitude setting has been changed and transmits it to the controller 52, which displays it on the UI display unit 52a. In other words, at this time, the mobile body 51 is at a flight altitude of 40m, which is higher than the RTH altitude setting of 30m, so RTH will start immediately.

[0145] This process allows for the rapid modification of RTH altitude settings, such as those related to travel (flight), based on the current location and map information, without requiring the user to propose or consent to the change. The system then notifies the user that the setting has been changed.

[0146] This makes it possible, for example, for the mobile object 51 to continue flying even if it is rapidly moving from the suburbs into an urban area with many high-rise buildings.

[0147] Furthermore, the types of anomalies that require a proposed setting change to be made to the user and their consent before the change is implemented to avoid the anomaly may be, for example, types of anomalies that are of low importance and low urgency related to movement (flight). In addition, the types of anomalies for which a setting change is performed after obtaining the user's consent and the types of anomalies for which a setting change is performed without obtaining the user's consent may be set in advance by the user 50.

[0148] <Mobile Device Control Processing (Part 3)> Next, referring to the flowchart in Figure 11, we will explain the mobile device control processing when an abnormality is determined to have occurred, and the operating settings are changed immediately without obtaining the user's consent, according to the type of abnormality that has occurred, and the user is notified of the change in operating settings afterward.

[0149] Furthermore, the processes in steps S71 to S76 and S78 to S81 in the flowchart of Figure 11 are the same as the processes in steps S51 to S60 in the flowchart of Figure 9, so their explanation will be omitted.

[0150] In other words, through the processing in steps S71 to S76, the operation input corresponding to the operation signal and the sensor detected value are associated and stored in the log information 74a of the storage unit 74 by overwriting it. The sensor detected value, the operation input, and the corresponding log information 74a are then compared, and if an abnormality is found, and there are candidates for suggesting setting changes to avoid the abnormality, the process proceeds to step S77.

[0151] In step S77, the environmental device information collection unit 91 determines whether the type of abnormality for which there are candidates for setting change proposals is of a type for which the operation settings must be changed only after obtaining the user's consent.

[0152] In step S77, if it is determined that the type of anomaly for which there are candidate suggestions for setting changes to avoid the anomaly is a type that must be changed only after obtaining the user's consent, the process proceeds to step S78. That is, in this case, candidate suggestions for setting changes to avoid the anomaly are presented, and if the user consents to the suggested setting changes is obtained, the operation settings are changed and the user is notified of the setting change. If consent is not obtained, the operation settings are not changed.

[0153] Furthermore, if in step S77 it is determined that the type of abnormality that is a candidate for a setting change is not one that requires user consent before being changed, steps S78 and S79 are skipped, and in step S80 the operation setting is changed without user consent. For example, a notification image 131 notifying the user of the setting change, as shown in Figure 12, is generated, notified to the controller 52, and displayed on the UI display unit 52a.

[0154] In the notification image 131 of Figure 12, it says "Drone settings have been changed automatically," and below that, it says "RTH altitude: 70m," indicating that the RTH altitude setting of the mobile object 51, the drone, has been automatically changed to 70m.

[0155] Through the above process, if it is determined that an anomaly has occurred, and depending on the type of anomaly, if it is of a high severity or urgency that requires a rapid corrective action, it becomes possible to quickly change the operating settings without notifying the system of the anomaly or suggesting any setting changes, and to notify the system of the changes retrospectively.

[0156] This allows for rapid changes to operating settings when an anomaly of a high importance and urgency occurs, enabling appropriate movement (flight).

[0157] As a result, it becomes possible to effectively assist in the remote control of mobile objects.

[0158] <<5. Fourth Embodiment>> In the above, we have described an example in which, when it is determined that an abnormality has occurred, depending on the type of abnormality, if it is of a high importance, high urgency, and requires the selection of a rapid avoidance method, the operating settings are changed before notification of the occurrence of the abnormality or before suggesting setting changes, and the change in operating settings is notified retrospectively.

[0159] However, it is also possible to estimate the items the user is focusing on from the patterns of operation inputs recorded in log information 74a, and to proactively present information related to the estimated items before the user requests it.

[0160] The abnormality suggestion processing unit 93 performs pattern matching to search for patterns of operation inputs that are frequently performed under predetermined conditions, based on the operation input and sensor detection value information recorded in the log information 74a.

[0161] The abnormality suggestion processing unit 93 then estimates that the processing to be executed based on the operation input pattern under the matched predetermined conditions is the processing for the item that the user is paying attention to.

[0162] More specifically, consider, for example, a case where, while moving at high speed, the user frequently (for example, at intervals shorter than a predetermined time interval) makes an operation input to display the battery level, as recorded in the log information 74a.

[0163] In this case, under the condition that the movement speed is faster than a predetermined value, a pattern of operation input consisting of a combination of an operation input that opens a UI including an operation button for displaying the battery level, and an operation input in which the operation button that instructs the display of the battery level is pressed from the opened UI, is repeatedly searched for more than a predetermined number of times.

[0164] Hereafter, a pattern of operation input consisting of an operation input that opens a UI including an operation button for displaying the battery level while moving at high speed, and an operation input in which an operation button that instructs the display of the battery level is pressed from the opened UI, will also be referred to as a battery level display operation input pattern.

[0165] In other words, based on the information recorded in log information 74a, if the sensor detects that the movement speed is faster than a predetermined speed, and pattern matching is performed using the battery level display operation input pattern, the battery level display operation input pattern will be searched repeatedly.

[0166] In such cases, the abnormality suggestion processing unit 93 estimates that the process of displaying the battery level while the user is moving at high speed is a process for an item that the user is paying attention to.

[0167] Then, the abnormality suggestion processing unit 93 automatically and repeatedly executes the processing performed by the battery level display operation input pattern at the average time interval during high-speed movement, treating it as processing for an item that the user is paying attention to, regardless of user input.

[0168] Through this process, when the same input pattern, such as the battery level display input pattern, is repeated a predetermined number of times or more while moving at high speed, the processing performed by the battery level display input pattern will be automatically preemptively executed at the average time interval between repeated inputs, and the battery level will be displayed.

[0169] <Mobile Body Control Processing (Part 4)> Next, referring to the flowchart in Figure 13, we will explain the mobile body control processing by the mobile body 51 when the processing or items that the user is paying attention to are estimated from the patterns of operation inputs recorded in the log information 74a, and the estimated processing is executed in advance before the user inquires.

[0170] Furthermore, the processes in steps S101 to S110 and S115 in the flowchart of Figure 13 are the same as the processes in steps S71 to S81 in the flowchart of Figure 11, so their explanation will be omitted.

[0171] In other words, the processes described above with reference to the flowchart in Figure 11 are carried out by the processes in steps S101 to S110.

[0172] In step S111, the abnormality suggestion processing unit 93 performs pattern matching based on past sensor detection values ​​and operation input information recorded in log information 74a.

[0173] In step S112, the abnormality suggestion processing unit 93 determines whether there is a pattern that matches a predetermined number or more of the patterns of sensor detection values ​​and operation inputs recorded in the log information 74a.

[0174] If, in step S112, it is determined that there is a pattern that matches a predetermined number or more of the sensor detection values ​​and operation input patterns, the process proceeds to step S113.

[0175] In step S113, the abnormality suggestion processing unit 93 estimates that the processing to be executed based on a pattern of combinations consisting of a predetermined number or more matched operation inputs corresponds to the processing of an item that the user is paying attention to.

[0176] In step S114, the abnormality suggestion processing unit 93 executes processing for the item the user is focusing on, which is performed based on a pattern of combinations consisting of a predetermined number or more matched operation inputs that constitute the estimated result, at the average time interval in which processing for the item the user is focusing on is detected.

[0177] For example, as described above, if the same operation input pattern, such as the battery level display operation input pattern, is repeated a predetermined number of times or more while moving at high speed, thereafter, when moving at high speed, a notification image 141 that displays the battery level, as shown in the upper part of Figure 14, is generated at the average time interval when the battery level display operation input pattern is repeatedly input, and is supplied to the controller 52 and displayed on the UI display unit 52a.

[0178] In Figure 14, notification image 141 displays "Battery level: 70%", indicating that the battery level is 70%.

[0179] Furthermore, in addition to presentation by image 141, as shown at the bottom of Figure 14, the controller 52 may also have a built-in speaker 52c or the like to make a statement such as "The battery level is 70%."

[0180] Through the above process, when the user repeatedly inputs an item that they are focusing on, if this input is repeated more than a predetermined number of times, the system can estimate that the user is repeatedly inputting an item that they are focusing on, and automatically present the item even if the user does not input an item at the average time interval detected.

[0181] As a result, it becomes possible to effectively assist in the remote control of mobile objects.

[0182] <<6. Description of a computer using this technology>>

[0183] The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0184] Figure 15 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0185] In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0186] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0187] The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0188] The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009.

[0189] The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0190] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0191] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0192] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0193] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0194] When the computer executes a program to perform the series of processes described above, the processing circuit 1001 (its processor) executes the program and functions as the environmental device information collection unit 91, log information management unit 92, abnormality suggestion processing unit 93, drive control unit 94, and camera control unit 95 in the processing circuit 71 of Figure 4.

[0195] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory, for example, only the processing circuit 1001, or a combination of the processing circuit 1001 to the bus 1004, etc., constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0196] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0197] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0198] Furthermore, for example, the various technologies relating to this disclosure can be implemented independently, as long as they do not conflict with each other. Of course, any combination of the disclosures can also be implemented. For example, some or all of the disclosures described in one embodiment can be implemented in combination with some or all of the disclosures described in another embodiment. Also, some or all of the aforementioned disclosures can be implemented in combination with other technologies not described above.

[0199] Furthermore, this disclosure may also take the following configurations: <1> An information processing method comprising: receiving operation input on a mobile body; collecting environmental and body information relating to the environment of the mobile body and the body of the mobile body; managing the operation input and environmental and body information as log information; comparing the current operation input and environmental and body information with the log information; and presenting notification information based on the comparison result obtained by the comparison. <2> The information processing method according to <1>, wherein the comparison includes determining whether there is an abnormality based on whether there is a difference between the current operation input and environmental and body information and the log information, and presenting the notification information includes presenting the notification information based on the comparison result obtained by the comparison. <3> The information processing method according to <2>, wherein the comparison includes determining that there is an abnormality when there is a difference, and presenting the notification information includes presenting notification information that notifies that the abnormality has occurred based on the comparison result. <4> The information processing method according to <3>, wherein the log information consists of past operation input and environmental and body information when the mobile body is in a state without abnormality. <5> The information processing method according to <2>, wherein the comparison includes determining that there is an abnormality when there is no difference, and the presentation of the notification information includes presenting the notification information that notifies that an abnormality has occurred based on the comparison result. <6> The information processing method according to <5>, wherein the log information consists of past operation inputs and environmental device information when the mobile body is in an abnormal state. <7> The information processing method according to <1>, wherein the information relating to the operation inputs and environmental device information further includes setting information of the mobile body set based on the operation inputs and environmental device information. <8> The information processing method according to <7>, wherein if the comparison determines that there is an abnormality based on the comparison result, the presentation of the notification information includes presenting the notification information based on the comparison result, past operation inputs and environmental device information included in the log information, and at least one of the setting information as a recommended setting for change.<9> If consent is obtained to change at least one of the past operation inputs and environmental device information and the setting information to the recommended settings, the comparison includes changing at least one of the current operation inputs and environmental device information and the current setting information to the recommended settings, and the presentation of the notification information includes presenting information indicating that at least one of the current operation inputs and environmental device information and the current setting information has been changed to the recommended settings. The information processing method described in <8>. <10> If it is determined that there is an abnormality based on the comparison result, the comparison includes changing at least one of the past operation inputs and environmental device information and the past setting information to the recommended settings, and the presentation of the notification information includes presenting information indicating that at least one of the current operation inputs and environmental device information and the current setting information has been changed to the recommended settings. The information processing method described in <8>. <11> The information processing method according to <10>, wherein the comparison is determined to be abnormal based on the comparison result, and the type of abnormality is a predetermined type, the comparison is performed to change at least one of the past operation input and environmental unit information and the past setting information to the recommended setting. <12> The information processing method according to <7>, wherein the comparison is performed to perform pattern matching in the log information between the operation input and environmental unit information and the past setting information, and the processing of the matched pattern is repeated. <13> The information processing method according to <12>, wherein the comparison is performed to repeat the processing of the matched pattern at the average time interval at which matching occurs in the pattern matching. <14> An information processing device comprising: an operation input receiving unit that receives operation inputs from a mobile device; an environmental body information collection processing unit that collects environmental body information relating to the environment of the mobile device and the body of the mobile device; a log management processing unit that manages the operation inputs and environmental body information as log information; a comparison unit that compares the current operation inputs and environmental body information with the log information; and a notification information presentation unit that presents notification information based on the comparison results of the comparison unit.<15> The information processing device according to <14>, wherein the comparison unit determines whether there is an abnormality based on whether there is a difference between the current operation input and environmental device information and the log information, and the notification information presentation unit presents the notification information based on the comparison result by the comparison unit. <16> The information processing device according to <15>, wherein the comparison unit determines that there is an abnormality when there is a difference, and the notification information presentation unit presents the notification information that notifies that the abnormality has occurred based on the comparison result. <17> The information processing device according to <15>, wherein the comparison unit determines that there is an abnormality when there is no difference, and the notification information presentation unit presents the notification information that notifies that the abnormality has occurred based on the comparison result. <18> The information processing device according to <14>, wherein the information relating to the operation input and environmental device information further includes the setting information of the mobile body set based on the operation input and environmental device information. <19> If the comparison unit determines that there is an abnormality based on the comparison result, the notification information presentation unit presents the notification information based on the comparison result of the comparison unit, and at least one of the past operation input and environmental device information and the setting information included in the log information as a recommended setting for changing the setting, as described in <18>. <20> The comparison unit performs pattern matching in the log information between the operation input and environmental device information and the past setting information, and repeats the processing of the matched pattern, as described in <18>.

[0200] 31 Mobile control system, 51 Mobile unit, 52 Controller, 52a Operation input unit, 52b UI display unit, 53 Server computer, 74a Log information, 81 Camera, 82 Sensor, 83 Drive unit, 91 Environmental unit information collection unit, 92 Log information management unit, 93 Anomaly suggestion processing unit, 94 Drive control unit, 95 Camera control unit

Claims

1. An information processing method comprising: receiving operation input from a mobile device; collecting environmental and system information relating to the environment of the mobile device and the body of the mobile device; managing the operation input and environmental and system information as log information; comparing the current operation input and environmental and system information with the log information; and presenting notification information based on the comparison results obtained from the comparison.

2. The information processing method according to claim 1, wherein the comparison includes determining whether there is an abnormality based on whether there is a difference between the current operation input and environmental device information and the log information, and the presentation of the notification information includes presenting the notification information based on the comparison result obtained by the comparison.

3. The information processing method according to claim 2, wherein the comparison includes determining that there is an abnormality when there is a difference, and the presentation of the notification information includes presenting the notification information that notifies that an abnormality has occurred based on the comparison result.

4. The information processing method according to claim 3, wherein the log information consists of past operation inputs and environmental device information when the mobile body is in a state of no abnormality.

5. The information processing method according to claim 2, wherein the comparison includes determining that there is an abnormality when there is no difference, and the presentation of the notification information includes presenting the notification information that notifies that an abnormality has occurred based on the comparison result.

6. The information processing method according to claim 5, wherein the log information consists of past operation inputs and environmental device information when the mobile body is in a state of abnormality.

7. The information processing method according to claim 1, wherein the information relating to the operation input and environmental device information further includes setting information of the mobile body set based on the operation input and environmental device information.

8. If the comparison determines that there is an abnormality based on the comparison result, the information processing method according to claim 7, wherein the presentation of notification information includes presenting the notification information based on the comparison result, past operation input and environmental device information included in the log information, and at least one of the setting information as a recommended setting for changing the setting.

9. The information processing method according to claim 8, wherein, if consent is obtained to change at least one of the past operation inputs and environmental device information and the setting information to the recommended settings, the comparison includes changing at least one of the current operation inputs and environmental device information and the current setting information to the recommended settings, and the presentation of the notification information includes presenting information indicating that at least one of the current operation inputs and environmental device information and the current setting information has been changed to the recommended settings.

10. The information processing method according to claim 8, wherein the comparison includes, if an abnormality is determined based on the comparison result, changing at least one of the past operation inputs and environmental device information, and the past setting information, to the recommended setting, and the presentation of the notification information includes presenting information indicating that at least one of the current operation inputs and environmental device information, and the current setting information, has been changed to the recommended setting.

11. The information processing method according to claim 10, wherein if the comparison determines that there is an abnormality based on the comparison result, and the type of the abnormality is of a predetermined type, the method further includes changing at least one of the past operation inputs and environmental device information, and the past setting information, to the recommended setting.

12. The information processing method according to claim 7, wherein the comparison includes performing pattern matching in the log information between the operation input and environmental device information, and the past setting information, and repeating the processing of the matched patterns.

13. The information processing method according to claim 12, wherein the comparison is repeated at the average time interval at which matching occurs in the pattern matching.

14. An information processing device comprising: an operation input receiving unit that receives operation inputs from a mobile device; an environmental body information collection processing unit that collects environmental body information relating to the environment of the mobile device and the body of the mobile device; a log management processing unit that manages the operation inputs and environmental body information as log information; a comparison unit that compares the current operation inputs and environmental body information with the log information; and a notification information presentation unit that presents notification information based on the comparison results of the comparison unit.

15. The information processing apparatus according to claim 14, wherein the comparison unit determines whether there is an abnormality based on whether there is a difference between the current operation input and environmental device information and the log information, and the notification information presentation unit presents the notification information based on the comparison result by the comparison unit.

16. The information processing apparatus according to claim 15, wherein the comparison unit determines that there is an abnormality when there is a difference, and the notification information presentation unit presents notification information indicating that an abnormality has occurred based on the comparison result.

17. The information processing apparatus according to claim 15, wherein the comparison unit determines that there is an abnormality when there is no difference, and the notification information presentation unit presents notification information indicating that an abnormality has occurred based on the comparison result.

18. The information processing device according to claim 14, wherein the information relating to the operation input and environmental device information further includes setting information of the mobile body set based on the operation input and environmental device information.

19. The information processing device according to claim 18, wherein, if the comparison unit determines that there is an abnormality based on the comparison result, the notification information presentation unit presents the notification information based on the comparison result of the comparison unit, past operation input and environmental device information included in the log information, and at least one of the setting information as a recommended setting for changing the setting.

20. The information processing apparatus according to claim 18, wherein the comparison unit performs pattern matching in the log information with respect to the operation input and environmental device information, as well as past setting information, and repeats the processing of the matched pattern.

Images