Information processing method, information processing device, and program

Abstract

The present disclosure relates to an information processing method, an information processing device, and a program that enable a preplanned movement route of a moving body to be appropriately and easily modified according to the situation at a site. A preplanned movement route of the moving body is superimposed and displayed, on the basis of map data, in a real space visible in a user's line-of-sight direction. The present disclosure is applicable to a technique for setting a movement route of a drone.

Description

Information Processing Method, Information Processing Apparatus, and Program 【0001】 The present disclosure relates to an information processing method, an information processing apparatus, and a program, and particularly to an information processing method, an information processing apparatus, and a program that enable an appropriately and easily modified pre-planned movement route of a moving object according to the local situation. 【0002】 Techniques have been proposed for planning in advance a movement route of a moving object such as a drone based on map data and moving the object along the planned movement route. 【0003】 For example, a technique has been proposed for presenting to a user a position visible to an operator remotely operating a moving object based on a pre-planned movement route of the moving object (see Patent Document 1). 【0004】 Japanese Patent Application Laid-Open No. 2024-058542 【0005】 However, in the technique of Patent Document 1, the movement route of the moving object planned in advance is generally planned based on past data such as map data created before the moving object actually moves along the movement route, and the situation at the time when the moving object actually moves along the movement route may not be reflected. 【0006】 For example, at the time when the movement route is planned, planting that did not exist on the movement route has grown and blocked the movement route, and the moving object may not be able to move. Also, the terrain itself may have changed due to landslides caused by rainfall or snowfall after the movement route is planned, and the moving object may not actually be able to move along the planned movement route. 【0007】 The present disclosure has been made in view of such a situation, and particularly enables an appropriately and easily modified pre-planned movement route of a moving object according to the local situation. 【0008】 An information processing method according to one aspect of the present disclosure is an information processing method including route display processing for superimposing and displaying a pre-planned movement route of a moving object based on map data in a real space visible in the user's line-of-sight direction. 【0009】 One aspect of this disclosure is an information processing device and program that includes a route display unit that superimposes and displays a predetermined movement path of a moving object based on map data within the real space visible in the user's line of sight. 【0010】 In one aspect of this disclosure, the movement path of a pre-planned moving object is superimposed on the real space visible in the user's line of sight, based on map data. 【0011】 This is a diagram illustrating the overview of this disclosure. This is a diagram illustrating the overview of this disclosure. This is a diagram illustrating an example configuration of the information processing system of this disclosure. This is a block diagram illustrating an example configuration of the information processing terminal and mobile device in Figure 3. This is a diagram illustrating an example of displaying a movement path superimposed on a camera image. This is a diagram illustrating an example of displaying when modifying the movement path. This is a diagram illustrating other examples of displaying a movement path superimposed on a camera image and an example of displaying when modifying the movement path. This is a diagram illustrating imaging control information that controls the imaging direction and zoom magnification of the camera. This is a diagram illustrating a method for controlling the imaging direction among the imaging control information. This is a diagram illustrating the amount of modification of the imaging control information when the movement path is modified. This is a diagram illustrating an example of controlling the zoom magnification when the movement path is modified. This is a flowchart illustrating the path confirmation process. This is a flowchart illustrating the movement control process. This shows an example of the computer configuration. 【0012】 Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted. 【0013】 The following describes embodiments for implementing the technology of this disclosure. The description will be in the following order: 1. Overview of this disclosure 2. Preferred embodiments 3. Description of a computer to which this technology is applied 【0014】<<1. Summary of this Disclosure>> This disclosure, in particular, enables the appropriate and easy modification of pre-planned movement routes of mobile objects in accordance with local conditions. 【0015】 First, I will explain the overview of the technology described in this disclosure. 【0016】 There is technology that allows flying objects, such as drones, to move safely along pre-planned routes. 【0017】 For example, consider the case where a travel path is planned for a mobile device 12, such as a drone, to fly along a building 11, as shown in the left part of Figure 1. 【0018】 In this case, as shown in the left part of Figure 1, the movement path is represented by the moving body 12 following the sequence of paths R1 to R3, which are indicated by arrows. More specifically, in Figure 1, the movement path involves the moving body 12 moving along path R1 to the left side of the building 11 in the figure, then changing its direction of movement to the right to move along path R2 to move along the front part of the building 11 in the figure, and then changing its direction of movement to the left to move along path R3 to move along the right side of the building 11 in the figure. 【0019】 Such travel routes are generally planned using map data. 【0020】 However, map data is created before the travel route is planned, and may not reflect the conditions at the time the object is actually moving along the route. 【0021】 For example, even if, at the time the map data was created, it was possible to move without any problems along the travel route consisting of paths R1 to R3 shown in the left part of Figure 1, when the moving object 12 later tries to move along the travel route, trees 13 may have grown to block path R2, as shown in the right part of Figure 1, becoming obstacles that hinder the movement of the moving object 12. 【0022】As shown in the right side of Figure 1, when the mobile body 12 attempts to move along a movement path consisting of routes R1 to R3 that have been planned in advance based on map data, the mobile body 12 will be unable to move safely due to interference from the trees 13 when moving along route R2. 【0023】 Therefore, in this disclosure, as shown in the left part of Figure 2, an HMD (Head Mounted Display) type information processing terminal 21 equipped with a display unit 22 and a camera 23 is used to superimpose the planned paths R1 to R3 and corresponding path images VR1 to VR3 onto the image of the real space captured by the camera 23, as shown in image V1 in the upper right part of Figure 2, and display them in AR (Augmented Reality) on the display unit 22. 【0024】 As a result, a user wearing the HMD-type information processing terminal 21 can visually check the current status of the planned travel route R1 to R3 in real time by viewing the image V1 in the upper right of Figure 2, which is displayed on the display unit 22. At this time, the travel route consisting of routes R1 to R3 is superimposed on the image displaying the real space, so the user can check for interference on the travel route without actually moving the mobile body 12 in the real space. 【0025】 In Figure 2, image V1 shows a building image Vo11, a moving object image Vo12, and a tree image Vo13, which are images of a building 11, a moving object 12, and a tree 13, respectively, captured by a camera 23 in the real space in the direction the user wearing the HMD-type information processing terminal 21 is looking. On this image, route images VR1 to VR3, corresponding to routes R1 to R3 on a pre-planned movement path, are superimposed. 【0026】 Furthermore, as shown in image V1 of Figure 2, the path image VR2 is displayed as a dotted line, indicating that interference is occurring on the path R2 due to obstacles such as trees 13. 【0027】As a result, a user wearing the HMD-type information processing terminal 21 can recognize that interference with trees 13 is occurring in the planned travel path R2 simply by looking at the route image VR2, which is displayed as a dotted line. 【0028】 Furthermore, the information processing terminal 21 is equipped with a function to modify the planned travel route. For example, as shown in image V2 in the lower right of Figure 2, it can modify the route image VR2 corresponding to the travel route R2 by marking it with an "X" to indicate that it should be excluded from the travel route. In addition, it can input modified route images VR11 to VR13, shown by the dashed lines in the figure, to avoid the trees 13, thereby easily modifying the travel route. 【0029】 By performing this simple modification operation of the travel route, it becomes possible for the user to easily plan modified routes R11 to R13 (none of which are shown) corresponding to modified route images VR11 to VR13, in place of route R2 among the routes R1 to R3 that constitute the travel route planned in advance using map data, through intuitive operation. 【0030】 This allows for the appropriate and easy modification of travel routes to reflect local conditions that are not captured in map data. Furthermore, because travel routes can be modified according to local conditions, positional discrepancies that may occur in map data can be visually confirmed and corrected appropriately and easily using the same method. 【0031】 Furthermore, since the majority of the travel route can be planned using map data, the entire travel route can be efficiently generated using map data, and only problematic sections along the travel route that arise according to local conditions need to be corrected. This makes it possible to appropriately and easily modify the travel route according to local conditions. 【0032】 As a result, the movement path of the mobile object can be appropriately and easily modified according to the local conditions. 【0033】<<2. Preferred Embodiments>> Next, with reference to Figure 3, an example configuration of an information processing system that enables confirmation and correction of the movement path of a moving object based on map data will be described. 【0034】 The information processing system 101, which enables confirmation and correction of the movement path of a mobile object based on the map data in Figure 3, consists of a camera 121, an HMD-type information processing terminal 111 equipped with a display unit 122, and a mobile object 112 such as a drone. 【0035】 The information processing terminal 111 is worn on the user's head and, as shown in the upper right of Figure 3, displays a superimposed image of the planned movement path of the mobile object 12 on an image of the real space captured by the camera 121, indicated by arrows and the like in the figure. 【0036】 In the right-hand portion of Figure 3, the dotted line indicates interference with trees. Furthermore, the movement path image may include warnings when other warnings are needed along the path. Examples of other warning displays will be described in detail later, referring to Figure 5. 【0037】 Furthermore, the information processing terminal 111 can modify the travel path by editing the AR-supervised travel path image, as shown in the upper left of Figure 3, through user operation. 【0038】 In the upper left of Figure 3, paths that interfere with the trees, indicated by the dotted arrows in the upper right of Figure 3, are marked with an "X". Furthermore, the images of movement paths that avoid the trees are edited to be indicated by dashed-dotted arrows, indicating that the movement paths have been modified. 【0039】 Furthermore, details regarding the modification of the travel path resulting from the editing of the travel path image will be described later, referring to Figure 6. 【0040】 As shown in the upper left of Figure 3, the information processing terminal 111 supplies the mobile body 112 with information on the pre-planned travel route, along with information on the modified route that has been edited. 【0041】That is, in the case of the upper left part of FIG. 3, the information processing terminal 111 transmits to the moving body 112 information combining the information on the movement route planned based on the pre-map data indicated by the solid-line arrow and the information on the corrected route generated by correcting the movement route by editing indicated by the dashed-line arrow. 【0042】 In FIG. 3, a configuration example of an HMD type is shown for the information processing terminal 111. However, as long as it can view the real space and can superimpose and display a movement route image corresponding to the movement route, other configurations may be used. For example, AR glasses, VR goggles, camera see-through terminals, etc. may be used. However, in this example, the information processing terminal 111 will be described assuming a configuration example of the HMD type. 【0043】 In the case of a see-through terminal, since a see-through type display device is provided, when the user wears the information processing terminal 111, the user can directly visually recognize the real space through the see-through type display device and visually recognize the movement route superimposed on the corresponding position in the directly visually recognized real space of the see-through type display device. 【0044】 When the moving body 112 acquires the information on the movement route based on the map data and the information on the corrected route from the information processing terminal 111, while confirming its own current position and orientation, it basically moves along the movement route planned based on the map data, and when a corrected route is given at the current position, it moves along the corrected route. 【0045】 In addition, the moving body 112 includes a camera 281 (FIG. 4) that images a predetermined object while moving along the movement route (and the corrected route). The camera 281 is configured to be able to change the imaging direction and zoom ratio for imaging, and the moving body 112 moves along the movement route while imaging so as to maintain the state where the object is within a predetermined angle of view by controlling the imaging direction and zoom ratio of the camera 281. 【0046】Regarding the control of the imaging direction and zoom ratio of the camera 281, it is planned at the stage when the movement path is planned, and basically, the plan is made to maintain the state where the object is captured at a predetermined angle of view. Basically, the moving body 112 moves while controlling the camera 281 to capture images at the imaging direction and zoom ratio specified for capturing the object given in the movement path information. 【0047】 When the moving body 112 moves along the corrected path, it moves while controlling the imaging direction and zoom ratio so that the object imaged by the camera 281 planned for the movement path is of a predetermined size and fits within a predetermined angle of view. 【0048】 Regarding the control of the imaging direction and zoom ratio for the camera 281 to image the object so that it is of a predetermined size and fits within a predetermined angle of view, details will be described later with reference to FIGS. 8 to 10. 【0049】 <Example of the Configuration of the Information Processing Terminal and the Moving Body> Next, with reference to FIG. 4, an example of the configuration of the information processing terminal 111 and the moving body 112 will be described. 【0050】 The information processing terminal 111 includes a control unit 151, a sensor unit 152, a position and orientation estimation unit 153, an image generation unit 154, a display unit 155, an input unit 156, a storage unit 157, and a communication unit 158. 【0051】 The control unit 151 is composed of a processor, a memory, etc., and controls the overall operation of the information processing terminal 111. The control unit 151 also includes a path interpretation unit 161 and a path correction unit 162. 【0052】 The path interpretation unit 161 reads out the path data 201 indicating the movement path planned in advance based on the map data stored in the storage unit 157 and the path correction data 202 indicating the corrected movement path by operating the input unit 156 by the user, interprets the movement path in the world coordinate system, and supplies the information of the movement path in the world coordinate system as the interpretation result to the image generation unit 154. 【0053】Based on the travel path information supplied by the travel path interpretation unit 161, the route correction unit 162 receives the correction details when the travel path displayed in real space by the image generation unit 154 is corrected by the operation of the input unit 156, and stores the corrected travel path information as route correction data 202 in the storage unit 157. 【0054】 The sensor unit 152 senses information for detecting the position and orientation of the HMD-type information processing terminal 111 and supplies it to the position and orientation estimation unit 153. More specifically, the sensor unit 152 includes a GPS 171 and an IMU 172. 【0055】 The GPS (Global Positioning System) 171 receives signals from satellites (not shown), determines the position on Earth as latitude and longitude based on the received signals, and outputs them to the position and attitude estimation unit 153. 【0056】 The IMU (Inertial Measurement Unit) 172 detects three-dimensional acceleration and angular velocity and outputs them to the position and attitude estimation unit 153 as information for determining the attitude. 【0057】 The position and orientation estimation unit 153 estimates the position and orientation of the information processing terminal 111 from the position information from the GPS 171 and the three-dimensional acceleration and angular velocity information supplied from the IMU 172, and supplies the estimated position and orientation information to the image generation unit 154. 【0058】 The image generation unit 154 generates an image that superimposes the movement path and the modified movement path onto an image of the real space captured by the camera 192, based on the movement path information supplied by the path interpretation unit 161 of the control unit 151, the modified path information supplied by the path correction unit 162 that has been added to the movement path, and the position and orientation estimation results of the information processing terminal 111 supplied by the position and orientation estimation unit 153, and outputs it to the display unit 155 for display. 【0059】 More specifically, the image generation unit 154 includes a path display unit 181 and an interference determination unit 182. 【0060】The route display unit 181 converts the travel route information and corrected travel route information in the world coordinate system into coordinate system information in the real space visible from the user's viewpoint, which is captured by the camera 192 and displayed on the display unit 155, based on the travel route information supplied by the route interpretation unit 161, the corrected route information added to the travel route supplied by the route correction unit 162, and the position and orientation estimation results of the information processing terminal 111 supplied by the position and orientation estimation unit 153, and outputs it to the display unit 155 for superimposed display. 【0061】 The interference determination unit 182 recognizes objects in the captured image space and, based on the information of the movement path in the world coordinate system and the information of the modified movement path, determines the position where interference (collision) occurs between the moving body 112 and the object when the object moves along the movement path and the modified movement path, and whether there is a position where the distance to the object is less than or equal to a predetermined distance, and supplies the determination result to the path display unit 181. 【0062】 As a result, if the route display unit 181 determines that there is a position where interference (collision) will occur, it will display that position on the travel path with a different color or line than other travel paths where interference (collision) will not occur. 【0063】 This allows users to recognize the location of any potential object interference simply by looking at the displayed movement path. 【0064】 Furthermore, the route display unit 181 determines whether there is a position where the distance to the object is determined to be less than or equal to a predetermined distance. If there is a position where the distance to the object is determined to be less than or equal to a predetermined distance, the distance from the travel path to the object is displayed at the corresponding position on the travel path (or in its vicinity). 【0065】 This allows users to understand the distance between the displayed movement path and the object. For example, if strong winds are expected due to the weather, they may adjust their movement path to stay at least a certain distance away from the object, as the wind could disrupt their path. Conversely, if there is no wind, they can leave the movement path as is, as the wind will not disrupt their path. 【0066】 The display unit 155 consists of an organic EL (Organic Electro Luminescence) or an LCD (Liquid Crystal Display), and displays images generated by the image generation unit 154. More specifically, as shown in Figure 3, the display unit 155 is positioned in front of the user's eyes and displays images of the real world captured by a camera 192 positioned close to the user's line of sight. This allows the user to perceive the external world in their line of sight even while wearing the HMD-type information processing terminal 111. The display unit 155 may also be a see-through display, in which case the user can directly view the real world while only the movement path is superimposed and displayed for visibility. 【0067】 The input unit 156 receives operation input from the user and supplies the operation signal corresponding to the received operation input to the control unit 151. 【0068】 More specifically, the input unit 156 includes a microphone 191, a camera 192, and a touch panel 193. 【0069】 When the microphone 191 receives voice input from the user, it receives the recognition result of the voice recognition process as the user's operation input and supplies the operation signal corresponding to the received operation input to the control unit 151. 【0070】 Camera 192 is composed of an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor and an optical block including a focusing lens and a zoom lens. It is controlled by the control unit 151 to capture images of the real space visible from the user's viewpoint and supply the captured images to the image generation unit 154. Camera 192 also captures gestures such as hand signs that correspond to user input and receives the image recognition results based on the captured images as input and supplies them to the control unit 151. 【0071】The touch panel 193 consists of a touch-type operation panel that receives user input and supplies operation signals corresponding to the input to the control unit 151. 【0072】 The storage unit 157 is composed of an HDD (Hard Disc Drive), an SSD (Solid State Drive), or semiconductor memory, and is controlled by the control unit 151 to write and read various types of data. More specifically, the storage unit 157 stores or reads route data 201, which consists of information on travel routes planned in advance based on map data, and route correction data 202, which consists of information on travel routes corrected by the user operating the input unit 156 to correct the travel route displayed on the display unit 155 based on the route data 201. 【0073】 The communication unit 158 ​​is controlled by the control unit 151 and transmits and receives various data and programs to and from the mobile device 112 using the internet or a communication line such as Bluetooth (registered trademark). More specifically, the communication unit 158 ​​is controlled by the control unit 151 and transmits route data 201 and route correction data 202 stored in the storage unit 157 to the mobile device 112. 【0074】 The mobile device 112 moves from the starting point to the destination based on the route data 201 and route correction data 202 transmitted from the information processing terminal 111, and also captures images of predetermined objects during its movement, records them as images, or transmits the images to other devices. 【0075】 In this embodiment, the description will proceed on the premise that the mobile body 112 is a flying object such as a drone. However, it is not limited to flying objects as long as it is a moving object; for example, it may be a vehicle, walking robot, ship (including surface boats and submarines), aircraft, rocket, or spacecraft. 【0076】 The mobile unit 112 includes a control unit 251, a sensor unit 252, a position and attitude estimation unit 253, a payload control unit 254, a movement control unit 255, a storage unit 256, and a communication unit 257. 【0077】The control unit 251 consists of a processor, memory, and other components, and controls the overall operation of the mobile body 112. The control unit 251 also includes a path interpretation unit 261 and a correction amount recognition unit 262. 【0078】 The path interpretation unit 261 reads the path data 201 and path correction data 202 supplied from the information processing terminal 111, which are stored in the memory unit 256, and interprets the movement path in the world coordinate system. Based on the position and orientation of the moving body 112 itself supplied by the position and orientation estimation unit 253, it controls the movement control unit 255, which controls the movement of the moving body 112, to move the moving body 112 along the movement path. 【0079】 The movement control unit 255 controls the movement of various drive mechanisms for moving the mobile body 112. For example, if the mobile body 112 is a drone that flies by rotating multiple propellers, the movement control unit 255 controls the movement of the mobile body 112 by controlling the direction and speed of movement, for example, by individually controlling the rotational speed of the motors that drive the multiple propellers and the yaw, roll, and pitch of the propellers. 【0080】 The correction amount recognition unit 262 reads the route data 201 and the route correction data 202, recognizes the correction amount of the information contained in the route data 201 that controls the imaging direction and zoom magnification of the camera 281 related to imaging of an object using the camera 281 as it moves, and controls the payload control unit 254 to control the imaging direction and zoom magnification of the camera 281. 【0081】 More specifically, the route data 201 includes, in addition to information relating to the travel route from the departure point to the destination, based on map data, information specifying the imaging direction and zoom magnification for controlling the imaging of a predetermined object by the camera 281 when traveling along the route. Hereafter, this information specifying the imaging direction and zoom magnification of the camera 281 will also be referred to as imaging control information. 【0082】When moving along a route based on pre-planned route data 201 associated with map data, the correction amount recognition unit 262 controls the payload control unit 254 using the imaging control information included in the original route data 201, without determining the correction amount, to control the imaging direction and zoom magnification of the camera 281. 【0083】 On the other hand, when moving along a corrected movement path based on the path correction data 202, the correction amount recognition unit 262 determines the amount of correction of the imaging control information required when moving along the corrected movement path based on the imaging control information contained in the original path data 201, and controls the payload control unit 254 with the correction amount to control the imaging direction and zoom magnification of the camera 281. 【0084】 In other words, the route data 201, which corresponds to a pre-planned travel route based on map data, is a collection of route data for each section. For example, in the case of the travel route shown in the upper right of Figure 2, the travel route image consists of route images VR1 to VR3, and the route data 201 consists of multiple section data consisting of information on the starting point and destination position for each section of the routes R1 to R3, which correspond to route images VR1 to VR3. In addition, each section data of the routes R1 to R3 is accompanied by individual imaging control information for controlling the imaging direction and zoom magnification of the camera 281 when moving through that section. 【0085】 In contrast, the route correction data 202 consists of multiple section data consisting of corrected routes R11 to R13 (not shown), which correspond to corrected route images VR11 to VR13, and are replaced with section data of route R2, which corresponds to route image RV2 that constitutes the route data 201. However, since corrected routes R11 to R13 only contain information on the corrected portion of the movement route, imaging control information is not pre-assigned to each of them. 【0086】Therefore, the correction amount recognition unit 262 estimates the position of the object from the position of path R2 corresponding to the path image RV2 and the imaging direction and zoom magnification in the corresponding imaging control information. Based on the estimated position of the object, it calculates the imaging direction and zoom magnification as the correction amount from the section information of the corrected paths R11 to R13 that replace path R2, so as to maintain imaging at the angle of view captured when the object is imaged while moving along path R2. 【0087】 When moving along a movement path based on the path data 201, the payload control unit 254 controls the zoom magnification of the camera 281 and the movement of the gimbal 282 to control the imaging direction of the camera 281, based on the imaging control information contained in the original path data 201 supplied by the correction amount recognition unit 262. 【0088】 When the payload control unit 254 moves along a corrected movement path based on the path correction data 202, the correction amount recognition unit 262 controls the zoom magnification of the camera 281 and the movement of the gimbal 282 to control the imaging direction of the camera 281 based on the correction amount calculated from the path correction data 202. 【0089】 Hereafter, the control of the zoom magnification of the camera 281 by the payload control unit 254 will also be referred to as zoom control, and the control of the imaging direction of the camera 281 by controlling the movement of the gimbal 282 in the payload control unit 254 will also be referred to as gimbal control. Furthermore, in gimbal control, the payload control unit 254 may, if necessary, further utilize attitude information supplied from the position and attitude estimation unit 253, which will be described later. 【0090】 The sensor unit 252 senses position information and attitude information for detecting the position and attitude of the moving object 112 and supplies it to the position and attitude estimation unit 253. More specifically, the sensor unit 252 includes a GPS 271 and an IMU 272. Note that the GPS 271 and IMU 272 are basically the same as the GPS 171 and IMU 172 in the information processing terminal 111, so their explanation is omitted. 【0091】The position and attitude estimation unit 253 estimates the position and attitude of the moving body 112 from the position information from the GPS 271 and the three-dimensional acceleration and angular velocity information supplied from the IMU 272, and supplies the estimated position and attitude information to the control unit 251. 【0092】 The storage unit 256 is composed of an HDD, SSD, or semiconductor memory, and is controlled by the control unit 251 to write and read various types of data. More specifically, the storage unit 256 stores or reads route data 201 and route correction data 202 supplied from the information processing terminal 111. 【0093】 The communication unit 257 is controlled by the control unit 251 and sends and receives various data and programs to and from the information processing terminal 111 using the internet or communication lines such as Bluetooth (registered trademark). More specifically, the communication unit 257 is controlled by the control unit 251 and receives route data 201 and route correction data 202 supplied from the information processing terminal 111 and stores them in the storage unit 256. 【0094】 <Example of route presentation> Next, with reference to Figure 5, an example of a presentation shown by the information processing terminal 111 based on route data 201 and route correction data 202 will be explained. 【0095】 Image V31 in Figure 5 shows an example of what is displayed on the display unit 155 of the information processing terminal 111 based on the route data 201. 【0096】 Image V31 presents the movement path of the mobile device 112 as it moves through the plant facility and takes images of a predetermined object, as a movement path image. More specifically, Image V31 is viewed in an environment where a user, while wearing an information processing terminal 111, can directly view the plant facility where the movement path of the mobile device 112 is set. 【0097】In other words, image V31 displays an image of the plant equipment captured by camera 192 (121) when the user, wearing the information processing terminal 111, looks towards the plant equipment. In addition, in image V31, the movement path of the mobile body 112, which was planned in advance based on map data in route data 201, is presented as a movement path image consisting of route images VR201 to VR204. 【0098】 Here, route image VR201 is presented as a route moving between locations WP1 and WP2 in the plant facility. Route image VR202 is presented as a route moving between locations WP2 and WP3 in the plant facility. Route image VR203 is presented as a route moving between locations WP3 and WP4 in the plant facility. Route image VR204 is presented as a route moving between locations WP4 and WP5 in the plant facility. 【0099】 In other words, in image V31, path images VR201 to VR204, which represent the movement path from positions WP1 to WP5, are superimposed on an image of the plant equipment as real space in the same direction as the user's line of sight, which is captured by camera 192. 【0100】 In this way, the route images VR201 to VR204 are superimposed on the image of the plant facilities as real space captured by the camera 192, allowing the user to visually confirm the movement route in the actual real space without actually moving the mobile object 112. 【0101】 Furthermore, in image V31, it is indicated that the distance between the movement path represented by the path image VR202 and the pipe-shaped object Vo201 is "3m". This indicates that the distance between the movement path represented by the path image VR202 and the pipe-shaped object Vo201 is 3m, and that there is a possibility of interference with the object as the moving body 112 actually moves. 【0102】In other words, as shown in Figure 5, when the distance between the movement path represented by the path image VR202 and the pipe-shaped object Vo201 is shorter than a predetermined distance, and interference may occur if the movement position becomes unstable due to strong winds or other reasons during movement, the distance between the movement path and the object is displayed. 【0103】 This allows the user to visually see and be alerted that, when the moving object 112 is actually moved along the planned movement path in the current route data 201, the distance to the object Vo 201 in the movement path represented by the route image VR 202 is close to a predetermined distance, and there is a risk of interference depending on the situation during movement. The distance between the movement path and the object may be displayed as a specific numerical value as shown in Figure 5, or the color and type of the route image (dashed line, dotted line, dashed line, etc.) may be switched according to the distance. 【0104】 Furthermore, in image V31, a portion of the path image VR203 is shown as a dashed line. This indicates that the movement path represented by the path image VR203 is interfering with the pipe-shaped object Vo201. 【0105】 In other words, when the moving object 112 is actually moved along the planned movement path in the current route data 201, it becomes possible to make the user visually aware of the interference with the object Vo 201 in the movement path represented by the dashed line in the route image VR 203, thereby drawing their attention to the issue. Note that the method for indicating interference in the route image does not have to be limited to dashed lines; other lines may be used, or the interference may be represented by changing the color. Furthermore, the characters indicating distance may also be displayed in a more conspicuous color. 【0106】 <Examples of modifying movement paths> Next, we will explain examples of modifying movement paths with reference to Figure 6. Figure 6 shows examples of modifying both VR202, a path image in Figure 5 that indicates a distance of 3m from the object and shows a potential for interference, and VR203, a path image that indicates interference. 【0107】In other words, in Figure 6, the route image VR202 is modified by the input unit 156 to become the route image VR202' represented by a dotted line, and the route image VR203 is modified to become the route image VR203' represented by a dotted line. 【0108】 More specifically, the dotted line path image VR202' is a modified version of path image VR202, where positions WP2 and WP3 are corrected to positions WP2' and WP3', and is modified to represent a path moving between positions WP2' and WP3' within the plant facilities. 【0109】 As a result, the path image VR202' is modified to a position at a predetermined distance of 4m or more from the pipe-shaped object Vo201, so that even when the moving body 112 is moved, the possibility of interference with the object Vo201 is reduced. 【0110】 Furthermore, the route image VR203', represented by a dotted line, is a modified version of route image VR203, where positions WP3 and WP4 are corrected to positions WP3' and WP4', and is modified to represent a route moving between positions WP3' and WP4' within the plant facilities. 【0111】 As a result, the path image VR203' is corrected to a position where it does not interfere with the pipe-shaped object Vo201, and even when the moving body 112 is moved, there is no interference with the object Vo201. 【0112】 Furthermore, the route data 201, which is generated in advance based on map data, consists of, for example, individual starting and ending points representing route images VR201 to VR204. In the case of Figure 5, it consists of a first section data corresponding to route image VR201 moving from position WP1 to WP2 in the plant facility, a second section data corresponding to route image VR202 moving from position WP2 to WP3, a third section data corresponding to route image VR203 moving from position WP3 to WP4, and a fourth section data corresponding to route image VR204 moving from position WP4 to WP5. 【0113】When modifications are made, route modification data 202 is generated according to the modifications. In the case of Figure 6, it consists of a first modification section data corresponding to the route image VR201' (not shown in reference numerals) that moves from position WP1 to WP2' in the plant equipment, a second modification section data corresponding to the route image VR202' that moves from position WP2' to WP3', a third modification section data corresponding to the route image VR203' that moves from position WP3' to WP4', and a fourth modification section data corresponding to the route image VR204' (not shown in reference numerals) that moves from position WP4' to WP5. 【0114】 In the case of Figure 6, in the movement path between position WP1 and WP5, the positions WP2 to WP4 that are passed through are modified to positions WP2' to WP4'. Therefore, when the moving object 112 moves based on the path data 201 and the path modification data 202, the newly generated path modification data 202 is used to replace the path data 201. 【0115】 Furthermore, for example, if there is a 0th section data (not shown) prior to position WP1 and a 5th section data (not shown) after position WP5, and there are no modifications, the moving body 112 reads and uses the 0th section data prior to position WP1 and the 5th section data after position WP5 from the route data 201, and reads and uses the route modification data 202 for the movement path between position WP1 and WP5 via positions WP2' to WP4'. 【0116】 In the explanation of how the movement path can be corrected by the user operating the touch panel 193 of the input unit 156, the movement path can also be corrected by voice input using the microphone 191, for example, by performing voice recognition when the user makes a statement such as, "Please correct the movement path so that it is more than 4m away" or "Please correct the movement path to a position where there is no interference." Alternatively, the camera 192 can be used to capture the user's hand movements as hand signs instructing the correction of the movement path, and the movement path can be corrected by gesture recognition or the like. 【0117】<Variations of Presentation and Modification of Travel Routes> In addition, the above has described an example in which a travel route is set assuming movement from a starting point to a destination. However, in the case of inspecting an object such as a steel tower, for example, as shown in image V51 of Figure 7, the travel route may be displayed as a route that circles the object Vo221, which consists of a steel tower, in a cylindrical shape while changing its height. In the case of image V51 of Figure 7, it is shown that the travel route is set to circle the object Vo221, which consists of a steel tower, by switching between five different heights. 【0118】 In this case, for example, as shown in image V51, the movement path at the central height in the height direction is shown to be 3m away from the object Vo221, which consists of a steel tower. In other words, it is shown that there is a possibility of interference with the object Vo221 in the three lower movement paths in the height direction of the object Vo221, which consists of a steel tower. 【0119】 In such cases, the modification of the movement path may be, for example, modified to widen the radius of rotation from the center of the object Vo221, which consists of a steel tower, as shown in image V61. 【0120】 Furthermore, the movement path may be modified, for example, to limit the movement path to only the upper two stages in the height direction, where the distance from the center of the object Vo221, which consists of a steel tower, is secured, as shown in image V62. 【0121】 In this case, the modifications may also be made so that the user can operate the touch panel 193 of the input unit 156 to make modifications such as increasing the radius of the movement path or deleting a part of the movement path. 【0122】 Furthermore, the system may use microphone 191 for voice input, enabling voice recognition based on utterances such as, "Please increase the turning radius so that it is more than 4 meters away," or "Please modify the movement path to include only non-interfering routes," thereby correcting the movement path. Alternatively, camera 192 may capture the user's hand movements as hand signs, allowing the movement path to be corrected through gesture recognition or similar methods. 【0123】 <Imaging control method based on imaging control information> Next, with reference to Figures 8 to 10, an imaging control method for the imaging direction and zoom magnification of the camera 281 of the mobile body 112 based on imaging control information included in the path data 201 will be described. 【0124】 The mobile body 112 moves along the route identified based on the route data 201 and route correction data 202 described above, and also images the set target object. 【0125】 The camera 281 mounted on the mobile body 112 has its zoom magnification and imaging direction controlled by the payload control unit 254. 【0126】 More specifically, the camera 281 is equipped with a zoom lens with variable zoom magnification (not shown), and the payload control unit 254 controls the zoom magnification based on the imaging control information and the correction amount. The gimbal 282 is also controlled by the payload control unit 254 to correct camera shake and change the imaging direction. 【0127】 With this configuration, the payload control unit 254 controls the imaging direction and zoom magnification of the camera 281 based on the imaging control information included in the route data 201. 【0128】 More specifically, consider the case where, as shown in Figure 8, there are two objects Vo251 and Vo252 consisting of buildings, a movement path consisting of routes R301 to R304 is set, and the moving body 112 moves along the set movement path to image the target objects VP1 to VP3 relative to the objects Vo251 and Vo252. 【0129】 In this case, the mobile body 112 moves along a movement path consisting of paths R301 to R304, as shown in Figure 9, for example, while changing the imaging direction of the camera 281 over the imaging targets VP1 to VP3 and capturing images. 【0130】 In other words, Figure 9 is a layout diagram of the objects Vo251 and Vo252, which consist of buildings in Figure 8, as seen from above, and shows the relationship between the top surfaces of objects Vo251 and Vo252 and the movement paths consisting of paths R301 to R304. 【0131】 Furthermore, Figure 9 shows cones AOV1 to AOV15 that represent the imaging direction of the camera 281 of the moving body 112 at each position along the paths R301 to R304. In Figure 9, the cones AOV1 to AOV15, which are isosceles triangles that represent the imaging direction, have vertices where the two equal sides intersect that represent the position of the moving body 112 on its movement path, and the direction of a straight line (not shown) perpendicular to the base, starting from the vertex where the two equal sides intersect, represents the imaging direction of the camera 281. 【0132】 In other words, while the moving body 112 moves along the path R301 to R302, the cones AOV1 to AOV4 are positioned with the imaging direction facing upwards in the figure, and the target object VP1 is continuously imaged. 【0133】 When the movement path of the moving body 112 changes from path R301 to R302 to path R303, the direction of movement changes to the vertical direction in the figure. As a result, the imaging direction gradually changes to the left in the figure, resulting in cones AOV5 to AOV9, and the target VP2 continues to be imaged. 【0134】 Furthermore, as the movement path of the mobile body 112 passes through path R303 and proceeds to path R304, the imaging target VP2 of object Vo251 is changed to the imaging target VP3 of object Vo252. As shown by cones AOV10 to AOV12, the imaging direction changes from the left in the figure through upwards to the right, and the imaging target VP3 is imaged. Then, as the mobile body 112 proceeds to the upper end of path R304, the area further to the right in the figure is imaged, as shown by cones AOV13 to AOV15. 【0135】 In Figure 9, only the imaging direction is represented by cones AOV1 to AOV15. The zoom magnification is set according to the distance from the vertex where the two equal sides intersect to the imaging targets VP1 to VP3. In principle, the zoom magnification is set so that in all cases the size of the individual subjects in imaging targets VP1 to VP3 fits within a field of view that is as uniform as possible. 【0136】<Method for Recognizing the Amount of Correction to Imaging Control Information Based on Path Correction Data> Next, with reference to Figure 10, we will explain how to recognize the amount of correction to imaging control information based on path correction data. 【0137】 For example, if we assume that the travel route consisting of routes R301 to R304, as shown in Figure 9, is a route based on route data 201, then consider the case where route R303 is modified to the modified routes R311 to R313, as shown in Figure 10. 【0138】 In this case, in the correction path R311 to R313, the imaging direction of the camera 281 needs to be corrected as shown by the cones AOV31 to AOV43. 【0139】 Therefore, the correction amount recognition unit 262 determines, based on the path correction data 202, the correction amount in the movement path, which corresponds to the correction amount in the imaging direction of cones AOV 31 to AOV 43 relative to the imaging targets VP2 and VP3 in the path R311 to R313, and the correction amount of the zoom magnification based on the distance to the imaging targets VP2 and VP3. 【0140】 More specifically, the correction amount recognition unit 262 estimates the positions of the imaging targets VP2 and VP3 from the imaging control information in the pre-correction paths R303 and R304. Furthermore, based on the estimated position of the imaging target VP2, the correction amount recognition unit 262 recognizes the imaging direction and zoom magnification as correction amounts from the direction and distance to the imaging target VP2 at each position as the mobile body 112 moves along the corrected paths R311 to R312. Similarly, based on the estimated position of the imaging target VP3, the correction amount recognition unit 262 recognizes the imaging direction and zoom magnification as correction amounts from the direction and distance to the imaging target VP3 at each position as the mobile body 112 moves along the corrected path R313. 【0141】 Then, when movement is performed based on the path correction data 202, the correction amount recognition unit 262 supplies the imaging direction and zoom magnification recognized as correction amounts to the payload control unit 254, instead of the imaging control information of the path data 201. Based on the correction amount, the payload control unit 254 implements zoom control and gimbal control of the camera 281, and controls the imaging direction and zoom magnification. 【0142】 <Adjusting the field of view using zoom magnification> The above explanation has basically described an example of determining the correction amount for the imaging direction and zoom magnification, assuming that the camera 281 and the object to be imaged are facing each other directly. However, if the camera 281 and the object to be imaged are not facing each other directly, distortion of the field of view may occur, and it may not be possible to properly image the object that should be imaged. 【0143】 In this case, the zoom magnification can be adjusted so that the field of view that should be captured when facing the subject directly is included, thereby ensuring that the object that should be captured is properly imaged. 【0144】 In other words, for example, as shown in the upper left of Figure 11, by imaging with the moving body 112 directly facing the imaging target Vo271, an AOV-R field of view as shown in the lower left is captured. 【0145】 However, in reality, as shown in the upper right of Figure 11, if there are plants such as weeds present, in order to avoid interference with the plants, the mobile body 112 moves to a position higher than the height at which the image target Vo271 is located, and from the position shown by the mobile body 112', the gimbal 282 adjusts the imaging direction of the camera 281 so that the image target Vo271 is viewed at an angle downwards. 【0146】 In this case, since the camera 281 is not directly facing the image target Vo 271, the field of view is distorted into a trapezoidal shape, as shown by the field of view AOV-S in the lower right of Figure 11. 【0147】 Therefore, in such cases, the correction amount recognition unit 262 adjusts to slightly reduce the zoom magnification so that the field of view AOV-SZ becomes the field of view AOV-SZ that includes the field of view AOV-R that should originally be captured. 【0148】 In the example shown in Figure 11, the gimbal 282 is used to correct the tilt direction of the camera 281's imaging direction, but similarly, if distortion occurs in the field of view in the roll or pitch direction, the zoom magnification is adjusted so that the field of view AOV-SZ includes the field of view AOV-R that should be captured. 【0149】Furthermore, while the above describes how to adjust when the positional relationship between the camera 281 and the object being imaged changes due to a modification of the movement path, for example, if there is an overall positional shift in the movement path based on the route data 201 that was planned in advance from map data, the movement path will be modified by applying an offset corresponding to the overall positional shift. In such cases, since the positional relationship between the camera 281 and the object being imaged is appropriately corrected by the modification of the path, distortion of the field of view will not occur, and adjustment using zoom magnification will not be necessary. 【0150】 <Route Confirmation Process> Next, the route confirmation process by the information processing terminal 111 will be explained with reference to the flowchart in Figure 12. 【0151】 In step S31, the route interpretation unit 161 reads route data 201 from the storage unit 157, interprets the travel route based on the route data 201, and supplies the interpreted travel route information to the image generation unit 154. 【0152】 In step S32, the route display unit 181 of the image generation unit 154 acquires position and orientation information of the information processing terminal 111 itself from the position and orientation estimation unit 153. More specifically, the position and orientation estimation unit 153 acquires position information from the GPS 171 of the sensor unit 152, and also acquires angular velocity and acceleration information from the IMU 172 as orientation information, and supplies it to the route display unit 181 of the image generation unit 154. Accordingly, the route display unit 181 acquires its own position information and orientation information supplied from the position and orientation estimation unit 153 in this manner. 【0153】 In step S33, the route display unit 181 converts the coordinate system of the movement path based on the route data 201 to the coordinate system of the image (camera image) supplied by the camera 192 based on the acquired position and orientation information, and displays the movement path superimposed on the image captured by the camera 192 on the display unit 155. Through this process, an image like the example V31 described with reference to Figure 5 is displayed on the display unit 155. 【0154】In step S34, the interference detection unit 182 recognizes an object from the image (camera image) supplied by the camera 192. 【0155】 In step S35, the interference determination unit 182 determines whether or not the movement path interferes with the recognized object. 【0156】 If it is determined in step S35 that the movement path interferes with a recognized object, the process proceeds to step S36. 【0157】 In step S36, the interference determination unit 182 notifies the path display unit 181 of the location of the movement path that interferes with the object. In response to this notification, the path display unit 181 changes the display of the movement path at the location that interferes with the object among the movement path displays superimposed on the camera image. 【0158】 In other words, as explained with reference to Figure 5, for example, non-interfering travel paths are shown with solid lines, while interfering travel paths are shown with dashed lines. 【0159】 Furthermore, if it is determined in step S35 that the movement path does not interfere with the recognized object, the process in step S36 is skipped. 【0160】 In step S37, the interference determination unit 182 determines whether or not there is a point on the movement path where the distance to a recognized object is less than or equal to a predetermined distance. 【0161】 If, in step S37, it is determined that there is a point on the movement path where the distance to the recognized object is less than or equal to a predetermined distance, the process proceeds to step S38. 【0162】 In step S38, the interference determination unit 182 notifies the path display unit 181 of the location on the movement path where the distance to the recognized object is less than or equal to a predetermined distance. In response, the path display unit 181 displays distance information near the location on the movement path where the distance to the recognized object is less than or equal to a predetermined distance. 【0163】In other words, as explained with reference to Figure 5, for example, specific distance information such as "3m" is displayed near movement paths that are less than or equal to a predetermined distance from an object. 【0164】 Furthermore, if it is determined in step S37 that there are no locations on the movement path where the distance to the recognized object is less than or equal to a predetermined distance, the process in step S38 is skipped. 【0165】 In step S39, the route correction unit 162 determines whether the input unit 156 has been operated by the user to correct the travel route. The travel route correction operation performed by the input unit 156 can be any of the following: a travel route correction instruction made by the user speaking and being picked up by the microphone 191 and recognized by voice recognition; a travel route correction instruction made by the user's gestures captured by the camera 192; or a travel route correction instruction made by operating the touch panel 193. 【0166】 If, in step S39, it is determined that the input unit 156 has been operated and a modification operation of the movement path has been performed, the process proceeds to step S40. 【0167】 In step S40, the route correction unit 162 supplies information indicating a change in the travel route, corresponding to the travel route correction operation performed by the user on the input unit 156, to the route display unit 181. When the route display unit 181 receives the information indicating a change in the travel route, it displays a preview of the travel route that has been changed according to the travel route correction operation on the display unit 155. The preview displayed here is, for example, the image V31 described with reference to Figure 6. 【0168】 In step S41, the route correction unit 162 determines whether or not there is an instruction to apply the correction operation performed by the user through the input unit 156. 【0169】 If, in step S41, it is determined that there is an instruction to apply a modification operation performed by the user via the input unit 156, the process proceeds to step S42. 【0170】In step S42, the route correction unit 162 generates route correction data 202 according to the correction operation performed by the user through the input unit 156, and stores it in the storage unit 157. 【0171】 Furthermore, if it is determined in step S41 that no instruction to apply a correction operation has been given, the process in step S42 is skipped. 【0172】 In step S43, the route correction unit 162 determines whether the input unit 156 has been operated by the user and whether the user has instructed the user to complete the correction of the travel route. 【0173】 In step S43, if the input unit 156 is not operated by the user to indicate the end of the movement path modification, the process returns to step S35, and the subsequent processes are repeated. 【0174】 If no correction operation is performed in step S39, the processes in steps S40 to S42 are skipped, and the process proceeds to step S43. 【0175】 In other words, if a modification operation on the movement path is performed and the modification operation is not instructed to end, any interference with an object will be displayed along the movement path, and the distance will be displayed for parts where the distance to the object is less than or equal to a predetermined distance. 【0176】 Therefore, after the modification operation is performed, it is checked whether there is any interference with the object and whether the distance is below a predetermined level. If no further modifications are needed, the modification operation is considered complete. 【0177】 Furthermore, if the completion of the correction operation is instructed in step S43, the process proceeds to step S44. 【0178】 In step S44, it is determined whether or not the termination of the route confirmation process has been instructed. If termination is not instructed, the process proceeds to step S45. 【0179】In step S45, the route interpretation unit 161 reads route data 201 and route correction data 202 from the storage unit 157, interprets the travel route based on the route data 201 and route correction data 202, and supplies the interpreted travel route information to the image generation unit 154. 【0180】 In step S46, the path display unit 181 of the image generation unit 154 acquires information on the position and orientation of the information processing terminal 111 itself from the position and orientation estimation unit 153. 【0181】 In step S47, the route display unit 181 converts the coordinate system of the travel route based on the route data 201 and route correction data 202 to the coordinate system of the image (camera image) supplied by the camera 192 based on the acquired position and orientation information, and displays the travel route superimposed on the image from the camera 192 on the display unit 155. 【0182】 In other words, if the route confirmation process is not completed, the information of the corrected travel route is updated based on the route data 201 and route correction data 202, and superimposed on the camera image captured by the camera 192. 【0183】 Furthermore, in steps S45 to S47, the process assumes that the movement path will be corrected. Therefore, in addition to the path data 201, path correction data 202 is read, and the movement path correction operation is reflected and superimposed on the camera image. However, if no correction operation is performed, the path correction data 202 is not read, and the movement path based only on the path data 201 is superimposed on the camera image. 【0184】 Then, if it is determined in step S44 that the route confirmation process has been instructed to end, the process proceeds to step S48. 【0185】 In step S48, the control unit 151 controls the communication unit 158 ​​to transmit the route data 201 and route correction data 202 stored in the storage unit 157 to the mobile unit 112, and then terminates the process. 【0186】Although not shown in the diagram, in response, the control unit 251 of the mobile unit 112 controls the communication unit 257 to receive the route data 201 and route correction data 202 transmitted from the information processing terminal 111 and store them in the storage unit 256. 【0187】 Through the above process, the user can simply wear the HMD-type information processing terminal 111 and look over the area where the travel route has been planned, and visually confirm whether there is any interference with objects along the travel route planned based on map data, as well as any points closer than a predetermined distance where interference may occur. 【0188】 Furthermore, during verification, movement paths that interfere with objects, and locations where the distance to an object is within a predetermined distance, are displayed differently from other movement paths. Therefore, interference points with objects on the movement path and locations where the distance to an object is within a predetermined distance can be appropriately and easily identified simply by visual inspection. 【0189】 Furthermore, the movement path can be corrected by operating the input unit 156 in cases of interference with objects on the movement path or approaches within a predetermined distance. 【0190】 In this process, the modified sections are recorded as route correction data 202 and can be transmitted to the mobile device 112 together with the route data 201 that was previously generated from the map data. 【0191】 Furthermore, if there is a positional shift in the map data, the entire travel path may be shifted by a certain distance from the intended travel path. As shown in this disclosure, by superimposing the travel path on the camera image, such a shift in the entire travel path can be visually confirmed. In such cases, the travel path can be corrected to an appropriate path by applying an offset equal to the overall positional shift, and no correction is required to the imaging control information. 【0192】 <Movement Control Processing> Next, the movement control processing by the moving body 112 will be explained with reference to the flowchart in Figure 13. 【0193】 In step S71, the control unit 251 reads the route data 201 and route correction data 202 stored in the storage unit 256. 【0194】 In step S72, the control unit 251 acquires information on the position and orientation of the moving body 112 from the position and orientation estimation unit 253. More specifically, the position and orientation estimation unit 253 acquires position information from the GPS 271 of the sensor unit 252, and also acquires angular velocity and acceleration information from the IMU 272 as orientation information, and supplies it to the control unit 251. In response, the control unit 251 acquires the position and orientation information of itself supplied from the position and orientation estimation unit 253 in this manner. 【0195】 In step S73, the route interpretation unit 261 recognizes the travel route based on the read route data 201 and route correction data 202, and identifies the current position on the recognized travel route. 【0196】 In step S74, the route interpretation unit 261 controls the movement control unit 255 to move the mobile body 112 so that the main body of the mobile body 112 moves from its current position on the identified movement path toward the destination. 【0197】 In step S75, the correction amount recognition unit 262 determines, based on the read route data 201 and route correction data 202, whether the current position is on a travel path based on the route correction data 202, which is a modified version of the travel path interpreted by the route data 201. 【0198】 If, in step S75, it is determined that the current location is not on a travel path based on the route correction data 202, which is a modified version of the travel path interpreted by the route data 201, the process proceeds to step S76. 【0199】In step S76, the correction amount recognition unit 262 supplies the current position and orientation, as well as imaging control information included in the route data 201, to the payload control unit 254. The payload control unit 254 controls the gimbal 282 by gimbal control and zoom control based on the current position and orientation, as well as the imaging control information included in the route data 201, to control the imaging direction of the camera 281 and the zoom magnification of the camera 281 to capture an image. 【0200】 On the other hand, if in step S75 it is determined that the current location is on a travel path based on the route correction data 202, which is a modified version of the travel path interpreted by the route data 201, the process proceeds to step S77. 【0201】 In step S77, the correction amount recognition unit 262 estimates the position of the imaging target from the imaging control information included in the route data 201, calculates the correction amount of the imaging control information based on the estimated position of the imaging target and the route correction data 202, and supplies it to the payload control unit 254. 【0202】 In step S78, the payload control unit 254 controls the gimbal 282 by gimbal control and zoom control based on the current position and attitude, as well as the amount of correction of the imaging control information calculated from the path correction data 202, thereby controlling the imaging direction of the camera 281 and controlling the zoom magnification of the camera 281 to capture an image. 【0203】 In step S79, the route interpretation unit 261 determines whether the current position of the moving object 112 is the destination. 【0204】 If, in step S79, it is determined that the current position of the moving object 112 is not the destination, the process returns to step S72, and the subsequent processes are repeated. 【0205】 In other words, the processes in steps S72 to S79 ​​are repeated from the start of movement until arrival at the destination. 【0206】 Then, in step S79, if it is determined that the current position of the moving object 112 is the destination and that the movement should be terminated, the process proceeds to step S80. 【0207】 In step S80, the path interpretation unit 261 controls the movement control unit 255 to stop the moving object 112 at its current position and terminate the movement. That is, if the moving object 112 is a flying object such as a drone, it will land at the destination and terminate the movement. 【0208】 Through the above processing, it becomes possible to move the moving object 112 toward the destination along the travel path recognized based on the route data 201 and the route correction data 202. 【0209】 In this case, when the current position is on a pre-planned route based on map data and route data 201, the imaging direction and zoom magnification of the camera 281 are controlled by gimbal control and zoom control based on the current position and orientation, as well as imaging control information included in the original route data 201, to capture an image. 【0210】 Furthermore, when the current position lies on a movement path based on the path correction data 202 generated by the path confirmation process described above, the current position and orientation, as well as the correction amount for the imaging control information included in the original path data 201, are calculated. Based on the correction amount, the imaging direction and zoom magnification of the camera 281 are controlled by gimbal control and zoom control to capture an image. 【0211】 As a result, it becomes possible to move along a path where no interference or proximity to objects has been confirmed, enabling the mobile body 112 to move safely to its destination. Furthermore, during this process, it becomes possible to appropriately image the target of imaging along the movement path while moving safely without interference or proximity to objects. 【0212】In the above, we have described an example in which the correction amount recognition unit 262 of the mobile body 112 determines the correction amount of the imaging control information based on the route correction data 202. However, when the route correction data 202 is generated in the information processing terminal 111, the imaging control information may also be attached to the route correction data 202 in the same way as the route data 201, as determined by the route correction unit 162. In this case, the mobile body 112 does not need the correction amount recognition unit 262 and only needs to perform zoom control and gimbal control based on the imaging control information attached to both the route data 201 and the route correction data 202. 【0213】 <<3. Description of a computer using this technology>> 【0214】 The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs. 【0215】 Figure 14 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program. 【0216】 In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004. 【0217】 An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010. 【0218】The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. 【0219】 The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009. 【0220】The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting. 【0221】 In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008. 【0222】 The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made. 【0223】 The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing). 【0224】 The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there. 【0225】When the above-described series of processes are performed by a computer executing a program, the processing circuit 1001 (its processor) functions as the path interpretation unit 161 and path correction unit 162 of the control unit 151 in the information processing terminal 111, the path display unit 181 of the image generation unit 154, and the path interpretation unit 261 and correction amount recognition unit 262 of the control unit 251 in the mobile body 112, by executing the program. 【0226】 In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory, for example, only the processing circuit 1001, or a combination of the processing circuit 1001 to the bus 1004, etc., constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system. 【0227】 Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step. 【0228】Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs. 【0229】 Furthermore, for example, the various technologies relating to this disclosure can be implemented independently, as long as they do not conflict with each other. Of course, any combination of the disclosures can also be implemented. For example, some or all of the disclosures described in one embodiment can be implemented in combination with some or all of the disclosures described in another embodiment. Also, some or all of the aforementioned disclosures can be implemented in combination with other technologies not described above. 【0230】Furthermore, this disclosure may also take the following configurations: <1> An information processing method that includes a route display process which superimposes and displays a pre-planned movement path of a moving object based on map data into a real space visible in the direction of the user's line of sight. <2> The information processing method according to <1> which further includes a position and orientation estimation process which estimates the user's position information and orientation information, wherein the route display process superimposes and displays the pre-planned movement path of a moving object based on the map data into a coordinate system based on the position information and orientation information in a real space visible in the direction of the user's line of sight. <3> The information processing method according to <1> which includes a route display process which displays an image of the real space visible in the direction of the user's line of sight, and superimposes and displays the pre-planned movement path of the moving object based on the map data into the real space displayed as the image. The information processing method according to <3>, further comprising: <4> Performing an imaging process to capture an image in the direction of the user's line of sight, wherein the route display process displays the real space visible in the direction of the user's line of sight as an image captured by the imaging process, and overlays and displays the movement path of the moving object, which has been planned in advance based on the map data, within the real space displayed as an image. <5> The information processing method according to <4>, further comprising: Recognizing an object in the image captured by the imaging process, and performing a determination process to determine a position in the movement path whose positional relationship with the object satisfies predetermined conditions, wherein the route display process overlays and displays the movement path in the real space, and displays the position in the movement path that satisfies the predetermined conditions in a different display method than other positions. <6> The information processing method according to <5>, wherein the determination process determines a position in the movement path that interferes with the object, and the route display process displays the position in the movement path that interferes with the object in a different display method than other positions. <7> The information processing method described in <6>, wherein the path display process displays the positions in the movement path that interfere with the object using lines of a different type or color than the lines that represent other positions.<8> The information processing method according to <5>, wherein the determination process determines a position in the movement path where the distance to the object is closer than a predetermined distance, and the path display process displays the position in the movement path where the distance to the object is closer than a predetermined distance using a different display method than the other positions. <9> The information processing method according to <8>, wherein the path display process displays the distance to the object at the position in the movement path where the distance to the object is closer than a predetermined distance. <10> The information processing method according to <1>, further comprising a modification process to modify the movement path which is displayed superimposed in the real space visible in the direction of the user's line of sight. <11> The information processing method according to <10>, wherein the modification process modifies the movement path of the pre-planned moving body in units of sections, records the modified movement path as path modification data, and transmits the path data consisting of the pre-planned movement path of the moving body and the path modification data to the moving body. <12> The information processing method according to <11>, wherein the moving body moves along the movement path based on the path data for sections excluding the section in which the movement path has been corrected based on the path correction data, and moves along the corrected path based on the path correction data for sections in which the movement path has been corrected. <13> The information processing method according to <11>, wherein the moving body is equipped with a camera that takes an image of a predetermined object while moving, based on the path data and the path correction data, the path data consisting of a predetermined movement path of the moving body includes imaging control information that controls the imaging direction and zoom magnification of the camera for the moving body to take an image of the object while moving, and when the moving body moves along the movement path based on the path data, the imaging direction and zoom magnification of the camera are controlled based on the imaging control information. <14> The information processing method described in <13>, wherein the moving body calculates the amount of correction for the imaging control information when moving along the corrected path based on the path correction data, based on the imaging control information of the path data and the path correction data, and when the moving body moves along the corrected path based on the path correction data, the imaging direction and zoom magnification of the camera are controlled based on the amount of correction.<15> The information processing method according to <10>, wherein the correction process corrects the movement path superimposed in the real space based on a correction instruction corresponding to at least one of the user's speech, hand signs, and operation of a touch panel. <16> An information processing device comprising a path display unit that superimposes and displays a predetermined movement path of a moving object based on map data in the real space visible in the user's line of sight. <17> A program that causes a computer to function as a path display unit that superimposes and displays a predetermined movement path of a moving object based on map data in the real space visible in the user's line of sight. 【0231】 101 Information processing system, 111 Information processing terminal, 112 Mobile unit, 151 Control unit, 152 Sensor unit, 153 Position and orientation estimation unit, 154 Image generation unit, 155 Display unit, 156 Input unit, 157 Storage unit, 158 Communication unit, 161 Route interpretation unit, 162 Route correction unit, 171 GPS, 172 IMU, 181 Route display unit, 182 Interference detection unit, 191 Microphone, 192 Camera, 193 Touch panel, 201 Route data, 202 Route correction data, 251 Control unit, 252 Sensor unit, 253 Position and orientation estimation unit, 254 Payload control unit, 255 Movement control unit, 261 Route interpretation unit, 262 Correction amount recognition unit, 271 GPS, 272 IMU, 281 camera, 282 gimbal

Claims

1. An information processing method that includes a route display process that superimposes the movement path of a pre-planned moving object onto the real space visible in the user's line of sight, based on map data.

2. The information processing method according to claim 1, further comprising performing a position and orientation estimation process to estimate the user's position and orientation information, wherein the route display process displays a pre-planned movement path of a moving object based on the map data, converted to a coordinate system based on the position and orientation information, superimposed in the real space visible in the user's line of sight.

3. The information processing method according to claim 1, wherein the route display processing displays an image of the real space visible in the user's line of sight, and superimposes the movement route of the moving object, which has been planned in advance based on the map data, onto the real space displayed as the image.

4. The information processing method according to claim 3, further comprising an imaging process that captures an image in the direction of the user's line of sight, wherein the route display process displays the real space visible in the direction of the user's line of sight as an image captured by the imaging process, and superimposes the movement route of the moving object, which has been planned in advance based on the map data, onto the real space displayed as an image.

5. The information processing method according to claim 4, further comprising: recognizing an object in the image captured by the imaging process; determining a position in the movement path that satisfies a predetermined condition in relation to the object, wherein the path display process superimposes and displays the movement path in the real space; and displays the position in the movement path that satisfies the predetermined condition in a different display method from the other positions.

6. The information processing method according to claim 5, wherein the determination process determines a position in the movement path that interferes with the object, and the path display process displays the position in the movement path that interferes with the object in a different display method than the other positions.

7. The information processing method according to claim 6, wherein the path display process displays the positions in the movement path that interfere with the object using lines of a different type or color than the lines that represent other positions.

8. The information processing method according to claim 5, wherein the determination process determines a position in the movement path where the distance to the object is closer than a predetermined distance, and the path display process displays the position in the movement path where the distance to the object is closer than a predetermined distance using a different display method than the other positions.

9. The information processing method according to claim 8, wherein the route display process displays the distance to the object at a position in the travel route where the distance to the object is closer than a predetermined distance.

10. The information processing method according to claim 1, further comprising a modification process for modifying the movement path that is superimposed on the real space visible in the user's line of sight.

11. The information processing method according to claim 10, wherein the correction process corrects the movement path of the pre-planned moving body in units of sections, records the corrected movement path as route correction data, and transmits the route data consisting of the pre-planned movement path of the moving body and the route correction data to the moving body.

12. The information processing method according to claim 11, wherein the moving body moves along the movement path based on the movement data for sections excluding the section in which the movement path has been corrected based on the movement path correction data, and moves along the corrected section in which the movement path has been corrected based on the movement path correction data.

13. The information processing method according to claim 11, wherein the moving body is equipped with a camera that takes an image of a predetermined object while moving based on the path data and the path correction data, the path data consisting of a predetermined movement path of the moving body includes imaging control information that controls the imaging direction and zoom magnification of the camera for the moving body to take an image of the object while moving, and when the moving body moves along the movement path based on the path data, the imaging direction and zoom magnification of the camera are controlled based on the imaging control information.

14. The information processing method according to claim 13, wherein the moving body calculates the amount of correction for the imaging control information when moving along the corrected path based on the path correction data, based on the imaging control information of the path data and the path correction data, and when the moving body moves along the corrected path based on the path correction data, the imaging direction and zoom magnification of the camera are controlled based on the amount of correction.

15. The information processing method according to claim 10, wherein the correction process corrects the movement path superimposed in the real space based on a correction instruction corresponding to at least one of the user's speech, hand signs, and touch panel operation.

16. An information processing device comprising a route display unit that superimposes and displays the movement path of a pre-planned moving object based on map data within the real space visible in the user's line of sight.

17. A program that causes a computer to function as a route display unit, overlaying and displaying a pre-planned movement path of a moving object based on map data within the real space visible in the user's line of sight.

Images