Information processing system, information processing method, and program

Abstract

This information processing system comprises: a managed object information storage unit that stores, for each object to be managed existing in a real space, position coordinates of at least one representative point of the object to be managed; an image data acquisition unit that acquires an image captured in the real space and image metadata including data indicating the imaging position and the imaging direction of the image; and an image sorting unit that identifies the positional relationship between the representative point and the imaging position on the basis of the image metadata and the position coordinates of the representative point, and identifies an object captured in the image from among the objects to be managed on the basis of the positional relationship.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present disclosure relates to an information processing system, an information processing method, and a program related to management of image data.

[0002] As shown in Patent Document 1 and Patent Document 2, systems are known that perform inspection by imaging structures, industrial facilities, or social infrastructure facilities (so-called infrastructure facilities) using imaging devices.

[0003] Japanese Unexamined Patent Publication No. 2019-179422 Japanese Unexamined Patent Publication No. 2020-183873

[0004] In the inspection of facilities as described above, a large number of images may be captured to record the states of numerous management targets (devices, equipment, etc.) present in the facility. To make effective use of these images, it is necessary to identify which management target each image captures, and to perform the work of sorting the images for each management target.

[0005] As a method for sorting images captured during inspection, it is conceivable to use image analysis technology. For example, Patent Document 1 discloses a method of identifying a target object in a captured image by pattern matching with previously acquired images, and Patent Document 2 discloses a method of identifying an observation target in a captured image through object recognition using deep learning. However, these image analysis technologies have high computational loads, and processing a large volume of images can take time. In addition, when there are multiple management targets with similar appearances, such as the case where multiple pieces of equipment of the same type exist in a facility, it is difficult to ensure accuracy through image analysis, and the actual situation is that manual sorting work by workers through visual inspection becomes necessary. Therefore, there is a demand for a versatile and convenient technology that sorts and manages captured images regardless of the number of images to be processed.

[0006] An object of the exemplary embodiments of the present disclosure is to provide an information processing system, an information processing method, and a program for supporting management of image data.

[0007] An information processing system relating to one aspect of the present disclosure includes: a managed object information storage unit that stores the position coordinates of at least one representative point of each managed object in real space; an image data acquisition unit that acquires an image taken in real space and image accessory information including data indicating the shooting position and direction of the image; and an image sorting unit that identifies the positional relationship between the representative point and the shooting position based on the image accessory information and the position coordinates of the representative point, and identifies the object captured in the image from among the managed objects based on the positional relationship.

[0008] Because the information processing system has the above-mentioned features, it is possible to sort and manage images acquired during inspection work, etc., according to the target object, in a more general and simpler manner than before. Further details regarding the problems disclosed in this application and their solutions will be clarified in the section on embodiments and drawings of this disclosure.

[0009] Figure 1 is a diagram illustrating the configuration of an information processing system according to one embodiment of the present disclosure. Figure 2 is a block diagram showing the hardware configuration of the management server shown in Figure 1. Figure 3 is a block diagram showing the hardware configuration of the user terminal shown in Figure 1. Figure 4 is a block diagram showing the hardware configuration of the mobile device shown in Figure 1. Figure 5 is a block diagram illustrating the functions of the management server. Figure 6 is a conceptual diagram showing an example of operation of the information processing system shown in Figure 1. Figure 7 is a conceptual diagram showing another example of operation of the information processing system shown in Figure 1. Figure 8 is a conceptual diagram showing an example of a method for identifying an object to be photographed. Figure 9 is a flowchart showing an example of a method for identifying an object to be photographed.

[0010] The information processing method, information processing system, and program disclosed herein include, for example, the following configurations: [Item 1] An information processing system comprising: a managed object information storage unit that stores the position coordinates of at least one representative point of each managed object in real space; an image data acquisition unit that acquires an image taken in real space and image accessory information including data indicating the shooting position and shooting direction of the image; and an image sorting unit that identifies the positional relationship between the representative point and the shooting position based on the image accessory information and the position coordinates of the representative point, and identifies the object to be photographed in the image from among the managed objects based on the positional relationship. [Item 2] The information processing system according to Item 1, wherein the image sorting unit calculates the distance between the representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction, for each of the managed objects, and identifies the managed object with the smallest distance and / or angle as the object to be photographed. [Item 3] The information processing system according to Item 1 or 2, wherein the image sorting unit extracts objects to be managed in which the distance between a representative point and the shooting position is within a predetermined range, and identifies the object to be photographed from among the objects to be managed in which the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction is smallest. [Item 4] The information processing system according to Item 1 or 2, wherein the image sorting unit extracts objects to be managed in which the representative point is located within a predetermined range based on the shooting direction, and identifies the object to be photographed from among the objects to be managed in which the distance between the representative point and the shooting position is smallest. [Item 5] The information processing system according to Item 1 or 2, wherein the image data acquisition unit acquires the image along with the position information of feature points measured at the time of image capture, and the image sorting unit identifies the object to be photographed in the image by specifying the position relationship between the representative point and the feature point in lieu of the position relationship between the representative point and the shooting position, or by specifying the position relationship between the representative point and the feature point in addition to the position relationship between the representative point and the shooting position.[Item 6] The position coordinates of the representative point are represented in a two-dimensional coordinate system including a first axis component and a second axis component, and the image sorting unit identifies the two-dimensional positional relationship between the representative point and the shooting position using data corresponding to the first axis component and the second axis component from the data indicating the shooting position and the shooting direction, as described in item 1 or 2. [Item 7] The position coordinates of the representative point are represented in a three-dimensional coordinate system, and the image sorting unit identifies the three-dimensional positional relationship between the representative point and the shooting position using the shooting position and the shooting direction represented in a three-dimensional coordinate system, as described in item 1 or 2. [Item 8] In identifying the positional relationship between the representative point and the shooting position, the image sorting unit automatically switches between a mode for identifying a two-dimensional positional relationship and a mode for identifying a three-dimensional positional relationship depending on the data format of the position coordinates of the representative point, as described in item 1 or 2. [Item 9] An information processing method in which a computer performs the following: for each managed object existing in real space, it stores the position coordinates of at least one representative point of the managed object; it acquires an image taken in the real space and image-related information including data indicating the shooting position and direction of the image; it identifies the positional relationship between the representative point and the shooting position based on the image-related information and the position coordinates of the representative point, and identifies the object captured in the image from among the managed objects based on that positional relationship. [Item 10] A program for causing a computer to perform the following: for each managed object existing in real space, it stores the position coordinates of at least one representative point of the managed object; it acquires an image taken in the real space and image-related information including data indicating the shooting position and direction of the image; it identifies the positional relationship between the representative point and the shooting position based on the image-related information and the position coordinates of the representative point, and identifies the object captured in the image from among the managed objects based on that positional relationship.

[0011] <Details of the Embodiment> An information processing system according to one embodiment of the present disclosure will be described with reference to the drawings. In the attached drawings, identical or similar elements are given identical or similar reference numerals and names, and redundant descriptions of identical or similar elements may be omitted in the description of the embodiment. The contents shown in each drawing are merely examples for explaining this embodiment and are only illustrative examples to facilitate the explanation of this embodiment. The contents of each drawing may be modified or changed to the extent that no technical problems arise.

[0012] <System Overview> The information processing system according to this embodiment (hereinafter referred to as "this system") is a system that performs processing to sort and manage multiple images captured in a predetermined real space according to the object captured in each image.

[0013] The purpose of capturing images managed by this system is not necessarily limited, and examples include various inspections for purposes such as facility management, construction management, patrols, security, disaster response such as damage assessment, and monitoring in agriculture, forestry, and fisheries. The target space for image acquisition is set according to the above-mentioned purpose and includes multiple objects to be managed. Objects to be managed are physical entities that exist in real space and should be individually identified and managed, and include buildings, structures, equipment, machinery, devices, topographic features, and vegetation. For example, each piece of equipment in a factory, each component of an infrastructure facility, each piece of machinery in a building, and each section of farmland can be objects to be managed. Other objects to be managed may include people and vehicles. The target space for image capture is mainly assumed to be outdoors, but is not necessarily limited to this, and may also be indoor space or space that spans both indoors and outdoors.

[0014] Furthermore, the method of capturing images managed by this system is not particularly limited. For example, the real space may be captured using a camera mounted on an unmanned mobile vehicle such as a drone, unmanned aerial vehicle (UAV), or unmanned ground vehicle (UGV). Alternatively, a person such as a worker may operate a camera (for example, a smartphone, tablet, digital camera, or other type of camera) while moving within the target space and taking pictures, or the target space may be captured using a camera mounted on a manned mobile vehicle.

[0015] In this embodiment, for the sake of brevity, we will describe in detail the information processing performed by this system by illustrating the case in which a mobile unit 4 is used to inspect a plant facility where multiple pieces of equipment exist as objects to be managed. As shown in Figures 6 and 7, the mobile unit 4 is flown within the plant facility, which is the target space, and each piece of equipment present in the facility is photographed with a camera 42 mounted on the mobile unit 4. The mobile unit 4 may be flown by a user (worker, etc.) using a remote control, or it may be autonomously navigated based on a flight path for which waypoints have been set in advance. The model and specifications of the mobile unit 4 are not particularly limited. For example, it is preferable that the mobile unit 4 is a model that can record various shooting conditions such as the position of the aircraft at the time of shooting (shooting position), the orientation of the aircraft, and the orientation of the gimbal as supplementary information to the image (e.g., Exif information (Exif; Exchangeable image file format)).

[0016] In facilities such as plants, there are numerous pieces of equipment to be managed, and inspection work generates a large number of images corresponding to the number of pieces of equipment being managed. These images need to be sorted and managed according to the subject (object being photographed) depicted in each image so that they can be easily used in facility management operations. Conventionally, attempts have been made to sort (group) images into predetermined areas according to the shooting location using location information stored as image metadata, but in reality, determining which specific piece of equipment each image captures still relies on the visual judgment of the worker.

[0017] Methods for identifying the object being photographed in each image include pattern matching between the photographed image and reference images of previously photographed images or each object being managed, and using image analysis techniques such as deep learning to detect objects within the photographed image. However, such image analysis techniques have a high computational load, and processing becomes time-consuming when dealing with thousands or more images. Furthermore, in facilities such as plants, there may be multiple pieces of equipment of the same type, and if there are multiple objects being managed that look similar, it is difficult to ensure accuracy through image analysis, and in reality, manual sorting by human visual inspection is often necessary.

[0018] In addition to the above, if a 3D model of the structure exists, another method is to project the images onto the 3D model to identify which part of the 3D model each image corresponds to. However, to adopt this method, it is necessary to construct an accurate 3D model that conforms to the real space in advance so that the object being photographed can be identified. In particular, when the target space is a wide outdoor area such as a large-scale plant, it is conceivable that a 3D model corresponding to the target space has not been constructed, or that it is difficult to prepare a 3D model in advance. Therefore, there is a need for a technology that can sort image data in a general-purpose and simple manner, regardless of the number of images to be processed.

[0019] In this system, the position coordinates of at least one representative point are registered for each managed object. The system compares the representative point of each managed object with the image-related information recorded with the image (shooting position, shooting direction, etc.) to determine the positional relationship between the representative point and the shooting position. The positional relationship between the representative point and the shooting position is indicated, for example, by the distance between the representative point and the shooting position, and / or the angle of the line connecting the representative point and the shooting position with respect to the shooting direction (i.e., the angle difference between the direction indicated by the shooting direction and the direction from the shooting position toward the representative point). This positional relationship is used to identify the object captured in the image from among the managed objects.

[0020] The system, possessing the features described above, can sort captured images in a simpler way than conventional image analysis techniques, even in environments where it is difficult to apply conventional image analysis techniques, such as when there are multiple objects with similar appearances. Furthermore, it does not require prior preparation such as preparing reference images such as past images for pattern matching, constructing image analysis models using deep learning, or constructing reference 3D models. Automatic sorting of images based on the position coordinates of representative points is possible, making this system highly versatile and applicable to various environments. The information processing system of this embodiment will be described in detail below, using the inspection of equipment within a facility using a UAV (mobile vehicle 4) as an example.

[0021] <System Configuration> As shown in Figure 1, the information processing system of this embodiment may include a management server 1, one or more user terminals 2, and one or more mobile devices 4. The management server 1, the user terminals 2, and the mobile devices 4 are connected to each other via a network NW so that they can communicate with one another. Note that the illustrated configuration is just an example and is not limited thereto. For example, the mobile device 4 does not have to be connected to a network NW. In that case, the mobile device 4 may be operated by a transmitter (so-called remote control) operated by the user, or image data acquired by the camera of the mobile device 4 may be stored in an auxiliary storage device connected to the mobile device 4 (for example, a memory card such as an SD card and / or a USB memory), and subsequently read from the auxiliary storage device to the user terminal 2 and / or management server 1 and stored by the user. The mobile device 4 may be connected to a network NW only for the purpose of operation or for the purpose of storing image data.

[0022] <Management Server 1> Figure 2 shows the hardware configuration of Management Server 1. Note that the configuration shown is just one example, and Management Server 1 may have a different configuration.

[0023] The management server 1 may be configured on-premises using a general-purpose computer such as a workstation or personal computer, or it may be logically implemented through cloud computing. The management server 1 includes at least a processor 10, memory 11, storage 12, a transceiver 13, an input / output unit 14, etc., which are electrically connected to each other via a bus 15.

[0024] The processor 10 is a computing device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and loaded into the memory 11 to perform various information processing tasks.

[0025] Memory 11 includes main memory, which is composed of volatile storage devices such as DRAM (Dynamic Random Access Memory), and auxiliary memory, which is composed of non-volatile storage devices such as flash memory and HDD (Hard Disk Drive). Memory 11 is used as a work area for the processor 10, and also stores the BIOS (Basic Input / Output System) executed when the management server 1 starts up, as well as various configuration information.

[0026] Storage 12 stores various programs, such as application programs. A database containing data used for each process may be built in storage 12. For example, the memory unit 120 described later may be provided in a part of the storage area of ​​storage 12.

[0027] The transmitting / receiving unit 13 is a communication interface for the management server 1 to communicate with user terminals 2 and mobile devices 4, etc., via a communication network. The transmitting / receiving unit 13 may further include Bluetooth® and BLE (Bluetooth Low Energy) short-range communication interfaces and / or USB (Universal Serial Bus) terminals, etc.

[0028] The input / output unit 14 consists of information input devices such as keyboards and mice, and output devices such as displays.

[0029] Bus 15 is connected in common to all of the above elements and transmits, for example, address signals, data signals, and various control signals.

[0030] <User Terminal 2> The user terminal 2 shown in Figure 3 also includes a processor 20, memory 21, storage 22, a transmitting / receiving unit 23, an input / output unit 24, etc., which are electrically connected to each other via a bus 25. The user terminal 2 may be, for example, a transmitter or other remote control, a general-purpose computer such as a workstation or personal computer, or a mobile terminal such as a smartphone or tablet. The functions of each element of the user terminal 2 can be configured in the same way as the management server 1 described above, and a detailed explanation of each element of the user terminal 2 is omitted.

[0031] <Mobile Unit 4> Figure 4 is a block diagram showing the hardware configuration of the mobile unit 4. The flight controller 41 may have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)).

[0032] The flight controller 41 may also have a memory 411, which is accessible. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412 such as inertial sensors (accelerometers, gyroscopes), GPS sensors, GNSS sensors, and proximity sensors (e.g., LiDAR).

[0033] The memory 411 may include, for example, a separable medium such as an SD card and random access memory (RAM), or an external storage device. Data acquired from the cameras / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still images and video data captured by the camera may be recorded in the internal memory or external memory, but is not limited to this; the data may also be recorded from the camera / sensors 42 or the internal memory via the network NW to at least one of the management server 1 and the user terminal 2. The camera 42 is mounted on the mobile body 4 via a gimbal 43.

[0034] The flight controller 41 includes a control module (not shown) configured to control the state of the mobile body 4. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ). ｘ , θ ｙ and θ ｚ To adjust the spatial arrangement, speed, and / or acceleration of the aircraft having the following characteristics, the propulsion mechanism (motor 45, etc.) of the aircraft is controlled via the ESC 44 (Electric Speed ​​Controller). The motor 45, powered by the battery 48, rotates the propeller 46, generating lift for the aircraft. The control module of the flight controller 41 may have functions to control the drive of the camera 42 and shooting conditions (e.g., sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). The control module can also control one or more of the states of the mounted parts and sensors.

[0035] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transceiver (remote control) 49, a terminal, a display device, or other remote control). The transceiver 49 may use any suitable means of communication, such as wired communication or wireless communication.

[0036] For example, the transmitting / receiving unit 47 may utilize one or more of the following: local area network (LAN), wide area network (WAN), infrared, wireless, Wi-Fi, point-to-point (P2P) network, telecommunications network, cloud communication, etc.

[0037] The transmitting / receiving unit 47 can transmit and / or receive one or more of the following: data acquired by cameras / sensors 42, processing results generated by the flight controller 41, predetermined control data, and user commands from a terminal or remote controller.

[0038] The camera / sensor array 42 may include inertial sensors (accelerometers, gyroscopes), GPS sensors, GNSS sensors, proximity sensors (e.g., LiDAR), or vision / image sensors (e.g., cameras).

[0039] <Functions of Management Server 1> Figure 5 is a block diagram illustrating the functions implemented in Management Server 1. In this embodiment, Management Server 1 may include an image data acquisition unit 101, an image sorting unit 102, and a data output unit 103. Furthermore, the storage unit 120 of Management Server 1 may include various databases such as a managed information storage unit 121, a captured image data storage unit 122, and a movement information storage unit 123.

[0040] First, the various databases of the storage unit 120 will be described in detail. The managed object information storage unit 121 is a database that stores information about each managed object that exists in the physical space of a plant facility, etc. The managed object information storage unit 121 stores identification information that uniquely identifies each managed object, basic information of each managed object, and the position coordinates of at least one representative point of each managed object for each managed object. The basic information of a managed object is not particularly limited and may include, for example, the name, type, manufacturer, installation date, last inspection date, maintenance history, dimensional information, weight information, and capacity information of the managed object. As an example of type, it may include information indicating the type of various equipment in the plant facility (for example, tanks, piping, pumps, valves, motors, generators, control devices, etc.).

[0041] A representative point is a point that represents the location of each managed object. For example, it may be the center position of an object such as equipment, or a specific part that is important for management may be designated as the representative point. For large managed objects or managed objects with complex shapes, multiple representative points may be set for a single managed object. The managed object information storage unit 121 stores the position coordinates of the representative points set for each managed object. The position coordinates of the representative points may be represented in a two-dimensional coordinate system including a first axis component and a second axis component. For example, it may be a two-dimensional coordinate system represented by latitude and longitude without height information, in which case latitude corresponds to the first axis component and longitude to the second axis component. Alternatively, the position coordinates of the representative points may be represented in a three-dimensional coordinate system. For example, a three-dimensional coordinate system of latitude, longitude, and height is applicable, in which case latitude corresponds to the first axis component, longitude to the second axis component and height to the third axis component. In addition to latitude and longitude, the two-dimensional coordinate system may also be represented by a two-dimensional Cartesian coordinate system (e.g., x-axis and y-axis components) that indicates the relative distance from a predetermined reference point. Similarly, in addition to latitude, longitude, and height, the three-dimensional coordinate system may also be represented by a three-dimensional Cartesian coordinate system (e.g., x-axis, y-axis, and z-axis components) that indicates the relative distance from a predetermined reference point.

[0042] In this embodiment, the data format for the position coordinates of the representative point may be selectable when registering information for each managed object, and the managed object information storage unit 121 may store information indicating whether the position coordinates of the representative point are represented in a two-dimensional coordinate system or a three-dimensional coordinate system for each managed object. Such information indicating the data format may be used by the image sorting unit 102, described later, to automatically switch between a mode for identifying a two-dimensional positional relationship and a mode for identifying a three-dimensional positional relationship, depending on the data format of the position coordinates of the representative point, when identifying the positional relationship between the representative point and the shooting position.

[0043] In addition to the above, the information related to the management target objects stored in the management target information storage unit 121 may include information that contributes to inspection work according to the attributes of the management target objects. For example, various information related to the inspection and maintenance of the management target objects may be included, such as information identifying drawings, specifications, etc. related to the management target objects, inspection items, inspection procedures, inspection frequencies, inspection histories, and response procedures upon abnormality detection. The data items (entities) of the information related to the management target objects are not necessarily limited, and may be appropriately set according to the characteristics and / or management purpose of the management target objects.

[0044] The captured image data storage unit 122 stores captured image data that is acquired by photographing the inside of the target space (inside the facility) by the cameras / sensors 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1. The captured image data acquired by photographing with the cameras / sensors 42 includes a captured image showing at least a part of the target space, and image auxiliary information associated with the captured image. The image auxiliary information includes, for example, data indicating the shooting position and shooting direction of the image, and may further include other shooting condition information.

[0045] The shooting position indicates the position of the camera 42 in real space at the time the image was captured and is expressed in a real-space coordinate system. The method for determining the shooting position at the time of shooting is not particularly limited. When the shooting position is obtained using a GNSS sensor or GPS sensor, the shooting position may be expressed in three-dimensional coordinates of latitude, longitude, and absolute altitude. Alternatively, when the shooting position is obtained in a non-GNSS environment using a distance measuring sensor such as LiDAR, an inertial sensor (accelerometer, gyroscope), or the camera's vision / image sensor, the shooting position may be expressed in relative three-dimensional coordinates with the starting position of the moving object 4 (shooting operation start position) as the reference position (origin). Furthermore, when the operator takes pictures using a mobile device such as a tablet or smartphone, the position data obtained by the GPS sensor and / or acceleration / direction sensor installed in the mobile device may be used as the shooting position. In this case as well, the shooting position may be expressed in three-dimensional coordinates of latitude, longitude, and altitude, or it may be expressed as a relative position from a pre-set reference point. Furthermore, the shooting location may be recorded in multiple representation formats within a single image. For example, both the shooting location coordinates (latitude, longitude, altitude) identified using a GPS sensor or GNSS sensor, and the shooting location coordinates identified using a self-position estimation method other than GNSS, such as LiDAR-SLAM or Visual-SLAM, may be recorded.

[0046] The shooting direction is information indicating the direction in the real space toward which the optical central axis of the camera 42 points at the time the image is shot, and the data format of the shooting direction and the method for specifying the shooting direction at the time of shooting are not particularly limited. For example, it may be expressed as an azimuth (azimuth angle) measured with a predetermined direction such as true north as a reference (0 degrees). In this case, east is 90 degrees, south is 180 degrees, and west is 270 degrees. Further, the shooting direction may be expressed by two angle values: a horizontal angle (azimuth angle) and a vertical angle (elevation angle / depression angle). Furthermore, the shooting direction may be expressed as a three-dimensional vector, and in this case, the shooting direction may be expressed as a unit vector composed of three components x, y, and z corresponding to "latitude, longitude, and altitude". In addition, the shooting direction may be expressed by three Euler angles representing the attitude of the camera: a roll angle (rotation angle around the optical axis), a pitch angle (inclination angle with respect to the horizontal plane), and a yaw angle (rotation angle within the horizontal plane). Regarding the shooting direction, the shooting direction may be recorded in a plurality of expression formats for one image. For example, both the shooting direction specified based on the Yaw angle of the gimbal and the shooting direction specified based on the built-in compass may be recorded.

[0047] The information on the shooting direction may be specified using any one or more pieces of information such as the orientation of a gimbal mounted on the moving body 4, the orientation of the airframe, and a built-in compass. Specifically, when the moving body 4 is a drone, Gimbal Yaw Degree (the yaw angle of the gimbal) may be recorded in the auxiliary information of the image as data indicating the shooting direction. In addition, when shooting is performed with a GPS-compatible camera, GPS Img Direction (the orientation of the camera at the time of shooting) and / or GPS Dest Bearing (the azimuth of the subject) may be recorded in the auxiliary information of the image as data indicating the shooting direction. When an operator performs shooting using a mobile terminal, azimuth information obtained from an azimuth sensor (electronic compass) built into the terminal may be recorded as the shooting direction.

[0048] In addition to the shooting position and direction mentioned above, the image supplementary information may also include various shooting condition information. For example, information regarding the optical characteristics of the camera may include the focal length, angle of view, zoom ratio, lens type, and distortion correction coefficient of the camera 42 at the time of shooting. Information regarding shooting settings may include exposure, shutter speed, aperture value (F-number), ISO sensitivity, white balance setting, and whether or not a flash was used. Information regarding the shooting situation may include the date and time of shooting, photographer information, name of the shooting location, and weather conditions (weather, temperature, humidity, etc.). Furthermore, information regarding image processing may include information useful for the purpose of shooting or post-processing, such as image resolution, image format, color space, compression ratio, overlap ratio (degree of overlap between images during continuous shooting), and baseline length in the case of stereo shooting. In addition, the image supplementary information may also include information acquired by various sensors at the time of shooting (for example, distance measurement information). These image-related information may be metadata attached to the captured image, and may be stored in formats such as Exif (Exchangeable image file format) information and / or XMP (Extensible Metadata Platform).

[0049] The captured image data storage unit 122 may store information indicating feature points acquired during image capture, linked to the image. Here, "feature point" refers to a point with three-dimensional coordinates indicating a physical position in real space, and refers to a point located on the surface of an object in the vicinity of the shooting position or within the shooting range. Feature points are used, for example, as reference points to understand the shape and / or position of objects in or around the shooting range. The method of acquiring feature points is not particularly limited; for example, feature points may be point cloud data acquired by a distance measuring sensor such as LiDAR mounted on the mobile body 4. Alternatively, they may be three-dimensional point cloud data acquired by a camera equipped with a vision / image sensor. Alternatively, they may be three-dimensional coordinates of feature points generated from multiple images taken from different viewpoints using Structure from Motion (SfM) technology, or feature points extracted in the process of constructing a three-dimensional map of the environment using Simultanate Localization and Mapping (SLAM) technology. The position coordinates of each feature point may be expressed in the same data format as the shooting location, specifically in the GNSS coordinate system (latitude, longitude, altitude) or in the local coordinate system. Alternatively, they may be recorded as relative coordinates with the shooting location as the origin. In addition to the position coordinates, the data for each feature point may include additional information such as the distance from the shooting location to the feature point, reflectance, color information, normal vector, and confidence score.

[0050] Each image data captured and stored in the image data storage unit 122 may be associated with information indicating the object to be photographed, which has been identified as the subject of each image by the image sorting unit 102 described later. This information indicating the subject to be photographed may include, for example, identification information that uniquely identifies the main subject of the image. It may also include the name of the object to be photographed, and information indicating the area within the target space (facility) where the object to be photographed is located. Furthermore, in addition to information indicating the main subject of the image, the associated information with the image data may include information indicating other equipment, etc., that are visible within the shooting range.

[0051] The movement information storage unit 123 stores movement information used in movement for purposes such as photographing various objects to be managed in a target space, such as a plant facility. The movement information is instruction information for executing movement and may include, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (shooting direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.). The movement information storage unit 123 may also store movement acquisition information acquired as actual results by executing movement. Movement acquisition information may include the actual movement route, movement speed, flight altitude, captured images, and shooting condition information associated with said captured images.

[0052] Movement information as instruction information may be generated, for example, by setting parameters of various information included in the movement information on the management server 1 or user terminal 2. As for the movement path, for example, the location of the mobile body storage device may be used as the start and end points of the movement, and a movement path passing through each waypoint may be generated. Alternatively, without a mobile body storage device, the location where the user carries the mobile body 4 may be used as the start point of the movement (so-called home point), or the user may retrieve the mobile body 4 at the end point of the movement (maybe returning to the home point). Alternatively, the movement path may be generated based on information about the mobile body storage device managed by the management server 1 (e.g., location information, storage status information, storage device information, etc.), including the location of the mobile body storage device selected as the start or end point of the movement.

[0053] Next, we will describe in detail each functional part of the processor 10.

[0054] The image data acquisition unit 101 acquires images taken in a real space such as a plant facility using a predetermined shooting means such as the camera 42 of the mobile body 4, and image-related information such as metadata recorded attached to the images. The images and image-related information acquired by the image data acquisition unit 101 are used as input information in the object identification process by the image sorting unit 102 described later. The image data acquisition unit 101 acquires data indicating the shooting position and / or shooting direction of the image as image-related information, and may also acquire other shooting condition information such as the focal length and angle of view when the image was taken with the camera 42.

[0055] If the image data acquisition unit 101 finds that the data format (representation format) of the shooting position and / or shooting direction in the image supplemental information differs from the data format of the representative point of each managed object stored in the managed object information storage unit 121, it may perform a process to convert the data format of the shooting position and / or shooting direction to the data format of the representative point. The data format conversion may be performed using a pre-set conversion coefficient. For example, if the position coordinates of the representative point are expressed in a geographic coordinate system including latitude and longitude, while the shooting position indicated in the image supplemental information is expressed in a three-axis Cartesian coordinate system of x, y, and z axes, which is different from the geographic coordinate system, the image data acquisition unit 101 may perform a process to convert the position coordinates of the shooting position to the geographic coordinate system using a conversion coefficient that shows the correlation between the geographic coordinate system and the three-axis Cartesian coordinate system. Note that the shooting position and / or shooting direction in the image supplemental information may be matched to the data format of the representative point as described above, or conversely, data format conversion such as coordinate transformation may be performed to match the data format of the representative point to the data format of the shooting position and / or shooting direction in the image supplemental information.

[0056] If the shooting position and / or shooting direction of the image-related information corresponding to a single image is recorded in multiple data formats (i.e., the image-related information includes multiple shooting positions and / or shooting directions expressed in different formats), the image data acquisition unit 101 may perform a process to select one shooting position and / or shooting direction to be used for identifying the object to be photographed from among the multiple representation formats included in the image-related information. The method of selecting the data in this case is not particularly limited, and the shooting position and / or shooting direction expressed in a data format that matches the representation format of the position coordinates of the representative point may be selected, or the shooting position and / or shooting direction may be selected according to a predetermined rule.

[0057] For example, if a shooting location for a single image is obtained as both a shooting location measured using a GPS sensor and represented by latitude, longitude, and altitude, and a shooting location measured using other sensors such as LiDAR or vision / image sensors and represented in a three-axis Cartesian coordinate system, then if the position coordinates of the representative point of each managed object are registered in a geographic coordinate system including latitude and longitude, the shooting location may also be adopted as the shooting location in the geographic coordinate system obtained by the GPS sensor. Furthermore, regarding the shooting direction, a selection rule may be established such that the first priority is "Gimbal Yaw Degree", the second priority is "GPS Image Direction", and the third priority is "GPS Dest Bearing", and a single shooting direction to be used to identify the object being photographed may be selected.

[0058] Furthermore, if multiple shooting locations measured using different methods are recorded in the image supplementary information, these multiple shooting locations may be integrated to calculate a single shooting location. The method for integrating coordinates in this case is not particularly limited, and any known method may be used. Similarly to the shooting direction, the shooting directions identified by multiple methods may be integrated to calculate a single shooting direction.

[0059] Furthermore, the image data acquisition unit 101 may acquire positional information of feature points measured at the time of image capture, along with acquiring the image. If multiple feature points are measured within the image capture range, the image data acquisition unit 101 may perform a process to select one feature point from among the acquired feature points to be referenced in the process of identifying the object to be captured. The method of selecting feature points is not necessarily limited; for example, the feature point closest to the shooting position within the shooting range may be selected, or feature points existing within the shooting direction or a predetermined angle range with the shooting direction as the axis may be initially extracted, and then the feature point closest to the shooting position may be selected from among the initially extracted feature points. In the latter case, where feature points are initially extracted based on the shooting direction, the center coordinates of the initially extracted feature points may be calculated and these center coordinates may be set as the positional information of the feature points.

[0060] The timing at which the image data acquisition unit 101 acquires images and associated image information (shooting position, shooting direction) is not particularly limited. For example, the image data acquisition unit 101 may receive images and associated image information from the mobile body 4 in real time at the time the mobile body 4 performs shooting. Alternatively, the unit may perform a process to acquire images and associated image information based on user instructions at any time during or after the shooting operation by the mobile body 4.

[0061] The image sorting unit 102 identifies the positional relationship between a representative point and the shooting position based on the supplementary information of the image acquired by the image data acquisition unit 101 and the position coordinates of the representative point stored in the managed object information storage unit 121, and performs a process to identify the object to be photographed from among the managed objects based on this positional relationship. Here, "positional relationship" refers to information that indicates the relative spatial relationship between the representative point and the shooting position, and is mainly expressed as the distance between the representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction. In addition to distance and angle, the positional relationship may also include quadrant information indicating which quadrant the representative point is located in in a coordinate system with the shooting position as the origin, visibility information indicating whether there is an obstacle between the representative point and the shooting position, and field of view information indicating whether the representative point is within the shooting angle of view. Based on such positional relationships, the image sorting unit 102 identifies the managed object that is most likely to be captured in the photographed image as the object to be photographed.

[0062] Specifically, the image sorting unit 102 may calculate the distance between a representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction, for each managed object, and identify the managed object with the smallest distance and / or angle as the target object for shooting. If multiple representative points are registered for a single managed object, the image sorting unit 102 may calculate, for each representative point of that single managed object, the positional relationship between the representative point and the shooting position, that is, the distance between the representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction. Figures 6 and 7 are conceptual diagrams illustrating the process by which the image sorting unit 102 identifies targets for shooting based on the positional relationship between a representative point and the shooting position. Figure 6 shows an example of identifying the positional relationship in a two-dimensional coordinate system, and Figure 7 shows an example of identifying the positional relationship in a three-dimensional coordinate system.

[0063] In the examples shown in Figures 6 and 7, the mobile unit 4 patrols the plant facility where equipment A-H are located as managed objects, and sequentially takes photographs along the movement path shown by the dotted line. For example, when identifying the object to be photographed from an image acquired at shooting position I, the image sorting unit 102 calculates the distance between the representative point of each piece of equipment A-H and shooting position I, and also calculates the angle of the straight line connecting the representative point and shooting position I with respect to the shooting direction I. At shooting position I in the example shown in Figure 6, both distance and angle are evaluated, and equipment D, which has the minimum distance between the representative point and shooting position and the minimum angle with respect to the shooting direction, is identified as the object to be photographed.

[0064] On the other hand, at shooting position II, although equipment G is closer to shooting position II than equipment F, equipment F is identified as the object to be photographed because it has the smallest angle with respect to shooting direction II. In this way, the object to be photographed may be identified by prioritizing the angle with respect to the shooting direction over distance (giving more weight to distance than angle), or conversely, the object to be photographed may be identified by prioritizing distance over the angle with respect to the shooting direction (giving more weight to distance than angle). The object to be photographed may be identified by referring to only one of the distance or angle, but it is preferable to identify the object to be photographed by considering both distance and angle.

[0065] In Figures 6 and 7, the representative points of each piece of equipment being managed are indicated by black circles. In the case of a two-dimensional coordinate system like that in Figure 6, the coordinates of the center of the equipment's base may be used as the representative point, while in the case of a three-dimensional coordinate system like that in Figure 7, the coordinates of the equipment's three-dimensional center may be used as the representative point. The representative point is not necessarily limited to the center of the equipment; it may be the centroid of the equipment, the center coordinates of multiple points designated within the equipment, or a specific part that is important for management purposes may be designated as the representative point.

[0066] Furthermore, in Figures 6 and 7, the straight line (dotted line) connecting the shooting position and the representative point indicates the direction from the shooting position to the representative point. Also, in Figures 6 and 7, the shooting direction is indicated by a thick black arrow. The angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction may be calculated, for example, as the angle difference between the direction from the shooting position to the representative point and the direction indicated by the shooting direction, if the shooting direction is expressed in terms of bearing. The method for calculating the angle with respect to the shooting direction is not limited to this, and an appropriate calculation method should be adopted depending on the data format of the shooting direction.

[0067] The method for calculating the distance between a representative point and the shooting location is not necessarily limited, and an appropriate distance calculation method is used depending on the data format of the position coordinates of the representative point and the shooting location. For example, if the representative point and the shooting location are represented in a geographic coordinate system including latitude and longitude, the distance between each representative point and the shooting location may be calculated using a method for calculating the distance between two points, such as the Haversine formula. If the representative point and the shooting location are represented in a three-axis orthogonal coordinate system with x, y, and z axes, the distance between each representative point and the shooting location may be calculated using a distance calculation formula in three-dimensional space (Euclidean distance) that uses the difference of each axis component.

[0068] When identifying objects to be photographed based on the distance and / or angle described above, for example, one may first extract (narrow down) the objects to be managed based on either the distance or the angle, and then identify a single object to be managed based on the other feature (either the distance or the angle).

[0069] For example, Figure 8(a) illustrates a pattern in which the objects to be managed are narrowed down by distance, and then the objects to be photographed are identified based on angle. Specifically, as shown in Figure 8(a), the image sorting unit 102 may extract objects to be managed in which the distance between the representative point and the shooting position is within a predetermined range, and then identify the object to be photographed from among these objects in which the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction is smallest. In Figure 8(a), the range of a predetermined distance (threshold) centered on the shooting position is shown by a dashed circle, and the objects to be managed (equipment A, equipment B, equipment C, equipment D, equipment F, equipment G) whose representative point is located within this circle are to be extracted. From among these extracted objects to be managed, equipment F, which has the smallest angle with respect to the shooting direction, is identified as the object to be photographed.

[0070] In addition, the above describes the case where a fixed threshold is used as the predetermined distance range. However, as a process for extracting managed objects whose distance is within the predetermined range, a method of extracting a predetermined number of managed objects in a ranking format in order of increasing distance may also be adopted. For example, the top three managed objects in order of proximity between the representative point and the shooting position may be extracted, and from among them, the managed object with the smallest angle to the shooting direction may be identified as the target object. Furthermore, when each image is taken with a fixed shooting direction, such as shooting from above with the shooting direction pointed directly below the aircraft (i.e., approximately perpendicular to the ground), the managed object to be photographed may be identified based on the distance between the representative point and the shooting position without referring to the shooting direction (azimuth) (for example, the managed object with the smallest distance between the representative point and the shooting position may be identified as the target object without referring to azimuth).

[0071] Figure 8(b) illustrates a pattern in which the objects to be managed are narrowed down by their angle to the shooting direction, and then the objects to be photographed are identified based on the distance between a representative point and the shooting position. Specifically, as shown in Figure 8(b), the image sorting unit 102 may extract the objects to be managed whose representative point is located within a predetermined range based on the shooting direction, and then identify the object among these objects whose distance from the representative point to the shooting position is the smallest as the object to be photographed. In Figure 8(b), a predetermined angular range with the shooting direction as the axis is shown by a dashed line, and the objects to be managed (equipment A, equipment B, equipment C) whose representative point is located within this angular range are to be extracted. From among these extracted objects to be managed, equipment C, which has the smallest distance from the shooting position, is identified as the object to be photographed.

[0072] In the example shown in Figure 8(b), in addition to equipment C, which is identified as the object to be photographed, equipment B may also be identified as a secondary object. Specifically, the image sorting unit 102 may identify the object to be managed (equipment C) with the shortest distance as the main object to be photographed, and then identify equipment B, which is located within the field of view and for which no other equipment is interposed between the shooting position and the representative point, as a secondary object.

[0073] The process of extracting target objects whose representative point is located within a predetermined range based on the shooting direction is not limited to the method described above. A method of extracting a predetermined number of target objects in a ranking format, in order of increasing angle relative to the shooting direction, may also be employed. For example, the top three target objects in order of decreasing angle may be extracted, and from among them, the target object whose distance between the representative point and the shooting position is the smallest may be identified as the target object.

[0074] Furthermore, when defining a predetermined range based on the shooting direction, if the image attachment information includes information indicating the field of view of the image, a predetermined angular range (angle threshold) with the shooting direction as the axis may be set based on that field of view. In this case, the objects to be managed whose representative point is located within the field of view with the shooting direction as the axis will be extracted. Note that if field of view information can be obtained, the objects to be managed may be extracted based solely on the field of view without referring to the shooting direction. For example, objects to be managed whose orientation from the shooting position toward the representative point is within the field of view with the shooting position as the origin may be extracted, and from among the extracted objects to be managed, the object with the smallest distance between the representative point and the shooting position may be identified as the object to be photographed.

[0075] When the image data acquisition unit 101 acquires positional information (feature point coordinates) of feature points measured during shooting, the image sorting unit 102 may identify the object captured in the image by identifying the positional relationship between the representative point and the feature point, instead of the positional relationship between the representative point and the shooting position, or by identifying the positional relationship between the representative point and the feature point in addition to the positional relationship between the representative point and the shooting position. The positional relationship between the representative point and the feature point is also identified by calculating the distance between the representative point and the feature point, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction (the angle difference between the direction indicated by the shooting direction and the direction from the feature point to the representative point), similar to the positional relationship between the representative point and the shooting position described above. In Figures 6 and 7, feature points measured during image shooting at shooting position II are indicated by stars. Even when comparing the positional relationship with the feature point for each managed object, equipment F, which has the smallest distance from the feature point to the representative point, is identified as the object to be photographed. The angle with respect to the shooting direction is also smallest for equipment F, so equipment F is identified as the object to be photographed.

[0076] As described above, instead of identifying the object based on the positional relationship between the shooting position and the representative point, the object may be identified based on the positional relationship between the feature point and the representative point. Alternatively, the positional relationship between the feature point and the representative point may be used to verify the accuracy of the object identification process using the positional relationship between the shooting position and the representative point. For example, if the object identified based on the positional relationship between the shooting position and the representative point matches the object identified based on the positional relationship between the feature point and the representative point, the accuracy of the processing result can be determined to be high.

[0077] Furthermore, if there are multiple objects that are closest to the shooting position and the representative point, and / or if there are multiple objects that are closest to the shooting position and the angle of the straight line connecting the representative point and the shooting position relative to the shooting direction, the object to be photographed may be selected from among these objects based on the positional relationship between the characteristic point and the representative point. In other words, candidate objects to be photographed may be identified based on the positional relationship between the shooting position and the representative point, and then one object to be photographed may be determined based on the positional relationship between the characteristic point and the representative point; or conversely, candidate objects to be photographed may be identified based on the positional relationship between the characteristic point and the representative point, and then one object to be photographed may be determined based on the positional relationship between the shooting position and the representative point.

[0078] The positional relationship between a representative point and a feature point may be represented in a two-dimensional coordinate system or a three-dimensional coordinate system, depending on the data format of the representative point's position coordinates and / or the data format of the shooting condition information (shooting position and / or shooting direction). For example, if the representative point's position coordinates are represented in a two-dimensional coordinate system including a first axis component (latitude) and a second axis component (longitude) (when height information for the representative point is not registered), the image sorting unit 102 uses the data corresponding to the first axis component (latitude) and the second axis component (longitude) from the data indicating the shooting position and shooting direction to determine the two-dimensional positional relationship between the representative point and the shooting position. In other words, as shown in Figure 6, the distance and angle on the horizontal plane are calculated by ignoring the height components of the shooting position and the representative point.

[0079] On the other hand, if the position coordinates of the representative point are expressed in a three-dimensional coordinate system, the image sorting unit 102 uses the shooting position and shooting direction expressed in the three-dimensional coordinate system to determine the three-dimensional positional relationship between the representative point and the shooting position. That is, as shown in Figure 7, the distance and angle in three-dimensional space are calculated by also considering the height components of the shooting position and the representative point. Even if the position coordinates of the representative point are registered in a three-dimensional coordinate system, if the shooting position and / or shooting direction indicated in the image supplementary information are expressed in a two-dimensional coordinate system, the positional relationship between the shooting position and the representative point is calculated by ignoring the height component and using the first axis component (latitude) and second axis component (longitude) of the representative point. The positional relationship between the representative point and feature points is the same as the positional relationship between the shooting position and the representative point, and may be expressed in a two-dimensional coordinate system or a three-dimensional coordinate system depending on the data representation format used during calculation.

[0080] The image sorting unit 102 may automatically switch between a mode for identifying a two-dimensional positional relationship (hereinafter referred to as the two-dimensional mode) and a mode for identifying a three-dimensional positional relationship (hereinafter referred to as the three-dimensional mode) depending on the data format of the position coordinates of the representative point when determining the positional relationship between the representative point and the shooting position. For example, if the position coordinate data of the representative point stored in the managed information storage unit 121 contains both representative points represented in a two-dimensional coordinate system and representative points represented in a three-dimensional coordinate system, the image sorting unit 102 may switch between the two-dimensional mode and the three-dimensional mode for each managed object.

[0081] The image sorting unit 102's process for identifying the object to be photographed may be configured to handle cases where results cannot be obtained in a single process. For example, if, as a result of performing the object identification process as described above, the evaluation values ​​of the positional relationships of multiple managed objects (a comprehensive evaluation value based on distance, angle, or a combination thereof) are the same, and the object to be photographed cannot be uniquely identified, the image sorting unit 102 may perform further narrowing-down processing based on predetermined auxiliary conditions (for example, the positional relationship between a representative point and a feature point). Alternatively, if the object to be photographed cannot be uniquely identified, all possible candidates may be associated with the image as candidates for the object to be photographed, and the user may be asked to confirm.

[0082] The image sorting unit 102 performs the above-described object identification process for each image acquired by the image data acquisition unit 101. Then, it associates each image with the object identified in the above process and registers it in the captured image data storage unit 122. As for the method of association, for example, a method may be adopted in which the identification information of the managed object identified as the object to be photographed is recorded as supplementary information to the image and the image is registered. Alternatively, a method may be adopted in which the identification information of the managed object is included in the image file name, or a method may be adopted in which the image and the managed object are associated using a database table structure without changing the image file itself. The method of associating images with objects to be photographed is not particularly limited. By associating and registering images with the identified objects to be photographed in this way, the image sorting unit 102 sorts the multiple images acquired from the target space according to the managed object.

[0083] The data output unit 103 reads a predetermined captured image stored in the captured image data storage unit 122 based on a user request and outputs it to the display screen of the user terminal 2. For example, the data output unit 103 may output the results of the image data sorting process performed by the image sorting unit 102 to the user terminal 2. Specifically, it displays a list of images sorted by managed object on the user terminal 2. The list data may include image-related information such as the shooting location and shooting direction for each image, as well as information indicating the managed object and sub-objects identified as the target object, and the user may be allowed to modify the sorting results using this list data. The data output unit may also receive image output requests from the user via the user terminal 2 and output captured images corresponding to those requests. When receiving an image output request, the data output unit 103 may accept the specification of output conditions such as the shooting date and time, the specification of the managed object, and the specification of the area within the target space, and identify a captured image corresponding to the specified output conditions by referring to the accompanying information of each captured image data. If multiple captured images satisfy the output conditions specified by the user, the data output unit 103 may output a list of the multiple captured images that satisfy the output conditions (image list). When the data output unit 103 displays the captured images (single image or image list) specified by the user on the user terminal 2, it may display the captured image and information indicating the object to be photographed, as identified by the image sorting unit 102, side by side.

[0084] In addition to the image search process described above, the data output unit 103 may also perform a filtering process to narrow down the list of images to those that meet the output conditions specified by the user, or a sorting process to change the order of the images based on sorting conditions specified by the user and output a list. In this case as well, the data output unit 103 accepts the user's request for conditions such as the date and time of shooting, the object to be managed, and the area within the target space, as filtering conditions (narrowing conditions) or sorting conditions.

[0085] The conditions for outputting captured images (search conditions, filtering conditions, sorting conditions) may be entered in text format or via a predetermined selection form such as a pull-down list. The data output unit 103 may also perform a process to output a report related to the specified image by applying the image specified by the user, along with supplementary information of the image, information indicating inspection results, etc., to a predetermined format. The functions of the data output unit 103 as described above may also be executed by the processor 20 of the user terminal 2. The processor 10 may also have other functional units not illustrated in Figure 5. For example, the processor 10 may have a functional unit that controls the movement of a mobile body 4 for the purpose of capturing actual images of various objects inside and outside a structure, based on various movement information stored in the movement information storage unit 123.

[0086] <An Example of Information Processing Method> Next, with reference to Figure 9, an information processing method (an information processing method related to the management of captured image data) by the information processing system according to this embodiment will be described. Figure 9 is a flowchart illustrating an example of the information processing method according to this embodiment.

[0087] First, in step SQ101, image data including supplementary information (shooting position, shooting direction) is acquired. Specifically, an image taken in real space and image supplementary information including data indicating the shooting position and shooting direction of the image are acquired. This process is performed at any time by the aforementioned image data acquisition unit 101. Next, in step SQ102, the position coordinates of a representative point of each managed object are read. Specifically, the position coordinates of at least one representative point of each managed object stored in the managed object information storage unit 121 are read.

[0088] Then, in step SQ103, the positional relationship between the representative point of each managed object and the shooting position is determined using the image-related information and the position coordinates of the representative point. Specifically, the distance between the representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction are calculated.

[0089] Subsequently, in step SQ104, the object to be photographed is identified from among the objects to be managed based on the positional relationship between the representative point and the shooting position. For example, the object to be managed that is the shortest distance from the representative point and the shooting position is selected, or the object to be managed that is the shortest angle with respect to the shooting direction is selected, or the object to be photographed is identified by considering both distance and angle. Alternatively, it is possible to narrow down the objects to be managed based on distance and then identify the final object to be photographed based on angle, or vice versa. If positional information of feature points is obtained when the image is taken, the positional relationship between the representative point and the feature points may be calculated and the object to be photographed may be identified using the positional relationship between the representative point and the feature points instead of the positional relationship between the representative point and the shooting position, or using the positional relationship between the representative point and the feature points together with the positional relationship between the representative point and the shooting position.

[0090] Finally, in step SQ105, the image is recorded in association with the target object identified as the object to be photographed. The processing in steps SQ103-SQ105 is performed for each image acquired by the image data acquisition unit 101 in SQ101. Through these processes, the images photographed in real space are sorted by target object and stored in the photographed image data storage unit 122.

[0091] As described above, the information processing system according to this embodiment uses the position coordinates of a representative point stored for each managed object in real space, and image-related information including data indicating the image shooting position and direction, to identify the positional relationship between the representative point of each managed object and the shooting position. Based on this positional relationship, the system identifies the object captured in the image from among the managed objects. This makes it possible to sort images acquired from real space into managed objects in a general and simple manner without using image analysis techniques. Furthermore, even when there are multiple managed objects with similar appearances, the object captured can be accurately identified based on the positional information and shooting direction.

[0092] The embodiments described above are merely illustrative to facilitate understanding of this disclosure and are not intended to limit it. This disclosure may be modified and improved without departing from its intent, and its equivalents are included.

[0093] For example, in the above embodiment, the information processing system related to the management of captured image data was described mainly using an example of inspecting facilities such as large-scale plants using an unmanned mobile device such as a drone. However, the application examples of the information processing system are not limited to the examples in the embodiment. The information processing system disclosed herein can be applied to various purposes such as maintenance, facility management, construction management, and on-site inspection before construction. The objects of inspection may be social infrastructure facilities such as buildings, bridges, tunnels, dams, roads, power facilities, and communication facilities, as well as industrial facilities, natural environments such as farmland and forests, and other structures such as buildings. Photography during inspection may be performed using portable devices such as smartphones, tablet terminals, and other cameras held by people, or it may be performed using a manned mobile device. In addition to inspection, the information processing system disclosed herein may also be applied to patrols, security, and on-site inspections during disasters.

[0094] Furthermore, although the above embodiment describes an example in which one object to be photographed is identified for one image, if multiple objects to be photographed are included in one image, multiple objects to be managed may be identified as objects to be photographed. For example, multiple objects to be managed whose positional relationship evaluation value is within a predetermined threshold may all be identified as objects to be photographed and associated with the image. In addition, the image data sorted by the information processing system of the present invention can be used for various purposes. For example, it can be used as an inspection record for each object to be managed, to analyze the deterioration status by comparing images before and after inspection, or to compare the condition of similar objects to be managed, thereby contributing to the efficiency and sophistication of facility management.

[0095] Furthermore, the mobile unit 4 used for imaging may also be equipped with devices, equipment, etc., used to inspect the presence or absence of predetermined events inside and / or outside the object being managed, in addition to the cameras / sensors 42 described above. More specifically, all devices necessary to know the condition of the structure to be inspected can be employed, such as imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring instruments, etc.), keying devices, detection devices (metal detectors), sound collection devices, odor measuring instruments, gas detectors, air pollution measuring instruments, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.). The information indicating the object to be photographed, identified by the image sorting unit 102, may be linked to inspection data acquired by the other inspection devices described above, similar to how it is linked to the captured images.

[0096] Furthermore, the purpose of photography in the information processing system is not limited to inspection, but may also include security, infrastructure monitoring, surveying, disaster response, etc., and the information processing system disclosed herein may be applied to image data acquired through photography for these purposes.

[0097] Some or all of the functions of the image data acquisition unit 101, image sorting unit 102, data output unit 103, etc., described in the above embodiment may be executed by the processor 20 of the user terminal 2 or the flight controller 41 of the mobile unit 4, rather than by the processor 10 of the management server 1.

[0098] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0099] 1. Management server 2. User terminal 4. Mobile device

Claims

1. An information processing system comprising: a management information storage unit that stores the position coordinates of at least one representative point of each management object existing in real space; an image data acquisition unit that acquires an image taken in real space and image supplementary information including data indicating the shooting position and direction of the image; and an image sorting unit that identifies the positional relationship between the representative point and the shooting position based on the image supplementary information and the position coordinates of the representative point, and identifies the object captured in the image from among the management objects based on the positional relationship.

2. The information processing system according to claim 1, wherein the image sorting unit calculates the distance between a representative point and the shooting position, and / or the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction, for each of the objects to be managed, and identifies the object to be managed with the smallest distance and / or angle as the object to be photographed.

3. The information processing system according to claim 1 or 2, wherein the image sorting unit extracts objects to be managed in which the distance between a representative point and the shooting position is within a predetermined range, and identifies the object to be photographed among the objects to be managed in which the angle of the straight line connecting the representative point and the shooting position with respect to the shooting direction is smallest.

4. The information processing system according to claim 1 or 2, wherein the image sorting unit extracts objects to be managed whose representative point is located within a predetermined range based on the shooting direction, and identifies the object to be photographed from among the objects to be managed whose distance from the representative point to the shooting position is the smallest.

5. The information processing system according to claim 1 or 2, wherein the image data acquisition unit acquires positional information of feature points measured at the time of image capture along with the image, and the image sorting unit identifies the object captured in the image by specifying the positional relationship between the representative point and the shooting position, or by specifying the positional relationship between the representative point and the feature point along with the positional relationship between the representative point and the shooting position.

6. The position coordinates of the representative point are represented in a two-dimensional coordinate system including a first axis component and a second axis component, and the image sorting unit uses data corresponding to the first axis component and the second axis component from the data indicating the shooting position and the shooting direction to determine the two-dimensional positional relationship between the representative point and the shooting position, according to claim 1 or 2.

7. The information processing system according to claim 1 or 2, wherein the position coordinates of the representative point are represented in a three-dimensional coordinate system, and the image sorting unit uses the shooting position and shooting direction, which are represented in a three-dimensional coordinate system, to determine the three-dimensional positional relationship between the representative point and the shooting position.

8. The information processing system according to claim 1 or 2, wherein the image sorting unit automatically switches between a mode for identifying a two-dimensional positional relationship and a mode for identifying a three-dimensional positional relationship, depending on the data format of the position coordinates of the representative point, when determining the positional relationship between the representative point and the shooting position.

9. An information processing method in which a computer performs the following steps: storing the position coordinates of at least one representative point of each managed object that exists in real space; acquiring an image taken in the real space and image-related information including data indicating the shooting position and direction of the image; and identifying the positional relationship between the representative point and the shooting position based on the image-related information and the position coordinates of the representative point, and identifying the object captured in the image from among the managed objects based on that positional relationship.

10. A program for causing a computer to perform the following actions: store the position coordinates of at least one representative point of each managed object that exists in real space; acquire an image taken in the real space and image-related information including data indicating the shooting position and direction of the image; and identify the positional relationship between the representative point and the shooting position based on the image-related information and the position coordinates of the representative point, and identify the object captured in the image from among the managed objects based on that positional relationship.

Images