Information processing system, information processing method, and information processing program

Abstract

This information processing system (10) comprises at least one processor. The at least one processor derives a three-dimensional position of a second imaging device (22) on the basis of: three-dimensional map data (42) on a real space (target space TS) generated by VSLAM-processing first captured image data obtained by imaging the real space by means of a movable first imaging device (20); and second captured image data (50) on the real space (the target space TS) obtained by imaging the real space by means of a second imaging device fixedly disposed in the real space (the target space TS).

Description

Information Processing System, Information Processing Method, and Information Processing Program

[0001] Embodiments of the present invention relate to an information processing system, an information processing method, and an information processing program.

[0002] The three-dimensional position of an object such as a person included in the captured image data captured by a capturing device fixedly arranged in the real space is managed. At this time, it is necessary to manage the position of the capturing device fixedly arranged in the real space. Furthermore, in order to derive the three-dimensional position of the object, it is necessary to manage the three-dimensional position and orientation of the capturing device with high precision.

[0003] However, in the prior art, the three-dimensional position and orientation of a fixedly arranged capturing device are manually managed, and it may be difficult to accurately derive the three-dimensional position of the fixedly arranged capturing device.

[0004] International Publication No. 2021 / 200432

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing system, an information processing method, and an information processing program capable of accurately deriving the three-dimensional position of a fixedly arranged capturing device.

[0006] The information processing system according to the embodiment includes at least one processor, and the at least one processor is based on the three-dimensional map data of the real space generated by the VSLAM process of the first captured image data of the real space captured by a movable first capturing device, and the second captured image data of the real space captured by a second capturing device fixedly arranged in the real space, to derive the three-dimensional position of the second capturing device.

[0007] Figure 1 is an explanatory diagram of the system of the embodiment. Figure 2 is a schematic diagram of 3D map data. Figure 3A is an explanatory diagram of an example of the 3D position and orientation of the second imaging device. Figure 3B is an explanatory diagram of an example of the 3D position and orientation of the second imaging device. Figure 4 is a schematic diagram of the data structure of the position and orientation management DB. Figure 5 is a schematic diagram of the data structure of the specification management DB. Figure 6 is an explanatory diagram of the formula for calculating coordinate transformation parameters. Figure 7 is a schematic diagram of the data structure of the parameter management DB. Figure 8 is a sequence diagram showing the flow of information processing performed by the system. Figure 9 is a flowchart showing the flow of information processing performed by the information processing system. Figure 10 is a flowchart showing the flow of information processing performed by the information processing system. Figure 11 is an explanatory diagram of the system of the embodiment. Figure 12 is a hardware configuration diagram.

[0008] The embodiments of the information processing system, information processing method, and information processing program will be described in detail below with reference to the attached drawings.

[0009] (First Embodiment) Figure 1 is an explanatory diagram of an example of System 1 of this embodiment.

[0010] System 1 is a system for performing processes such as estimating the three-dimensional position and orientation of a second imaging device 22 fixedly positioned in the target space TS, and estimating the three-dimensional position of an object T captured in the second image data captured by the second imaging device 22.

[0011] The target space TS is the real space to be managed in System 1. The target space TS can be any predetermined space in the real world. For example, the target space TS may be a space inside a specific building, a predetermined space outdoors, etc. In this embodiment, the form in which the target space TS is a space inside a predetermined building will be described as an example.

[0012] The second imaging device 22 is an imaging device fixedly positioned in the target space TS. The imaging device is a known imaging device having an imaging function that acquires image data by imaging. Fixed positioning means that it is positioned without being moved for at least a predetermined time. In this embodiment, the second imaging device 22 is described as being fixedly positioned by being installed on a structure that does not move, such as a wall, in the target space TS. The second imaging device 22 acquires second image data by imaging. The second image data is an example of image data.

[0013] One or more second imaging devices 22 are fixedly arranged in the target space TS. In this embodiment, a configuration in which multiple second imaging devices 22 are fixedly arranged in the target space TS will be described as an example.

[0014] The target space TS includes a first imaging device 20. The first imaging device 20 is a movable imaging device. For example, the first imaging device 20 is held by a movable object such as a user and moves in conjunction with the movement of the movable object. The first imaging device 20 acquires first image data by imaging. The first image data is an example of image data. Details of the target T will be described later.

[0015] System 1 comprises an information processing system 10, an external system 30, a first imaging device 20, and a second imaging device 22.

[0016] The information processing system 10, the external system 30, the first imaging device 20, and the second imaging device 22 are connected wirelessly or via wired communication through a network NW or the like.

[0017] The external system 30 is one or more information processing devices located outside the information processing system 10. The external system 30 consists of one or more dedicated or general-purpose computers. Examples of the external system 30 include a cloud server, a mobile terminal, etc.

[0018] Next, the information processing system 10 will be described.

[0019] The information processing system 10 is one or more information processing devices. The information processing system 10 is composed of one or more dedicated or general-purpose computers.

[0020] The information processing system 10 comprises a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15. The communication unit 11, input unit 12, output unit 13, and storage unit 14 and the control unit 15 are communicated to each other via a bus or the like. At least one of the external system 30, the output unit 13, and the storage unit 14 functions as an output device 17, which is the output destination for various types of information from the control unit 15.

[0021] The information processing system 10 may further include at least one of a first imaging device 20 and a second imaging device 22. These first imaging device 20 and second imaging device 22 should be capable of imaging the target space TS and should be connected to the control unit 15 of the information processing system 10 via a network NW or the like.

[0022] The communication unit 11 communicates with an external system 30, a first imaging device 20, a second imaging device 22, etc., via a network NW or the like. The input unit 12 accepts various operations from the user. The output unit 13 outputs various types of information. The output unit 13 includes a display for showing various types of information, a speaker for outputting various types of sound, etc. At least one of the input unit 12 and the output unit 13 may be configured to be located outside the information processing system 10 and to be connected to the control unit 15 in a manner that allows communication.

[0023] The storage unit 14 stores various types of data. The storage unit 24 is, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disc. The storage unit 14 may also be a storage device located outside the information processing system 10. Furthermore, the storage unit 14 may be a storage medium on which programs and various types of information are downloaded and stored or temporarily stored via a LAN (Local Area Network) or the Internet.

[0024] In this embodiment, the storage unit 14 stores a position and orientation management DB (database) 14A, a specification management DB 14B, a parameter management DB 14C, two-dimensional map data 40, and three-dimensional map data 42, etc. Details of this data will be described later.

[0025] The control unit 15 performs information processing in the information processing system 10.

[0026] The control unit 15 includes a 3D map data generation unit 15A, an acquisition unit 15B, a 3D position and orientation derivation unit 15C, a parameter derivation unit 15D, a determination unit 15E, an output control unit 15F, a target 2D position derivation unit 15G, a conversion unit 15H, and a correction unit 15I.

[0027] The 3D map data generation unit 15A, acquisition unit 15B, 3D position and orientation derivation unit 15C, parameter derivation unit 15D, determination unit 15E, output control unit 15F, target 2D position derivation unit 15G, conversion unit 15H, and correction unit 15I are implemented by one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be implemented by at least one processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0028] Alternatively, at least one of the above-mentioned components included in the control unit 15 may be mounted on an external information processing device such as an external system 30 that is connected to the information processing system 10 via a network NW or the like.

[0029] The 3D map data generation unit 15A generates 3D map data of the target space TS, which is a real space.

[0030] The 3D map data generation unit 15A acquires first image data of the target space TS captured by the movable first imaging device 20. For example, a moving object such as a user moves within the target space TS while holding the first imaging device 20 and continuing to capture images with the first imaging device 20. The first image data captured by the first imaging device 20 is sequentially transmitted to the information processing system 10. The 3D map data generation unit 15A of the information processing system 10 sequentially acquires the first image data captured by the first imaging device 20. The 3D map data generation unit 15A may also acquire the first image data by reading the first image data captured by the first imaging device 20 and stored in the storage unit 14.

[0031] The 3D map data generation unit 15A generates 3D map data 42 by performing a known VSLAM (Visual Simultaneous Localization and Mapping) process on the first captured image data captured by the first imaging device 20.

[0032] The 3D map data 42 is information registered in a three-dimensional coordinate space which is the world coordinate system space of the target space TS, including the feature quantities of each feature point included in the target space TS, the 3D position of the feature point, the first image data captured at each position in the target space TS, the position and orientation of the first imaging device 20 at the time the first image data was captured, and so on.

[0033] For example, the 3D map data generation unit 15A performs feature extraction processing and matching processing between the first captured image data for multiple first captured image data taken at different timings. More specifically, the 3D map data generation unit 15A performs matching processing to identify corresponding feature points between multiple first captured image data using feature quantities between them.

[0034] The 3D map data generation unit 15A uses a plurality of matching feature points to estimate the 3D position and orientation of the first imaging device 20 relative to the first captured image data through projection transformation, etc., and registers it in the corresponding position in the 2D map data 40. The 2D map data 40 is data of a 2D map of the target space TS.

[0035] Three-dimensional position is represented by three-dimensional position coordinates in the target space TS, i.e., the world coordinate system. Orientation is represented by pitch, roll, and yaw.

[0036] The 3D map data generation unit 15A performs perspective projection transformation processing using the amount of movement (translation and rotation) from the estimated 3D position of the first imaging device 20, determines the 3D position of the matched feature points, and registers it in the 2D map data 40.

[0037] Through these processes, the 3D map data generation unit 15A generates 3D map data 42. The 3D map data generation unit 15A may also correct at least one of the feature points and the 3D position and orientation of the first imaging device 20 at the time of shooting, for example, by using the least squares method, so that the sum of the difference in distance in the target space TS between the previously calculated 3D position and the newly calculated 3D position is minimized for feature points that have been matched multiple times between multiple first imaging image data taken at different shooting timings. Alternatively, the 3D map data generation unit 15A may also correct at least one of the feature points and the position and orientation of the first imaging device 20 at the time of shooting by detecting AR (Augmented Reality) markers placed in the target space TS using a known method.

[0038] Figure 2 is a schematic diagram of an example of 3D map data 42. The 3D map data 42 contains the 3D position and feature quantity of each feature point included in the target space TS, the first image data captured at each 3D position, and the 3D position and orientation of the first imaging device 20 at the time the first image data was captured.

[0039] Returning to Figure 1, we continue the explanation.

[0040] The 3D map data generation unit 15A stores the generated 3D map data 42 in the storage unit 14.

[0041] The acquisition unit 15B acquires the second captured image data of the target space TS captured by the second imaging device 22. In the present embodiment, the acquisition unit 15B acquires the identification information of each of the plurality of second imaging devices 22 fixedly arranged in the target space TS and the second captured image data captured by the second imaging device 22. The acquisition unit 15B may acquire the second captured image data by reading the second captured image data captured by the second imaging device 22 and stored in the storage unit 14 from the storage unit 14.

[0042] The 3D position and orientation derivation unit 15C derives the 3D position and orientation of the second imaging device 22 that captured the second captured image data based on the 3D map data 42 of the target space TS and the second captured image data. Note that the 3D position and orientation derivation unit 15C may derive at least the 3D position of the second imaging device 22 that captured the second captured image data based on the 3D map data 42 of the target space TS and the second captured image data. This is because the derivation and recording of the orientation of the second imaging device 22 are not essential when the purpose is to manage the installation position of the second imaging device 22. In the present embodiment, a case where the 3D position and orientation derivation unit 15C derives the 3D position and orientation of the second imaging device 22 that captured the second captured image data will be described as an example.

[0043] As described above, the 3D position is represented by 3D position coordinates in the target space TS, that is, the world coordinate system. The 3D position coincides with the 3D position in the 3D map data 42. The orientation is represented by, for example, pitch, roll, and yaw.

[0044] The 3D position and orientation derivation unit 15C may derive the 3D position and orientation of the second imaging device 22 that captured the second captured image data by a known method using the 3D map data 42 of the target space TS generated by the VSLAM process and the second captured image data of the target space TS.

[0045] For example, the 3D position and orientation derivation unit 15C identifies the feature amounts of feature points included in the second captured image data captured by the second imaging device 22 by a known image analysis method, and creates a histogram of the feature amounts. Then, the 3D position and orientation derivation unit 15C identifies, among the first captured image data included in the 3D map data 42 and associated with each 3D position of the target space TS captured by the first imaging device 20, the first captured image data in which the histogram of the feature amounts of the included feature points is most similar to the second captured image data. The 3D position and orientation derivation unit 15C derives the 3D position and orientation of the first imaging device 20 at the time of capturing the identified first captured image data, which are registered in the 3D map data 42, as the 3D position and orientation of the second imaging device 22 that captured the second captured image data.

[0046] FIGS. 3A and 3B are explanatory diagrams of an example of the 3D position and orientation of the second imaging device 22. FIG. 3A is an explanatory diagram of an example of the target space TS and the 3D map data 42. FIG. 3A shows, as an example, an example in which a plurality of second imaging devices 22, i.e., second imaging devices 22A to 22E, are fixedly arranged in the target space TS.

[0047] Assume a scenario where the 3D position and orientation derivation unit 15C acquires, as the second captured image data captured by the second imaging device 22C, the second captured image data 50C shown in FIG. 3B. The second captured image data 50C is an example of the second captured image data 50 captured by the second imaging device 22.

[0048] The 3D position and orientation derivation unit 15C executes the above processing using the 3D map data 42 and the second captured image data 50C to derive the 3D position and orientation of the second imaging device 22C in the target space TS (3D map data 42).

[0049] As described above, the 3D position and orientation derivation unit 15C derives the 3D position and orientation of the second imaging device 22 using the 3D map data 42 generated by the VSLAM process. Therefore, the 3D position and orientation derivation unit 15C can accurately derive the 3D position and orientation of the second imaging device 22 fixedly arranged in the target space TS.

[0050] Furthermore, the 3D position and orientation derivation unit 15C may further register the position where the height coordinate of the derived 3D position of the second imaging device 22 is set to "0" as the 2D position of the second imaging device 22 in the 2D map data 40. At this time, the 3D position and orientation derivation unit 15C may also register information representing the orientation of the second imaging device 22 in the 2D map data 40, associating it with the 2D position.

[0051] Returning to Figure 1, we continue the explanation.

[0052] The output control unit 15F outputs various types of data to the output device 17. The output control unit 15F outputs the data to the output device 17 by performing at least one of the following processes: storing the various types of data in the storage unit 14, displaying or outputting audio to the output unit 13, and transmitting the data to the external system 30 via the communication unit 11.

[0053] The output control unit 15F stores the three-dimensional position and orientation of the second imaging device 22 derived by the 3D position and orientation derivation unit 15C, along with the identification information of the second imaging device 22, in the position and orientation management DB 14A. The output control unit 15F can identify the identification information of the second imaging device 22 that corresponds to the three-dimensional position and orientation of the stored data by reading the identification information of the second imaging device 22 that captured the second imaging image data 50, which was acquired together with the second imaging image data 50.

[0054] Figure 4 is a schematic diagram of an example of the data structure of the position and orientation management DB14A.

[0055] The position and orientation management DB 14A is a database that associates the identification information (camera ID (identification)) of the second imaging device 22 with the three-dimensional position of the second imaging device 22. The position and orientation management DB 14A only needs to be data that associates this information, and the data format is not limited to a database.

[0056] Each time the 3D position and orientation of the second imaging device 22 are derived by the 3D position and orientation derivation unit 15C, the output control unit 15F stores the 3D position and orientation in the position and orientation management DB 14A, associating them with the identification information of the second imaging device 22. Furthermore, if the 3D position and orientation of the second imaging device 22 to be stored are already registered in the position and orientation management DB 14A as associated with the identification information of the second imaging device 22, the output control unit 15F overwrites the newly derived 3D position and orientation for the second imaging device 22 and stores them in the position and orientation management DB 14A. As a result, the latest derived 3D position and orientation are stored in the position and orientation management DB 14A for each second imaging device 22.

[0057] Returning to Figure 1, we continue the explanation.

[0058] The parameter derivation unit 15D derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 to the world coordinate system, based on the three-dimensional position and orientation of the second imaging device 22 and the optical specification information of the second imaging device 22.

[0059] Optical specification information refers to information representing the optical specifications of the second imaging device 22. Specifically, optical specification information refers to information representing the optical specifications used to derive coordinate transformation parameters such as focal length and optical center. In this embodiment, an example will be described in which the optical specification information represents the focal length and optical center of the second imaging device 22.

[0060] Optical specification information is pre-stored in the specification management DB 14B of the storage unit 14 for each second imaging device 22.

[0061] Figure 5 is a schematic diagram of an example of the data structure of the specification management DB14B.

[0062] The specification management DB14B is a database that associates optical specification information with identification information (camera ID) of the second imaging device 22. The data format of the specification management DB14B is not limited to the database.

[0063] Returning to Figure 1, we continue the explanation.

[0064] The parameter derivation unit 15D reads optical specification information corresponding to the identification information of the second imaging device 22, from which the coordinate transformation parameters are to be derived, from the specification management DB 14B. Through this process, the parameter derivation unit 15D obtains the optical specification information of the second imaging device 22. Then, based on the acquired optical specification information and the three-dimensional position and orientation of the second imaging device 22, the parameter derivation unit 15D derives the coordinate transformation parameters using equation (1) shown in Figure 6.

[0065] Figure 6 is an explanatory diagram of an example of equation (1) for calculating coordinate transformation parameters from the optical specification information of the second imaging device 22 and the three-dimensional position and orientation of the second imaging device 22.

[0066] The parameter derivation unit 15D derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 to the world coordinate system by calculating equation (1).

[0067] The parameter derivation unit 15D performs a derivation process to derive coordinate transformation parameters for converting the camera coordinates of the second captured image data 50, captured by the second shooting device 22, into the world coordinate system in the target space TS and the 3D map data 42.

[0068] In this way, the parameter derivation unit 15D derives the coordinate transformation parameters of the second imaging device 22 using the three-dimensional position and orientation of the second imaging device 22, which are derived with high accuracy using the three-dimensional map data 42 generated by VSLAM processing. Therefore, the parameter derivation unit 15D can derive the coordinate transformation parameters of the second imaging device 22, which is fixedly positioned in the target space TS, with high accuracy.

[0069] Returning to Figure 1, we continue the explanation.

[0070] The output control unit 15F stores the coordinate transformation parameters of the second imaging device 22 derived by the parameter derivation unit 15D and the identification information of the second imaging device 22 in the parameter management DB 14C, associating them with each other.

[0071] Figure 7 is a schematic diagram of an example of the data structure of the parameter management DB14C.

[0072] The parameter management DB 14C is a database that associates the identification information (camera ID) of the second imaging device 22 with the coordinate transformation parameters of the second imaging device 22. The position and orientation management DB 14A only needs to associate this information, and the data format is not limited to a database.

[0073] Each time the 3D position and orientation derivation unit 15C derives coordinate transformation parameters for the second imaging device 22, the output control unit 15F stores the coordinate transformation parameters in the parameter management DB 14C, associating them with the identification information of the second imaging device 22. Furthermore, if the coordinate transformation parameters to be stored are already registered in the parameter management DB 14C in association with the identification information of the second imaging device 22, the output control unit 15F overwrites the newly derived coordinate transformation parameters for the second imaging device 22 and stores them in the parameter management DB 14C. Therefore, the parameter management DB 14C stores the latest derived coordinate transformation parameters for each second imaging device 22, associating them with the data.

[0074] Returning to Figure 1, we continue the explanation.

[0075] The determination unit 15E determines whether at least one of the following has been successful: the derivation of the three-dimensional position and orientation of the second imaging device 22, and the derivation of the coordinate transformation parameters of the second imaging device 22.

[0076] For example, the determination unit 15E detects the target T included in the second captured image data 50 captured by the second imaging device 22 using a known object detection method.

[0077] The target T is the object detected by the information processing system 10. The target T may be, for example, a person, a non-human organism, a non-living thing, etc., but is not limited to these. In this embodiment, the explanation will be based on the assumption that the target T is a person.

[0078] The determination unit 15E then converts the two-dimensional position of the target T in the second captured image data 50 into a three-dimensional position using the coordinate transformation parameters of the second imaging device 22 that captured the second captured image data 50. The determination unit 15E then determines whether the three-dimensional position obtained by this conversion process is a position in the coordinate space predetermined as the target space TS.

[0079] If the determination unit 15E determines that the three-dimensional position obtained by the transformation process is a position within the coordinate space predetermined as the target space TS, it determines that the derivation of the three-dimensional position and orientation of the second imaging device 22, and the derivation of the coordinate transformation parameters of the second imaging device 22, has been successful. If the determination unit 15E determines that the three-dimensional position obtained by the transformation process is a position outside the coordinate space predetermined as the target space TS, it determines that at least one of the derivation of the three-dimensional position and orientation of the second imaging device 22, and the derivation of the coordinate transformation parameters of the second imaging device 22, has failed.

[0080] The output control unit 15F outputs the determination result from the determination unit 15E to the output device 17. The determination result includes, for example, identification information of the second imaging device 22 used for the determination, and information indicating success or failure as a result of the determination process. As described above, at least one of the output unit 13, the storage unit 14, and the external system 30 functions as the output device 17.

[0081] The output control unit 15F outputs the judgment result to the output unit 13. By outputting the judgment result to the output unit 13, the judgment result can be provided to users who can access the output unit 13. The output control unit 15F may also output the judgment result to an external system 30, such as a mobile terminal operated by the person who installed the second imaging device 22 in the target space TS. In this case, the judgment result can be provided to the person who installed it. The output control unit 15F may also store the judgment result in the storage unit 14.

[0082] The target 2D position derivation unit 15G derives the two-dimensional position of the target T on the second image data 50 captured by the second imaging device 22, which is acquired by the acquisition unit 15B. The two-dimensional position on the second image data 50 refers to the position in the two-dimensional second image represented by the second image data 50.

[0083] This will be explained using Figure 3B. The target 2D position derivation unit 15G detects the target T included in the second captured image data 50 using a known image analysis method for detecting objects from image data, and derives the 2D position of the target T in the second captured image data 50.

[0084] Returning to Figure 1, we continue the explanation.

[0085] The conversion unit 15H converts the two-dimensional position of the target T contained in the second captured image data 50, which is derived by the target 2D position derivation unit 15G, into a three-dimensional position represented in the world coordinate system, based on coordinate transformation parameters associated with the identification information of the second imaging device 22 that captured the second captured image data 50.

[0086] This will be explained using Figures 3A and 3B. The conversion unit 15H converts the two-dimensional position of the target T contained in the second captured image data 50C into a three-dimensional position in the target space TS (three-dimensional map data 42) represented by the world coordinate system, using the coordinate transformation parameters of the second imaging device 22C that captured the second captured image data 50C.

[0087] Returning to Figure 1, we continue the explanation.

[0088] The output control unit 15F registers the three-dimensional position of the second imaging device 22, which has been converted by the conversion unit 15H, into the three-dimensional map data 42. More specifically, the output control unit 15F associates the identification information of the target T and the information representing the three-dimensional position of the target T with the region in the three-dimensional map data 42 that corresponds to the three-dimensional position.

[0089] The output control unit 15F may further register a position where the height coordinate of the 3D position of the target T is set to "0" as the 2D position of the target T in the 2D map data 40.

[0090] The correction unit 15I corrects at least one coordinate transformation parameter of each of the multiple second imaging devices 22 so that the three-dimensional positions obtained by transforming the two-dimensional positions on each of the second imaging image data 50 of the same target T, which are captured at the same timing and acquired from each of the multiple second imaging devices 22 fixedly arranged in the target space TS, based on the coordinate transformation parameters corresponding to each of the multiple second imaging devices 22, are the same.

[0091] The correction unit 15I acquires second image data 50 captured at the same time by multiple different second imaging devices 22. The correction unit 15I then identifies coordinate transformation parameters registered in association with the identification information of each of the multiple second imaging devices 22 from the parameter management DB 14C. The correction unit 15I detects the target T included in the multiple acquired second image data 50 in the same manner as the target 2D position derivation unit 15G. The correction unit 15I also identifies the target T among the multiple target T included in each of the multiple second image data 50 whose feature quantities match or are most similar as the same target T among the multiple second image data 50. The correction unit 15I converts the 2D position of the identified same target T included in each of the multiple second image data 50 into a 3D position based on the coordinate transformation parameters corresponding to the second imaging device 22 that captured each second image data 50. The correction unit 15I corrects at least a portion of the coordinate transformation parameters of each of the multiple second imaging devices 22 so that the multiple three-dimensional positions obtained by the transformation are in the same position. The correction unit 15I updates the parameter management DB 14C by overwriting and storing the corrected coordinate transformation parameters in the parameter management DB 14C in association with the identification information of the corresponding second imaging device 22.

[0092] Next, the flow of information processing performed in System 1 of this embodiment will be described.

[0093] Figure 8 is a sequence diagram showing an example of the information processing flow performed in System 1 of this embodiment.

[0094] For example, a moving object such as a user holds the first imaging device 20 and moves within the target space TS while continuing to take images with the first imaging device 20. The first image data captured by the first imaging device 20 is sequentially transmitted to the information processing system 10 (step S100).

[0095] The 3D map data generation unit 15A of the information processing system 10 generates 3D map data 42 by performing VSLAM processing on the first captured image data captured by the first shooting device 20 (step S102). The 3D map data generation unit 15A stores the generated 3D map data 42 in the storage unit 14 (step S104).

[0096] Meanwhile, one or more second imaging devices 22 are physically and permanently positioned within the target space TS by the user (step S106). The second imaging devices 22 transmit the second image data 50, which captures the target space TS, to the information processing system 10 (step S108).

[0097] The 3D position and orientation derivation unit 15C of the information processing system 10 derives the 3D position and orientation of the second imaging device 22 that captured the second imaging image data 50, based on the 3D map data 42 of the target space TS generated in step S102 and the second imaging image data 50 acquired in step S108 (step S110).

[0098] The parameter derivation unit 15D of the information processing system 10 derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 to the world coordinate system, based on the three-dimensional position and orientation of the second imaging device 22 derived in step S110 and the optical specification information of the second imaging device 22 (step S112).

[0099] The output control unit 15F stores the three-dimensional position and orientation of the second imaging device 22 derived in step S110, along with the identification information of the second imaging device 22, in the position and orientation management DB 14A (step S114). The output control unit 15F also stores the coordinate transformation parameters of the second imaging device 22 derived in step S112, along with the identification information of the second imaging device 22, in the parameter management DB 14C (step S114).

[0100] The determination unit 15E determines whether at least one of the following has been successful: the derivation of the three-dimensional position and orientation of the second imaging device 22, and the derivation of the coordinate transformation parameters of the second imaging device 22 (step S116).

[0101] The output control unit 15F outputs the determination result from the determination unit 15E to an output device 17 such as an external system 30 (step S118).

[0102] The information processing system 10 can perform the processing in steps S108 to S118 for each of the multiple second imaging devices 22 installed in the target space TS.

[0103] On the other hand, when detecting the position of target T, the target 2D position derivation unit 15G of the information processing system 10 acquires the second captured image data 50 captured by the second imaging device 22 (step S120). The target 2D position derivation unit 15G derives the two-dimensional position of target T on the second captured image data 50 that was acquired in step S120 (step S122).

[0104] The conversion unit 15H of the information processing system 10 converts the two-dimensional position of the target T in the second captured image data 50 acquired in step S120 into a three-dimensional position in the target space TS (three-dimensional map data 42) represented by the world coordinate system, using the coordinate transformation parameters of the second imaging device 22 that captured the second captured image data 50C (step S124).

[0105] The output control unit 15F of the information processing system 10 registers the three-dimensional position of the second imaging device 22, which has been converted by the conversion unit 15H, into the three-dimensional map data 42 (step S126).

[0106] The correction unit 15I corrects at least one coordinate transformation parameter of the multiple second imaging devices 22 so that the three-dimensional positions obtained by transforming the two-dimensional positions on each of the second imaging image data 50 of the same target T, which are captured at the same timing and acquired from each of the multiple second imaging devices 22 fixedly arranged in the target space TS, based on the coordinate transformation parameters corresponding to each second imaging device 22, are the same (step S128).

[0107] And with that, this sequence ends.

[0108] Figure 8 illustrates an example configuration in which the second imaging device 22 is fixedly positioned in the target space TS after the three-dimensional map data 42 of the target space TS has been generated. However, the timing of the positioning of the second imaging device 22 is not limited to this timing.

[0109] For example, the second imaging device 22 may be fixedly positioned in the target space TS before the 3D map data generation process 42 is performed by the 3D map data generation unit 15A.

[0110] Furthermore, the timing of the execution of the 3D map data generation process by the 3D map data generation unit 15A is not limited to the timing shown in Figure 8. For example, the 3D map data generation unit 15A may generate 3D map data 42 each time a predetermined first condition is met. The first condition is, for example, the passage of a predetermined time, the receipt of a creation instruction by operating the user's input unit 12, the placement of a new second imaging device 22 in the target space TS, etc.

[0111] In other words, the information processing system 10 may repeatedly execute the processes of steps S100 to S104 and steps S108 to S118 each time the first predetermined condition is met.

[0112] Furthermore, the information processing system 10 may repeatedly perform the processing of steps S108 to S118 for each of the multiple first imaging devices 20 arranged in the target space TS, whenever a predetermined second condition is met. The second condition is the elapsed time, receipt of an execution instruction by operation of the user's input unit 12, etc.

[0113] Furthermore, the information processing system 10 may execute at least a portion of the processes in steps S108 to S118, which derive the three-dimensional position and orientation of the second imaging device 22 and the coordinate transformation parameters, and at least a portion of the processes in steps S120 to S128, which derive the three-dimensional position of the target T included in the second imaging image data 50, in parallel in time.

[0114] Furthermore, the correction processing by the correction unit 15I (see step S128 in Figure 8) may occur at a timing after the coordinate transformation parameters in step S114 have been derived and stored, or at a timing after the determination processing in step S116.

[0115] Next, an example of the information processing flow executed by the information processing system 10 of this embodiment will be described.

[0116] Figure 9 is a flowchart showing an example of the information processing flow performed by the information processing system 10 of this embodiment.

[0117] Figure 9 shows an example of the process flow for deriving the three-dimensional position and orientation of the first imaging device 20 and the coordinate transformation parameters by the information processing system 10. Furthermore, the explanation assumes that when the information processing shown in Figure 9 is executed, three-dimensional map data 42 is created by the three-dimensional map data generation unit 15A and stored in the storage unit 14.

[0118] The acquisition unit 15B acquires second image data 50 of one of the multiple second imaging devices 22 arranged in the target space TS, which has not undergone processing for three-dimensional position and orientation, as well as coordinate transformation parameter derivation (step S200).

[0119] The 3D position and orientation derivation unit 15C derives the 3D position and orientation of the second imaging device 22 that captured the second imaging image data 50, based on the second imaging image data 50 acquired in step S200 and the 3D map data 42 of the target space TS (step S202).

[0120] The parameter derivation unit 15D derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 to the world coordinate system, based on the three-dimensional position and orientation of the second imaging device 22 derived in step S202 and the optical specification information of the second imaging device 22 (step S204).

[0121] The output control unit 15F stores the three-dimensional position and orientation of the second imaging device 22 derived in step S202, along with the identification information of the second imaging device 22, in the position and orientation management DB 14A (step S206). The output control unit 15F also stores the coordinate transformation parameters of the second imaging device 22 derived in step S204, along with the identification information of the second imaging device 22, in the parameter management DB 14C (step S206).

[0122] The determination unit 15E determines whether at least one of the derivation of the three-dimensional position and orientation of the second imaging device 22 in step S202 and the derivation of the coordinate transformation parameters of the second imaging device 22 in step S204 was successful (step S208).

[0123] The output control unit 15F outputs the determination result from step S208 to an output device 17 such as an external system 30 (step S210).

[0124] Next, the control unit 15 determines whether or not the processing in steps S200 to S210 has been performed for all second imaging devices 22 fixedly arranged in the target space TS (step 212).

[0125] For example, each time a second imaging device 22 is newly and permanently placed in the target space TS, the control unit 15 stores the identification information (camera ID) of the second imaging device 22 in the storage unit 14. Also, when the control unit 15 executes the processing in steps S200 to S210, it stores the execution time of the processing and a flag indicating that the processing is complete, associated with the identification information of the second imaging device 22 that performed the processing. The control unit 15 receives the identification information of the second imaging device 22 along with the second image data 50 from the second imaging device 22, and each time it executes the processing in steps S200 to 210 for the second imaging device 22 identified by the received identification information, it stores the execution time of the processing and a flag indicating that the processing is complete, associated with the identification information. Then, the control unit 15 executes the decision in step S212 by determining whether or not a flag indicating that the processing is complete is associated with the identification information of all second imaging devices 22 stored in the storage unit 14.

[0126] If the result in step S212 is negative (step S212: No), the process returns to step S200. If the result in step S212 is positive (step S212: Yes), this routine terminates.

[0127] Figure 10 is a flowchart showing an example of the information processing flow performed by the information processing system 10 of this embodiment.

[0128] Figure 10 shows an example of the process flow for deriving the three-dimensional position of the target T by the information processing system 10.

[0129] The acquisition unit 15B acquires the second image data 50 captured by the second imaging device 22 located in the target space TS (step S300).

[0130] The target 2D position derivation unit 15G derives the 2D position of the target T included in the second captured image data 50 acquired in step S300 on the second captured image data 50 (step S302).

[0131] The conversion unit 15H converts the two-dimensional position of the target T in the second captured image data 50, which was derived in step S300 and is included in the second captured image data 50 acquired in step S300, into a three-dimensional position in the target space TS (three-dimensional map data 42) using coordinate transformation parameters associated with the identification information of the second imaging device 22 that captured the second captured image data 50 (step S304).

[0132] The output control unit 15F registers the three-dimensional position of the second imaging device 22, which was converted in step S304, into the three-dimensional map data 42 (step S306).

[0133] The correction unit 15I corrects at least one coordinate transformation parameter of the multiple second imaging devices 22 so that the three-dimensional positions obtained by transforming the two-dimensional positions on each of the second imaging image data 50 of the same target T, which are captured at the same timing and acquired from each of the multiple second imaging devices 22 fixedly arranged in the target space TS, based on the coordinate transformation parameters corresponding to each second imaging device 22, are the same (step S308).

[0134] Next, the control unit 15 determines whether or not to terminate the process (step S310). The control unit 15 makes the determination in step S310 by determining whether or not it has received a signal indicating the termination of the process, for example, by an operation instruction from the user to the input unit 12. If the determination in step S310 is negative (step S310: No), the process returns to step S300. If the determination in step S310 is positive (step S310: Yes), the routine terminates.

[0135] As described above, the information processing system 10 of this embodiment includes at least one processor. The at least one processor derives the three-dimensional position of the second imaging device 22 based on the three-dimensional map data 42 of the real space generated by VSLAM processing of the first image data of the real space (target space TS) captured by the movable first imaging device 20, and the second image data 50 of the real space (target space TS) captured by the second imaging device fixedly positioned in the real space (target space TS).

[0136] As described above, the information processing system 10 of this embodiment derives the three-dimensional position of the second imaging device 22 using the three-dimensional map data 42 generated by VSLAM processing. Therefore, the information processing system 10 of this embodiment can derive the three-dimensional position of the second imaging device 22, which is fixedly arranged in the target space TS, with high accuracy.

[0137] Therefore, the information processing system 10 of this embodiment can derive the three-dimensional position of a fixedly positioned imaging device (second imaging device 22) in real space with high precision.

[0138] Furthermore, at least one processor of the information processing system 10 in this embodiment further derives the orientation of the second imaging device 22.

[0139] Therefore, the information processing system 10 of this embodiment can derive the three-dimensional position and orientation of the imaging device (second imaging device 22) fixedly positioned in real space with high precision.

[0140] Furthermore, at least one processor of the information processing system 10 of this embodiment derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 to the world coordinate system, based on the three-dimensional position and orientation of the second imaging device 22 and the optical specification information of the second imaging device 22.

[0141] Thus, the information processing system 10 of this embodiment derives the coordinate transformation parameters of the second imaging device 22 using the three-dimensional position and orientation of the second imaging device 22, which are derived with high accuracy using the three-dimensional map data 42 generated by VSLAM processing. Therefore, in addition to the above effects, the information processing system 10 of this embodiment can derive the coordinate transformation parameters of the second imaging device 22, which is fixedly placed in the target space TS, with high accuracy. Furthermore, since the information processing system 10 of this embodiment does not require the user to pre-associate the two-dimensional position on the second imaging image data 50 with the three-dimensional position in the target space TS through user input operations, it is possible to improve processing speed, reduce the user's burden, and derive high-precision and dynamic coordinate transformation parameters.

[0142] Furthermore, at least one processor in the information processing system 10 of this embodiment stores the three-dimensional position and orientation of the second imaging device 22 in association with the identification information of the second imaging device 22. Therefore, in addition to the above effects, the information processing system 10 of this embodiment can manage the highly accurate three-dimensional position and orientation of the second imaging device 22 so that it can be used in other processes.

[0143] Furthermore, at least one processor in the information processing system 10 of this embodiment stores the coordinate transformation parameters in association with the identification information of the second imaging device 22. Therefore, in addition to the above effects, the information processing system 10 of this embodiment can manage the highly accurate coordinate transformation parameters of the second imaging device 22 so that they can be used in other processes.

[0144] Furthermore, at least one processor of the information processing system 10 in this embodiment converts the two-dimensional position of the target T included in the second captured image data 50 onto the second captured image data 50 into a three-dimensional position represented by the world coordinate system, based on coordinate transformation parameters for converting the camera coordinate system of the second imaging device 22 that captured the second captured image data 50 into the world coordinate system.

[0145] The information processing system 10 of this embodiment uses highly accurate coordinate transformation parameters to convert the two-dimensional position of the target T included in the second captured image data 50 into a three-dimensional position, thereby enabling the derivation of a highly accurate three-dimensional position of the target T.

[0146] The second imaging device 22 may be fixedly positioned in real space (target space TS) before the 3D map data 42 generation process. Therefore, in the information processing system 10 of this embodiment, the highly accurate 3D position and orientation of the second imaging device 22 can be derived regardless of the timing of its installation.

[0147] Furthermore, at least one processor of the information processing system 10 of this embodiment generates 3D map data 42 by VSLAM processing based on the first captured image data each time a predetermined first condition is met. Therefore, in addition to the above effects, the information processing system 10 of this embodiment can suitably update the 3D map data 42 to a more up-to-date state.

[0148] Furthermore, at least one processor of the information processing system 10 in this embodiment outputs to the output device 17 a determination result of whether at least one of the derivation of the three-dimensional position and orientation of the second imaging device 22 and the derivation of the coordinate transformation parameters of the second imaging device 22 was successful or not. Therefore, the information processing system 10 in this embodiment can provide the user with confirmation of whether or not these derivations were successful.

[0149] Furthermore, at least one processor of the information processing system 10 in this embodiment corrects at least one coordinate transformation parameter of the plurality of second imaging devices 22 so that the three-dimensional positions obtained by transforming the two-dimensional positions on each of the second imaging image data 50 of the same object T, which are captured at the same timing and acquired from each of the plurality of second imaging devices 22, based on the coordinate transformation parameter corresponding to each second imaging device 22, are the same.

[0150] Therefore, the information processing system 10 of this embodiment can derive coordinate transformation parameters with even higher accuracy.

[0151] In this embodiment, the target space TS is the real space to be managed in System 1, and as an example, it is described as a predetermined space such as a space within a specific building in the real space. However, the target space TS is not limited to a real space; it can be any real space.

[0152] For example, the target space TS may be any subspace in real space that includes at least the area where the second imaging device 22 is located. In this case, for example, the person who installs the second imaging device 22 in the target space TS can input the approximate location of the second imaging device 22 in real space by operating the input unit 12 of the information processing system 10. The control unit 15 of the information processing system 10 can then use the subspace in real space that includes at least the input location as the target space TS.

[0153] The 3D map data generation unit 15A then generates 3D map data 42 of the target space TS, which is the subspace.

[0154] (Second Embodiment) In the above embodiment, an example was described in which the information processing system 10 performs the following: generation of 3D map data 42, derivation of the 3D position and orientation of the second imaging device 22 and coordinate transformation parameters based on the 3D map data 42, and derivation of the 3D position of the target T. In this embodiment, an embodiment is described in which some of these processes are performed by different information processing systems.

[0155] In this embodiment, the same reference numerals are used for the same parts as in the above embodiment, and detailed descriptions are omitted.

[0156] Figure 11 is an explanatory diagram of an example of system 1B of this embodiment.

[0157] System 1B comprises an information processing system 60, an external system 30, a first imaging device 20, and a second imaging device 22. The information processing system 60, the external system 30, the first imaging device 20, and the second imaging device 22 are connected to each other via a network NW or the like, either wirelessly or via a wired connection.

[0158] System 1B is the same as System 1 in the above embodiment, except that it includes an information processing system 60 instead of an information processing system 10.

[0159] The information processing system 60 includes a first information processing system 60A and a second information processing system 60B.

[0160] The first information processing system 60A includes at least one processor, which generates three-dimensional map data 42 by VSLAM processing of first captured image data of the target space TS captured by the first imaging device 20. The second information processing system 60B includes at least one processor, which derives the three-dimensional position and orientation of the second imaging device 22 based on the three-dimensional map data 42 and the second captured image data 50 captured by the second imaging device 22.

[0161] In detail, the first information processing system 60A is one or more information processing devices. The first information processing system 60A is composed of one or more dedicated or general-purpose computers.

[0162] The first information processing system 60A comprises a communication unit 61, an input unit 62, an output unit 63, a storage unit 64, and a control unit 65. The communication unit 61, input unit 62, output unit 63, and storage unit 64 and the control unit 65 are communicated to each other via a bus or the like. At least one of the external system 30, the output unit 63, and the storage unit 64 functions as an output device 67 to which the control unit 65 outputs various types of information.

[0163] The communication unit 61 communicates with an external system 30, a first imaging device 20, a second imaging device 22, and a second information processing system 60B, etc., via a network NW or the like. The input unit 62 accepts various operations from the user. The output unit 63 outputs various types of information. The output unit 63 includes a display for showing various types of information, a speaker for outputting various types of sound, etc. At least one of the input unit 62 and the output unit 63 may be configured to be located outside the first information processing system 60A and to be connected to the control unit 65 in a manner that allows communication.

[0164] The storage unit 64 stores various types of data. The storage unit 64 may be, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disk. The storage unit 64 may also be a storage device located outside the first information processing system 60A. Alternatively, the storage unit 64 may be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN or the Internet.

[0165] In this embodiment, the storage unit 64 stores two-dimensional map data 40, three-dimensional map data 42, etc. The two-dimensional map data 40 and three-dimensional map data 42 are the same as in the above embodiment.

[0166] The control unit 65 performs information processing in the information processing system 10.

[0167] The control unit 65 includes a 3D map data generation unit 15A. The 3D map data generation unit 15A is the same as in the above embodiment.

[0168] The functional units included in the control unit 65 are implemented by one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be implemented by at least one processor such as a dedicated IC, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0169] Alternatively, at least one of the components included in the control unit 65 may be mounted on an external information processing device such as an external system 30 that is connected to the information processing system 10 via a network NW or the like.

[0170] The second information processing system 60B is one or more information processing devices. The second information processing system 60B is composed of one or more dedicated or general-purpose computers.

[0171] The second information processing system 60B includes a communication unit 71, an input unit 72, an output unit 73, a storage unit 74, and a control unit 75. The communication unit 71, input unit 72, output unit 73, and storage unit 74 and the control unit 75 are communicated to each other via a bus or the like. At least one of the external system 30, the output unit 73, and the storage unit 74 functions as an output device 77 to which the control unit 75 outputs various types of information.

[0172] The communication unit 71 communicates with the external system 30, the first imaging device 20, the second imaging device 22, and the first information processing system 60A, etc., via a network NW or the like. The input unit 72 accepts various operations from the user. The output unit 73 outputs various information. The output unit 73 is a display for displaying various information, a speaker for outputting various sounds, etc. At least one of the input unit 72 and the output unit 73 may be configured to be located outside the second information processing system 60B and to be connected to the control unit 75 in a manner that allows communication.

[0173] The storage unit 74 stores various types of data. The storage unit 64 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disc. The storage unit 74 may also be a storage device located outside the second information processing system 60B. Alternatively, the storage unit 74 may be a storage medium on which programs and various types of information are downloaded and stored or temporarily stored via a LAN, the Internet, or the like.

[0174] In this embodiment, the storage unit 74 stores position and orientation management DB 14A, specification management DB 14B, parameter management DB 14C, 2D map data 40, and 3D map data 42, etc. These data and databases are the same as in the above embodiment.

[0175] The control unit 75 performs information processing in the information processing system 10.

[0176] The control unit 65 includes an acquisition unit 15B, a 3D position and orientation derivation unit 15C, a parameter derivation unit 15D, a determination unit 15E, an output control unit 15F, a target 2D position derivation unit 15G, a conversion unit 15H, and a correction unit 15I. The control unit 75 is the same as the control unit 15 of the information processing system 10 of the above embodiment, except that it does not include a 3D map data generation unit 15A.

[0177] The functional units included in the control unit 75 are implemented by one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be implemented by at least one processor such as a dedicated IC, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0178] Alternatively, at least one of the components included in the control unit 65 may be mounted on an external information processing device such as an external system 30 that is connected to the information processing system 10 via a network NW or the like.

[0179] The control unit 75 is the same as the control unit 15 in the above embodiment, except that it acquires the 3D map data 42 generated by the first information processing system 60A from the first information processing system 60A and stores it in the storage unit 74.

[0180] Thus, in this embodiment, the information processing system 60 includes a first information processing system 60A and a second information processing system 60B.

[0181] The first information processing system 60A includes at least one processor, which generates three-dimensional map data 42 by VSLAM processing of first captured image data of the target space TS captured by the first imaging device 20. The second information processing system 60B includes at least one processor, which derives the three-dimensional position and orientation of the second imaging device 22 based on the three-dimensional map data 42 and the second captured image data 50 captured by the second imaging device 22.

[0182] Thus, even if the information processing system 60 performs the generation process of 3D map data 42, the derivation process of the 3D position and orientation of the second imaging device 22 and coordinate transformation parameters based on the 3D map data 42, and the derivation process of the 3D position of the target T using different information processing systems 60 (first information processing system 60A, second information processing system 60B), the same effects as in the first embodiment can be obtained.

[0183] Next, an example of the hardware configuration of the information processing system 10, information processing system 60, and external system 30 of the above embodiment will be described.

[0184] Figure 12 is a hardware configuration diagram of an example of the information processing system 10, information processing system 60, and external system 30 of the above embodiment.

[0185] The information processing system 10, information processing system 60, and external system 30 in the above embodiment are interconnected by a bus 88, with a CPU (Central Processing Unit) 80, ROM (Read Only Memory) 82, RAM (Random Access Memory) 84, and I / F 86, etc., and have a hardware configuration that utilizes a normal computer.

[0186] The CPU 80 is a computing device that controls the information processing system 10, the information processing system 60, and the external system 30 in the above embodiment. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes performed by the CPU 80. The I / F 86 is an interface connected to the storage unit, input unit, output unit, sensor, and communication unit, etc., for sending and receiving data.

[0187] In the information processing system 10, information processing system 60, and external system 30 of the above embodiment, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above-mentioned functional units on the computer.

[0188] Furthermore, the programs for executing the above-described processes performed by the information processing system 10, information processing system 60, and external system 30 in the above embodiment may be stored in an HDD (hard disk drive). Alternatively, the programs for executing the above-described processes performed by the information processing system 10, information processing system 60, and external system 30 in the above embodiment may be pre-installed and provided in ROM 82.

[0189] Furthermore, the programs for executing the above-described processes performed by the information processing system 10, information processing system 60, and external system 30 of the above-described embodiment may be provided as a computer program product by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD). Alternatively, the programs for executing the above-described information processing performed by the information processing system 10, information processing system 60, and external system 30 of the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the programs for executing the above-described information processing performed by the information processing system 10, information processing system 60, and external system 30 of the above-described embodiment may be provided or distributed via a network such as the Internet.

[0190] Although embodiments have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0191] Furthermore, this technology can also be configured as follows: (1) An information processing system comprising at least one processor, wherein the at least one processor derives the three-dimensional position of a second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first image data of the real space captured by a movable first imaging device, and second image data of the real space captured by a second imaging device fixedly positioned in the real space. (2) The information processing system according to (1), wherein the at least one processor further derives the orientation of the second imaging device. (3) The information processing system according to (1) or (2), wherein the at least one processor derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device to a world coordinate system based on the three-dimensional position of the second imaging device and the optical specification information of the second imaging device. (4) The information processing system according to any one of (1) to (3), wherein at least one processor stores the three-dimensional position of the second imaging device and the identification information of the second imaging device in association. (5) The information processing system according to (3) or (4), wherein at least one processor stores the coordinate transformation parameters and the identification information of the second imaging device in association. (6) The information processing system according to any one of (3) to (5), wherein at least one processor converts the two-dimensional position of an object included in the second imaging image data onto the second imaging image data into a three-dimensional position represented by the world coordinate system based on the coordinate transformation parameters of the second imaging device that captured the second imaging image data. (7) The information processing system according to any one of (1) to (6), wherein the second imaging device is fixedly positioned in the real space before the three-dimensional map data generation process. (8) The information processing system according to any one of (1) to (7), wherein at least one processor generates the three-dimensional map data by VSLAM processing based on the first captured image data each time a predetermined condition is met.(9) The information processing system according to any one of (3) to (8), wherein the at least one processor outputs to an output device a determination result of whether at least one of the derivation of the three-dimensional position of the second imaging device and the derivation of the coordinate transformation parameters of the second imaging device has been successful. (10) The information processing system according to any one of (3) to (9), wherein the at least one processor corrects the coordinate transformation parameters of at least one of the plurality of second imaging devices so that the three-dimensional positions obtained by transforming the two-dimensional position on the second imaging image data of the same object, which is included in the second imaging image data taken at the same timing and obtained from each of the plurality of second imaging devices, based on the coordinate transformation parameters corresponding to each of the second imaging devices, are the same. (11) The information processing system according to any one of (1) to (10), wherein the at least one processor generates the three-dimensional map data of the subspace by VSLAM processing based on the first captured image data of the subspace including at least the region in which the second imaging device exists in the real space. (12) The information processing system according to any one of (1) to (11), comprising: a first information processing system in which the at least one processor generates the three-dimensional map data by VSLAM processing of the first captured image data of the real space captured by the first imaging device; and a second information processing system in which the at least one processor derives the three-dimensional position of the second imaging device based on the three-dimensional map data and the second captured image data captured by the second imaging device. (13) An information processing method performed by an information processing system comprising at least one processor, the method comprising deriving the three-dimensional position of a second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first image data of the real space captured by a movable first imaging device, and second image data of the real space captured by a second imaging device fixedly positioned in the real space.(14) An information processing program for causing a computer to perform the following steps: (14) Deriving the three-dimensional position of the second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first imaging image data of the real space captured by a movable first imaging device, and second imaging image data of the real space captured by a second imaging device fixedly positioned in the real space.

[0192] 10, 60 Information Processing System 15A 3D Map Data Generation Unit 15C 3D Position and Orientation Derivation Unit 15D Parameter Derivation Unit 15E Determination Unit 15F Output Control Unit 15G Target 2D Position Derivation Unit 15H Conversion Unit 15I Correction Unit 60A First Information Processing System 60B Second Information Processing System

Claims

1. An information processing system comprising at least one processor, wherein the at least one processor derives the three-dimensional position of a second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first image data of the real space captured by a first movable imaging device, and second image data of the real space captured by a second imaging device fixedly positioned in the real space.

2. The information processing system according to claim 1, wherein the at least one processor further derives the orientation of the second imaging device.

3. The information processing system according to claim 1, wherein the at least one processor derives coordinate transformation parameters for converting the camera coordinate system of the second imaging device to a world coordinate system based on the three-dimensional position of the second imaging device and the optical specification information of the second imaging device.

4. The information processing system according to claim 1, wherein at least one processor stores the three-dimensional position of the second imaging device and identification information of the second imaging device in association with each other.

5. The information processing system according to claim 3, wherein the at least one processor stores the coordinate transformation parameters and the identification information of the second imaging device in association with each other.

6. The information processing system according to claim 3, wherein the at least one processor converts the two-dimensional position of an object included in the second captured image data onto the second captured image data into a three-dimensional position represented by the world coordinate system, based on the coordinate transformation parameters of the second imaging device that captured the second captured image data.

7. The information processing system according to claim 1, wherein the second imaging device is fixedly positioned in the real space before the three-dimensional map data generation process.

8. The information processing system according to claim 1, wherein the at least one processor generates the three-dimensional map data by VSLAM processing based on the first captured image data each time a predetermined condition is met.

9. The information processing system according to claim 3, wherein the at least one processor outputs to an output device a determination result of whether at least one of the derivation of the three-dimensional position of the second imaging device and the derivation of the coordinate transformation parameters of the second imaging device was successful.

10. The information processing system according to claim 3, wherein the at least one processor corrects the coordinate transformation parameter of at least one of the plurality of second imaging devices such that the three-dimensional positions obtained by transforming the two-dimensional positions on the second imaging image data of the same object, which are included in the second imaging image data acquired from each of the plurality of second imaging devices and captured at the same timing, based on the coordinate transformation parameter corresponding to each of the plurality of second imaging devices, are the same.

11. The information processing system according to claim 1, wherein the at least one processor generates the three-dimensional map data of the subspace by VSLAM processing based on the first captured image data of the subspace including at least the region where the second imaging device exists in the real space.

12. The information processing system according to claim 1, comprising: a first information processing system in which at least one processor generates the three-dimensional map data by VSLAM processing of the first captured image data of real space captured by the first imaging device; and a second information processing system in which at least one processor derives the three-dimensional position of the second imaging device based on the three-dimensional map data and the second captured image data captured by the second imaging device.

13. An information processing method performed by an information processing system comprising at least one processor, the method comprising deriving the three-dimensional position of a second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first image data of the real space captured by a movable first imaging device, and second image data of the real space captured by a second imaging device fixedly positioned in the real space.

14. An information processing program for causing a computer to perform the following steps:

14. Deriving the three-dimensional position of a second imaging device based on three-dimensional map data of the real space generated by VSLAM processing of first image data of the real space captured by a first movable imaging device, and second image data of the real space captured by a second imaging device fixedly positioned in the real space.

Images