Information processing device and information processing method

Abstract

An acquisition unit 13 acquires image data that represents captured images that have been captured by an imaging device 20. The acquired image data is stored at a storage unit 12. An estimation unit 14 uses a well-known technique such as SLAM or SfM to estimate an imaging position for the imaging device 20 from a plurality of pieces of image data that represent images of a subject as captured by the imaging device 20. A correction unit 15 corrects the imaging position estimated by the estimation unit 14 so as to reduce the difference between the distance between a first imaging position and a second imaging position that have been determined as positions at which the imaging device 20 captures images and the distance between the first imaging position and the second imaging position as calculated using the position estimated by the estimation unit 14.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present invention relates to a technique for estimating an imaging position of an imaging device based on image data.

[0002] As a method for estimating the imaging position of an imaging device based on a plurality of image data captured by the imaging device, techniques such as Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM) are known. The imaging position and imaging direction of the imaging device estimated for each image data are used, for example, to generate a three-dimensional model from a two-dimensional image by a technique called Gaussian Splatting.

[0003] Patent Document 1 discloses a first management unit that manages a first database that stores a plurality of imaging information each including image data, position data representing the imaging position where the image data was captured, and depth data representing the distance from the imaging position represented by the position data to an object, an acquisition unit that acquires target image data, a position estimation unit that estimates target position data representing the imaging position where the target image data was captured based on the target image data, a depth estimation unit that estimates target depth data representing the distance from the imaging position represented by the target position data to an object based on the target image data, a registration unit that registers new imaging information including the target image data, the target position data, and the target depth data in the first database, and a display control unit that causes a display device to display map data representing a map of the environment where the image data was captured. The display control unit causes at least one of the image data and the depth data included in specified specified imaging information among the plurality of imaging information stored in the first database to be displayed in correspondence with a pixel position corresponding to the position data included in the specified imaging information in the map data. An information processing apparatus is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2022-105442

[0005] When estimating the imaging position of an imaging device using only image data, the accuracy of the estimation is poor.

[0006] Therefore, the present invention aims to improve the accuracy of estimating the imaging position by an imaging device, even when using image data.

[0007] To solve the above problems, the present invention provides an information processing device that, when identifying the imaging position by a moving imaging device from image data showing an image captured by the imaging device, uses the distance between a first imaging position and a second imaging position determined as the position where the imaging device performs imaging.

[0008] According to the present invention, even when using image data, the accuracy of estimating the imaging position by the imaging device can be improved.

[0009] This is a block diagram showing an example of the hardware configuration of an information processing device 10 according to one embodiment of the present invention. This is a block diagram showing an example of the hardware configuration of an imaging device 20 according to the same embodiment. This is a block diagram showing an example of the functional configuration of the information processing device 10. This is a plan view illustrating the relationship between the movement route of the imaging device 20, the imaging timing, and the set distance. This is a diagram illustrating data stored by the information processing device 10. This is a diagram illustrating data stored by the information processing device 10. This is a flowchart showing an example of the operation of the information processing device 10.

[0010] [Embodiment] [Configuration] An embodiment of the present invention will be described below. Figure 1 is a diagram showing the hardware configuration of an information processing device 10 according to an embodiment of the present invention. Physically, the information processing device 10 is configured as a computer including a processor 1001, memory 1002, storage 1003, communication device 1004, and a bus connecting these. Each of these devices operates on power supplied from a battery (not shown). In the following description, the word "device" can be read as a circuit, device, unit, etc. The hardware configuration of the information processing device 10 may be configured to include one or more of the devices shown in Figure 1, or it may be configured to omit some of the devices. Alternatively, multiple devices with different housings may be connected to each other via communication to constitute the information processing device 10.

[0011] Each function in the information processing device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of data reading and writing in the memory 1002 and storage 1003.

[0012] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, and so on.

[0013] The processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described later. Functional blocks of the information processing device 10 may be stored in the memory 1002 and implemented by control programs that run on the processor 1001. Various processes may be executed by one processor 1001, but may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted to the information processing device 10 via a telecommunications line.

[0014] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out the method according to this embodiment.

[0015] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device.

[0016] The communication device 1004 is hardware (transmitting / receiving device) for communication between computers, and is also called a network device, network controller, network card, or communication module. Image data showing two-dimensional images captured from multiple imaging positions by an imaging device (described later) is input to the information processing device 10 via the communication device 1004.

[0017] Each device, such as the processor 1001 and the memory 1002, is connected by a bus for communicating information. The bus may be configured using a single bus, or different buses may be used for each device.

[0018] The information processing device 10 may include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by this hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0019] Figure 2 shows an example of the electrical configuration of the imaging device 20. The imaging device in the present invention can be any device that captures an image of a subject from multiple imaging positions, but below we will describe an imaging device 20 that is self-propelled and capable of capturing an image of a subject from multiple imaging positions.

[0020] Physically, the imaging device 20 is configured as a computer device including a processor 2001, memory 2002, storage 2003, communication device 2004, user interface device 2005, imaging device 2006, wheel control mechanism 2007, and a bus connecting these. The electrical configuration of the imaging device 20 may include one or more of the devices shown in the figure, or it may be configured without some of the devices. The processor 2001, memory 2002, storage 2003, and communication device 2004 are the same hardware as the processor 1001, memory 1002, storage 1003, and communication device 1004 described above.

[0021] The user interface device 2005 includes an input device (e.g., keys, switches, buttons, etc.) that receives input from the user and an output device (e.g., a display, speaker, LED lamp, etc.) that provides output to the user. The input device and the output device may be configured as an integrated unit (e.g., a touchscreen).

[0022] The imaging device 2006 includes a lens and an image sensor (not shown) and generates two-dimensional image data showing the captured image. The communication device 2004 transmits the image data showing the two-dimensional images captured by the imaging device 2006 from multiple imaging positions to the communication device 1004 of the information processing device 10.

[0023] The wheel control mechanism 2007 is a mechanism that controls wheels (not shown) for the imaging device 20 to travel on the road surface, and includes a motor mechanism for rotating the wheels and a steering mechanism for changing the direction of the wheel axles. The wheel control mechanism 2007 drives the motor mechanism and steering mechanism under the control of the processor 2001.

[0024] Next, Figure 3 is a block diagram showing the functional configuration of the information processing device 10. In the information processing device 10, the processor 1001 reads programs and the like from the storage 1003 into the memory 1002 and executes them, thereby realizing the functions of the imaging motion control unit 11, the storage unit 12, the acquisition unit 13, the estimation unit 14, the correction unit 15, and the output unit 16.

[0025] The imaging operation control unit 11 remotely controls the operation of the imaging device 20. In this embodiment, as shown in Figure 4, the movement path R by which the imaging device 20 moves for imaging and the positions p1, p2, p3, p4... (hereinafter referred to as set imaging positions) on the movement path by which the imaging device 20 performs imaging are predetermined. Since the movement path R and the set imaging positions p1, p2, p3, p4... are predetermined, the set distances d1, d2, d3, d4... connecting any two set imaging positions are also predetermined. In Figure 4, the distance d1 when positions p1 and p2 are connected by a line segment (referred to as the set distance), the set distance d2 connecting positions p2 and p3, the set distance d3 connecting positions p3 and p4, and the set distance d4 connecting positions p1 and p4 are shown as examples, but the set distances connecting any two set imaging positions are predetermined.

[0026] The imaging operation control unit 11 instructs the imaging device 20 to move along the movement path R at a certain speed, and also instructs the imaging device 20 to take an image at the timing when it will arrive at a set imaging position when moving along the movement path at the aforementioned speed. For this reason, as shown in Figure 5, the storage unit 12 stores position information indicating the location of each set imaging position. Furthermore, as shown in Figure 6, the storage unit 12 stores any two set imaging positions and the distance between them (referred to as the set distance) in association. In addition, the storage unit 12 also stores position information (not shown) indicating the location of the movement path R.

[0027] The acquisition unit 13 acquires image data representing the captured images taken at each set imaging position by the imaging device 20. The acquired image data is stored in the storage unit 12.

[0028] The estimation unit 14 estimates the imaging position by the imaging device 20 for each of the multiple image data representing images of the subject captured by the imaging device 20. Known methods for estimating the imaging position and imaging direction (relative imaging position and imaging direction) from multiple captured image data include, for example, Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SfM). Therefore, the estimation unit 14 uses these well-known techniques such as SLAM or SfM to estimate the imaging position by the imaging device 20 for each image data.

[0029] The correction unit 15 corrects the imaging position estimated by the estimation unit 14 so that the difference between the set distance, which is the distance between a certain imaging setting position (first imaging position) and another imaging setting position (second imaging position) determined as the position where the imaging device 20 takes images, and the distance between the first imaging position and the second imaging position calculated using the position estimated by the estimation unit 14, becomes smaller. Here, "so that the difference becomes smaller" means adjusting the scale in the image so that the distance between the first imaging position and the second imaging position calculated using the position estimated by the estimation unit 14 using SLAM or SfM, etc. (estimated distances x1, x2, x3, x4… in Figure 6) matches the set distances (d1, d2, d3, d4…) of the first imaging position and the second imaging position stored in the storage unit 12. When the position of the imaging device is estimated from image data using SLAM or SfM, etc., a scale that serves as a reference for the size of the subject is calculated in the image shown by that image data. The correction unit 15 adjusts the size of this scale to make the estimated distance match the set distance. This corrects the imaging position estimated by the estimation unit 14.

[0030] More specifically, the correction unit 15 performs correction based on the following equation (1), where di is the distance traveled by the imaging device 20 along the movement path R, xi is the distance between each set imaging position (set distance), and s is the scale.

[0031] Through the estimation and correction described above, the information processing device 10 identifies the imaging position of the moving imaging device 20 from image data representing an image captured by the imaging device 20, using the distance between the first imaging position and the second imaging position, which are determined to be the positions where the imaging device 20 performs imaging.

[0032] The output unit 16 outputs the imaging position of the imaging device 20, corrected by the correction unit 15 for each image data, in association with that image data. The imaging position of the imaging device 20 in each image data is used, for example, to generate a 3D model from 2D images captured from multiple viewpoints using Gaussian Splatting. The 3D model referred to here is a model for rendering an image viewed from an arbitrary viewpoint in 3D space.

[0033] [Operation] Next, the operation of this embodiment will be described. Figure 7 is a flowchart showing an example of the operation of the information processing device 10. In Figure 7, the acquisition unit 13 acquires a plurality of image data showing images of a subject captured by the imaging device 20 (step S11).

[0034] The estimation unit 14 estimates the imaging position by the imaging device 20 for each of the multiple image data showing images of the subject captured by the imaging device 20 (step S12).

[0035] The correction unit 15 performs a correction based on the aforementioned equation (1), where Ti is a predetermined timing for imaging, di is the distance the imaging device 20 moves along the movement path, and s is the scale (step S13). As a result, the imaging position estimated by the estimation unit 14 is corrected.

[0036] The output unit 16 outputs the imaging position of the imaging device, which has been corrected by the correction unit 15 for the image data, in correspondence with the image data (step S14).

[0037] According to the embodiments described above, it is possible to improve the accuracy of estimating the imaging position by the imaging device, even when using only image data.

[0038] [Modifications] The present invention is not limited to the embodiments described above. The embodiments described above may be modified as follows. Furthermore, two or more of the following modifications may be combined and implemented.

[0039] [Modification 1] In the above embodiment, the estimated imaging positions were corrected by adjusting the scale in the image so that the estimated distance between the first imaging position and the second imaging position, calculated using the positions estimated by the estimation unit 14 using SLAM or SfM, etc., matched the set distance between the first imaging position and the second imaging position stored in the storage unit 12.

[0040] Alternatively, the estimation unit 14 may estimate the imaging position by the imaging device 20 by limiting the error between the estimated distances of the first and second imaging positions estimated from image data using SLAM or SfM, etc., and the set distances of the first and second imaging positions stored in the storage unit 12, so as to minimize the error.

[0041] Specifically, the estimation unit 14 estimates the imaging position by the imaging device 20 based on the following equation (2). Here, di is the distance traveled by the imaging device 20 on the movement path R, xi is the distance between each set imaging position (set distance), s is the scale, A is the parameter when the imaging device 20 is operating, R is the rotation matrix (element component r), t is the translation vector, Xw (X, Y, Z components) is the world coordinate point, n is the number of images, m is the number of world coordinate points, Iij is a function that is 1 if the i-th world coordinate point is captured in the j-th image, and 0 otherwise, u and v are the coordinates of feature points extracted between images i and j, and λ is a parameter that adjusts the weight of the regularization term. In the bundle adjustment process, which minimizes equation (2), a regularization term is added.

[0042] [Other Modifications] The block diagram used in the description of the above embodiment shows functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may be realized by combining the above one device or the above multiple devices with software.

[0043] Functions include, but are not limited to, judgment, decision-making, determination, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, election, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation (allocating, mapping), assignment, etc. For example, a functional block (component) that enables transmission is referred to as a transmission control unit or a transmitter. As described above, the implementation method is not particularly limited.

[0044] For example, the information processing apparatus 10 in an embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure.

[0045] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems that utilize LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (ULtra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, as well as next-generation systems extended based on these. Further, a combination of multiple systems (for example, a combination of at least one of LTE and LTE-A and 5G) may be applied.

[0046] For each processing procedure, sequence, flowchart, etc. described in the present disclosure, the order may be changed as long as there is no contradiction. For example, regarding the methods described in the present disclosure, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0047] The input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0048] The determination may be made based on a value represented by 1 bit (0 or 1), may be made based on a boolean value (true or false), or may be made by comparing numerical values (e.g., comparison with a predetermined value).

[0049] As described above, the present disclosure has been described in detail. However, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and changed forms without departing from the spirit and scope of the present disclosure defined by the claims. Therefore, the description of the present disclosure is for the purpose of illustrative explanation and has no restrictive meaning with respect to the present disclosure.

[0050] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name. Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0051] The information, signals, etc., described herein may be represented using any of the following different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms used herein and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meaning.

[0052] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0053] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0054] Any reference to elements using the designations “First,” “Second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the First and Second elements do not imply that only two elements may be employed, or that the First element must precede the Second element in any way.

[0055] In the above-described configuration of each device, the term "part" may be replaced with "means," "circuit," "device," etc.

[0056] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0057] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0058] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0059] 10: Information processing device, 11: Imaging motion control unit, 12: Memory unit, 13: Acquisition unit, 14: Estimation unit, 15: Correction unit, 16: Output unit, 1001: Processor, 1002: Memory, 1003: Storage, 1004: Communication device, 20... Imaging device, 2001: Processor, 2002: Memory, 2003: Storage, 2004: Communication device, 2005... User interface device, 2006... Imaging device, R... Movement path, p1, p2, p3, p4... Set imaging position, d1, d2, d3, d4... Set distance, x1, x2, x3, x4... Estimated distance.

Claims

1. An information processing device characterized in that, when identifying the imaging position by a moving imaging device from image data showing an image captured by the imaging device, the identification is performed using the distance between a first imaging position and a second imaging position determined as the position where the imaging device performs imaging.

2. The information processing apparatus according to claim 1, comprising: an estimation unit that estimates an imaging position from image data showing an image captured by a moving imaging device; and a correction unit that corrects the imaging position estimated by the estimation unit so that the difference between the distance between a first imaging position and a second imaging position determined as the position where the imaging device performs imaging and the distance between the first imaging position and the second imaging position calculated using the position estimated by the estimation unit becomes small.

3. An information processing apparatus according to claim 1, comprising an estimation unit for estimating an imaging position from image data showing an image captured by a moving imaging device, the estimation unit for estimating an imaging position such that the difference between the distance between a first imaging position and a second imaging position determined as the position where the imaging device performs imaging is small, and the distance between the first imaging position and the second imaging position calculated using the estimated position is small.

4. The information processing apparatus according to claim 2, characterized in that, when the movement distance of the imaging device is di, the distance between positions where the imaging device performs imaging is xi, and s is the correction value, the correction unit performs the correction based on the following formula.

5. Let \(d_i\) be the moving distance of the imaging device, \(x_i\) be the distance between the positions where the imaging device performs imaging, \(s\) be the scale, \(A\) be a parameter when the imaging device operates, \(R\) be a rotation matrix (element components \(r\)), \(t\) be a translation vector, \(X_w(X, Y, Z\) components) be a world coordinate point, \(n\) be the number of image sheets, \(m\) be the number of world coordinate points, \(I_{ij}\) be a function that becomes 1 if the \(i\)-th world coordinate point appears in the \(j\)-th image and 0 otherwise, \(u, v\) be the feature point coordinates extracted between images \(i, j\), and \(\lambda\) be a parameter for adjusting the weight of the regularization term. The estimation unit performs estimation based on the following formula, and the information processing apparatus according to claim 3 is characterized in that.

6. An information processing method characterized in that, when identifying the imaging position by a moving imaging device from image data showing an image captured by the imaging device, the identification is performed using the distance between a first imaging position and a second imaging position determined as the position where the imaging device performs imaging.

Images