Information processing device and method

Abstract

The present disclosure pertains to an information processing device and method that make it easier to perform calibration between imaging units fixedly connected to each other in the relative orientations at which the imaging units do not have a common field of view. As processing related to calibration of a first imaging unit and a second imaging unit fixedly connected to each other in the relative orientations at which the imaging units do not have a common field of view: feature points are extracted from a first captured image captured by the first imaging unit; feature points are extracted from a second captured image captured by the second imaging unit; first matching is executed between the feature points extracted from the first captured image and the feature points extracted from the second captured image; and as a result of the first matching, matched feature points are presented in a state distinguishable from unmatched feature points. The present disclosure can be applied to, for example, information processing devices, information processing methods, and programs.

Description

Information Processing Apparatus and Method

[0001] The present disclosure relates to an information processing apparatus and method, and particularly relates to an information processing apparatus and method that can more easily perform calibration between imaging units that are fixedly connected in a relative posture without a common field of view.

[0002] Conventionally, in fields such as video production, there has been a method of tracking the position and posture of a main camera for video imaging using a marker-less camera tracking system that employs a camera. When using such a camera tracking system, it is necessary to estimate in advance the positional and postural offset (external parameters) between the tracking camera for tracking and the main camera for imaging.

[0003] For example, in a case where cameras share a common field of view such as a stereo camera, the relative positional relationship between the cameras can be easily estimated geometrically from the positional relationship of markers in images by, for example, capturing an image of a common marker. However, since a tracking camera used in a tracking system generally does not share a common field of view with the main camera, it has been difficult to apply such a method. Further, although estimation of external parameters can be performed by using a marker having a dedicated shape that can be imaged simultaneously by both the tracking camera and the main camera, this not only increases the cost of the marker but also requires complicated work.

[0004] Incidentally, the position and posture of a camera can be estimated using SfM (Structure from Motion), which estimates the three-dimensional structure of a subject from a plurality of two-dimensional images (see, for example, Non-Patent Document 1).

[0005] Roger Mohr, 2 others, "Relative 3D Reconstruction Using Multiple Uncalibrated Images", The International Journal of Robotics Research, SAGE Publications, 1995, 14 (6), pp.619-632, December 1, 1995

[0006] However, when applying such SfM to the calibration of relative positions between cameras, the procedure is complex, and if the work is not performed efficiently, the workload and costs could increase.

[0007] This disclosure has been made in view of the above circumstances and aims to make it easier to perform calibration between imaging units that are fixedly connected in a relative orientation without a common field of view.

[0008] One aspect of this technology is an information processing device comprising: a calibration processing unit that performs a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view, which includes extracting feature points from a first image captured by the first imaging unit and extracting feature points from a second image captured by the second imaging unit, and performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and an information presentation unit that, as a result of the first matching, presents the matched feature points in a state that can be distinguished from the unmatched feature points.

[0009] One aspect of this technology is an information processing method that, as a calibration process for a first imaging unit and a second imaging unit fixedly connected in a relative orientation without a common field of view, includes extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image, and presenting the matched feature points as a result of the first matching in a state that can be distinguished from the unmatched feature points.

[0010] In one aspect of this technology, the information processing device and method involve a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view. This process involves extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image, and as a result of this first matching, the matched feature points are presented in a way that allows them to be distinguished from the unmatched feature points.

[0011] This is a diagram illustrating the tracking system. This is a diagram illustrating an example of the calibration process. This is a diagram illustrating an example of the calibration process. This is a diagram illustrating the common observation area. This is a diagram illustrating an example of the calibration process. This is a diagram illustrating an example of the information presentation process main configuration of the imaging tracking system. This is a functional block diagram showing the functions of the control device. This is a flowchart illustrating an example of the calibration process flow. This is a flowchart following Figure 13, showing an example of the calibration process flow. This is a block diagram illustrating an example of the main configuration of the computer.

[0012] The following describes the embodiments for implementing this disclosure. The explanation will be given in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. Estimation of relative positional relationship between cameras 3. Presentation of information regarding calibration 4. First embodiment (imaging tracking system) 5. Appendix

[0013] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0014] Non-patent document 1: (mentioned above)

[0015] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0016] <2. Estimation of Relative Positional Relationship Between Cameras> <Calibration of Camera Tracking System> Conventionally, in fields such as video production, there has been a method of tracking the position and orientation of the main camera used for video capture using a markerless camera tracking system that uses cameras, as shown in Figure 1. For example, in the camera tracking system of Figure 1, the position and orientation of the main camera 11 used for capturing is tracked based on the image of the tracking camera 12. When using such a camera tracking system, it was necessary to perform position and orientation calibration (i.e., estimation of the position and orientation offset (external parameter) between those cameras) between the tracking camera 12 (coordinate system) and the main camera 11 (coordinate system) in advance.

[0017] For example, in the case of a stereo camera, where there is a common field of view between cameras (a field of view area that falls within the angle of view of both cameras), as shown in Figure 2A, the stereo cameras (camera 21 and camera 22) capture the same marker 23, and the relative positional relationship of the cameras can be easily estimated geometrically from the positional relationship of the marker 23 in the image.

[0018] However, as shown in Figure 2B, for example, the tracking camera 12 used in a tracking system is generally oriented in a different direction from the main camera 11 and does not share a common field of view with the main camera 11. Therefore, it was difficult to apply techniques such as those used in the case of a stereo camera.

[0019] Furthermore, as shown in Figure 3, for example, external parameters can be estimated by using a marker with a special shape that can be captured simultaneously by both the tracking camera and the main camera. In the example in Figure 3, a special L-shaped marker 31 is prepared, and calibration is performed by capturing images simultaneously with both the tracking camera 12 and the main camera 11. In this case, since the shape of the marker 31 is known, the positional relationship of the markers captured by each camera is known. Therefore, calibration can be performed using this known positional relationship.

[0020] However, this method requires the preparation of a dedicated marker 31, which could increase costs. Furthermore, compared to a general marker 23 like the example in Figure 2, the shape becomes more complex, potentially increasing the difficulty of managing the marker 31.

[0021] Furthermore, in some cases, the relative positions of the tracking camera 12 and the main camera 11 were provided using CAD (Computer-Aided Design) design values. However, this method required the user to calculate and manually input the values, which was a cumbersome process.

[0022] Therefore, it was considered to perform estimation using Structure from Motion (SfM) with rig constraints. As shown in Non-Patent Literature 1, SfM is a method for reconstructing the position and orientation of cameras and a 3D point cloud from images taken using one or more arbitrary cameras. There are various formulations of SfM, and among them, a problem setting in which the relationship between the position and orientation of multiple cameras is fixed (rig constraint) is also known. Colmap, a well-known SfM software, provides a function to perform estimation with stereo rig constraints, and can simultaneously estimate external parameters between cameras on the premise that two cameras constitute a stereo camera. In order to perform SfM estimation accurately and stably, it is necessary to capture something common between images. In the case of stereo rig constraints, there is always a common field of view of the cameras, so this does not need to be considered, but in the case of no common field of view, as in the camera tracking system described above, the movement of the cameras during imaging becomes important. For example, in order to estimate external parameters with high accuracy, it is necessary to obtain a sufficiently large number of common feature points between cameras, but if the imaging method is inappropriate, these common feature points may be insufficient, and the accuracy of external parameter estimation may decrease.

[0023] For example, as shown in Figure 4, in SfM, feature points are obtained from the image captured by camera 41 by moving camera 41 and capturing images of object 50 to generate multiple captured images. Similarly, feature points are obtained from the image captured by camera 42 by capturing images of object 50 using camera 42. By obtaining feature points at the same position with both cameras, external parameters between the cameras can be estimated. In other words, in order to estimate external parameters, it is necessary not only to capture a common area with multiple cameras, but also to obtain common feature points. For example, in the example in Figure 4, feature points 51 to 53 are obtained by imaging with camera 41, and feature points 52 and 53 are obtained by imaging with camera 42. In other words, feature point 51 is not obtained by imaging with camera 42. Therefore, it is desirable to move camera 42, for example, as shown by the dotted arrow, so that feature point 51 can be obtained. However, without information on the obtained feature points and without know-how, it was difficult for an operator to grasp such camera movements.

[0024] In other words, while it is necessary to perform the work efficiently using the correct procedures, if the workers lack the necessary know-how, the workload and costs could increase. Furthermore, there was a risk that the accuracy of external parameter estimation would decrease.

[0025] <3. Presentation of Calibration Information> <Presentation of Feature Point Matching Results> Therefore, information that will be useful for the operator to perform the work more efficiently will be presented. For example, feature point matching will be performed between cameras (captured images), and the matching results will be presented in a way that allows identification of matched and unmatched feature points.

[0026] For example, the information processing device may include a calibration processing unit that performs a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view, which involves extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, and performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and an information presentation unit that, as a result of the first matching, presents the matched feature points in a manner that allows them to be distinguished from the unmatched feature points.

[0027] Furthermore, the information processing method executed by the information processing device includes, as a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view, the following steps: extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image, and, as a result of the first matching, presenting the matched feature points in a state that can be distinguished from the unmatched feature points.

[0028] By presenting this type of information, operators can perform imaging more efficiently, based on that information, in order to obtain more matched feature points between captured images. Furthermore, a reduction in the accuracy of external parameter estimation can be suppressed. In other words, calibration between fixedly connected imaging units in relative orientations without a common field of view can be made easier (i.e., the complexity of the work can be reduced).

[0029] <Resolving Scale Indetermination> Furthermore, the external parameters obtained by solving the SfM problem are scale-indeterminate, and in order to know how far apart the cameras are physically, it is necessary to combine this with a method for determining the physical scale separately.

[0030] Therefore, an IMU (Inertial Measurement Unit) can be used to provide a physical scale. For example, IMU data can be combined as input during SfM optimization to add constraints on the physical scale. An IMU is a sensor that measures acceleration and angular velocity and is often used in combination with SfM or SLAM (Simultaneous Localization and Mapping). Although it is rarely installed in the main camera, it is almost always installed in tracking cameras. Therefore, this method can be easily implemented.

[0031] Alternatively, you can use markers of known size. For example, you can photograph a calibration board with a checkerboard pattern to obtain a scale. Since the sole purpose is to obtain a scale, it is not necessary to capture the object within the field of view of both the main camera and the tracking camera simultaneously; it is sufficient to obtain images taken from multiple locations using only one camera.

[0032] <Presentation of information in accordance with the calibration process> Furthermore, in calibration work using SfM, by presenting appropriate information according to the process, operators can perform the work more appropriately (more efficiently), and calibration between imaging units that are fixedly connected in a relative orientation without a common field of view can be performed more easily.

[0033] <Phase 1> First, in Phase 1, as shown in Figure 5, the user (operator) takes images of, for example, an object 61 as the subject with the main camera 11 until a sufficient number of feature points are extracted. At that time, the user takes images while moving the main camera 11. At that time, the feature points extracted from the image obtained by the main camera 11 may be superimposed on the image and presented.

[0034] For example, in an information processing device, a calibration processing unit may, as part of the calibration process, extract feature points from a first image captured by the main camera 11 in parallel with the imaging by the first imaging unit (main camera 11). The information presentation unit may then superimpose the extracted feature points onto the first image and present them.

[0035] For example, a display screen 70 as shown in Figure 6 may be displayed. The display screen 70 is provided with a region 71 for displaying the first captured image and a region 72 for displaying the second captured image captured by the second imaging unit (tracking camera 12). In region 71, the first captured image with extracted feature points (×) superimposed is displayed.

[0036] By doing so, the user can easily check the status of feature point acquisition while operating the main camera 11.

[0037] The conditions for ending imaging by the main camera 11 are determined by quantitatively judging the number of points and their distribution on the image. For example, in an information processing device, the calibration processing unit may, as part of the calibration process, terminate imaging by the first imaging unit (main camera 11) and start imaging by the second imaging unit (tracking camera 12) when the number of extracted feature points reaches a predetermined standard.

[0038] <Phase 2> This termination decision may be made automatically, or the user may choose to make the termination decision at any time when the termination conditions are met. Once the termination decision is made, the system transitions to Phase 2, and the image captured by the main camera 11, i.e., the first image displayed in area 71, may be locked. In other words, the updating of the displayed image may be stopped, and the same image may continue to be displayed. For example, in an information processing device, the information presentation unit may fix the presentation of the first image when imaging by the second imaging unit (tracking camera 12) is started. Only the superimposed feature points may be updated.

[0039] In Phase 2, as shown in Figure 7, the user (operator) moves the camera rig so that the tracking camera 12 captures the area that the main camera 11 was previously viewing. In parallel with this imaging, feature points may be extracted from the second image obtained by this imaging and matched with the feature points extracted from the first image. The results of this matching may then be superimposed on the first and second images for display. For example, in an information processing device, a calibration processing unit may, as part of the calibration process, extract feature points from the second image in parallel with imaging by the second imaging unit (tracking camera 12), and perform a second matching between the feature points extracted from the first image and the feature points extracted from the second image. The information display unit may then superimpose the results of this second matching onto the first and second images for display.

[0040] For example, as shown in Figure 8, in the area 72 of the display screen 70, if the tracking camera 12 can extract feature points from the same object 61, the display changes (for example, from "×" to "○"), allowing the user to understand that the same area is visible. Here, "○" indicates that the same feature point has been observed by both cameras. Feature points in this state can contribute to the estimation of external parameters.

[0041] The system may proceed to the next phase once the number of feature points in the "○" state reaches a predetermined standard and the SfM optimization is successful. For example, in an information processing device, the calibration processing unit may, as part of the calibration process, initially estimate external parameters if the number of matched feature points in the second matching reaches a predetermined standard. Then, as part of the calibration process, if the result of the initial estimation is normal, the calibration processing unit may extract feature points from the first and second captured images and perform the first matching. Alternatively, if the result of the initial estimation is not normal, the calibration processing unit may control the system to restart the process from imaging by the first imaging unit (main camera 11).

[0042] At this point, the lock on the first image captured by the main camera 11 may be released, and the latest image may be displayed in area 71. The colors and symbols of feature points, etc., are merely examples, and any display method is acceptable. For example, various display methods are possible, such as displaying the number of observations or confidence level using a heat map.

[0043] <Phase 3> In Phase 3, the worlds of the main camera 11 and the tracking camera 12 are connected via the matched feature points ("○"), making it possible to display observations from both in 3D. In other words, in Phase 3, a display screen 70 as shown in Figure 9 is displayed.

[0044] For example, in an information processing apparatus, the information presentation unit may present matched feature points and unmatched feature points in different expressions to enable identification thereof in the matching between feature points extracted from a first captured image and feature points extracted from a second captured image (which may be either first matching or second matching). For example, in the display screen 70 of FIG. 9, feature points extracted only from the first captured image captured by the main camera 11 are indicated by white circles, feature points extracted only from the second captured image captured by the tracking camera 12 are indicated by black circles, and feature points extracted from both the first captured image and the second captured image are indicated by gray circles. That is, gray circles indicate feature points matched in the first matching or the second matching, and white circles and black circles indicate feature points not matched in the first matching or the second matching. By changing the expression of feature points in this manner, the user can more easily identify matched feature points and unmatched feature points in the first matching. It should be noted that the difference in expression that enables identification of matched feature points and unmatched feature points in this manner is not limited to the example (color) in FIG. 9, and may be any difference. For example, the shapes may be different, the patterns may be different, the sizes may be different, the densities may be different, or the line expressions (for example, solid lines versus dotted lines, colors, thicknesses, etc.) may be different.

[0045] In addition, the information presentation unit may present guidance for the movement direction of a second imaging unit (tracking camera 12) such that the number of matched feature points increases. For example, on the display screen 70 of FIG. 9, an icon 81 corresponding to the main camera 11 and an icon 82 corresponding to the tracking camera 12 are displayed. Furthermore, an arrow 83 is shown indicating a direction recommended as the moving direction of the icon 82 (i.e., the tracking camera 12) (a direction for increasing the number of matched feature points in the matching between feature points extracted from the first captured image and feature points extracted from the second captured image). By presenting such an arrow 83, the user can more easily grasp how the tracking camera 12 should be moved.

[0046] Furthermore, the information presentation unit may arrange and present the matched feature points and unmatched feature points in a virtual three-dimensional space. For example, on the display screen 70 in Fig. 9, the area 73 is an area for displaying the virtual three-dimensional space, and black dots, white dots, and gray dots indicating feature points are displayed in the area 73 as a first matching result. That is, the matched feature points and the unmatched feature points are arranged in the virtual three-dimensional space. In addition, an icon 81, an icon 82, an arrow 83, and the like are also arranged in this virtual three-dimensional space.

[0047] Accordingly, the user can intuitively grasp the positional relationship between the cameras and objects (feature points). Therefore, the user can more easily move the camera to an appropriate direction. That is, calibration between imaging units that are fixedly connected in a relative posture with no common field of view can be performed more easily.

[0048] It should be noted that the information presentation unit may further present a first captured image and a second captured image on which the matched feature points and the unmatched feature points are superimposed. For example, the display screen 70 in Fig. 9 is also provided with an area 71 and an area 72, and feature points are displayed superimposed on the first captured image and the second captured image. By presenting various information in this way, the user can more easily grasp the current situation (the progress of feature point detection). Therefore, calibration between imaging units that are fixedly connected in a relative posture with no common field of view can be performed more easily.

[0049] While checking these displays (the display of the captured images and the virtual three-dimensional space), the user performs an imaging operation to update feature points that can only be observed by one camera to feature points that can be observed by both cameras. Since observing at the same position does not necessarily increase the number of observations, the user increases observations while changing the position and posture of the camera. Feature points that can only be detected from one viewpoint cannot be displayed in 3D, so the user can only confirm them via superimposition on the captured images in the area 71 and the area 72.

[0050] <Phase 4> When the number of matched feature points reaches a predetermined standard, the calibration processing unit performs SfM optimization, calculates external parameters, and proceeds to Phase 4. The timing of this execution may be determined automatically or left to the user. For example, in an information processing device, the calibration processing unit may estimate external parameters as part of the calibration process when the number of matched feature points reaches a predetermined standard. The information presentation unit may also present the results of this estimation.

[0051] For example, information such as that shown in Figure 10 may be displayed on the display screen 70. For example, the estimation results may include the estimated component values ​​of each external parameter, a 3D display of the external parameter, and the estimation error (SfM reprojection error). In addition, an overall evaluation (a grade notation that allows the user to intuitively grasp the accuracy of the calibration) may be displayed. For example, in an information processing device, a calibration processing unit may evaluate the accuracy of the calibration. The information display unit may then present the results of that evaluation.

[0052] By presenting various types of information in this way, users can understand the calibration results more easily and accurately.

[0053] Another possible verification method involves superimposing the estimated 3D point cloud onto the image and allowing the user to qualitatively judge the degree of overlap. This method constitutes validation rather than evaluation using the training data used for calibration. Based on these results, the user can choose to save the estimation results and exit, or retry.

[0054] <4. First Embodiment> <Imaging Tracking System> The technology described above can be applied to any configuration (device, system, unit, processing unit, etc.). Figure 11 shows an example of the main configuration of an imaging tracking system, which is one embodiment of an information processing system to which this technology is applied.

[0055] The imaging tracking system 300 shown in Figure 11 is a system that tracks the position and orientation of the main camera 311, which performs imaging, using a tracking system with a tracking camera 312. As shown in Figure 11, in addition to the main camera 311 and the tracking camera 312, the imaging tracking system 300 also includes a monitor 313, a control device 314, and a monitor 315. The main camera 311 and the tracking camera 312 (tracking system) are connected to the control device 314 via a cable 310 so as to be able to communicate with it.

[0056] Cable 310 is a communication cable conforming to a predetermined standard and represents one form of communication medium. In other words, any other communication medium may be used instead of cable 310. This communication medium may also include space. That is, each device such as the main camera 311, tracking camera 312, and control device 314 may perform wireless communication.

[0057] The monitor 313 is configured as a display unit for the main camera 311. However, the monitor 313 may be configured as a separate device from the main camera 311, such as a tablet terminal. In that case, the monitor 313 should have a communication function and be able to exchange information with other devices.

[0058] The monitor 315 is configured as a peripheral device of the control device 314. However, the monitor 315 may be configured as a device independent of the control device 314 (a device that can be operated even without the control device 314), such as a tablet terminal. In that case, the monitor 315 should have a communication function and be able to exchange information with other devices.

[0059] The control device 314 controls the driving of the main camera 311 and the tracking camera 312 to perform imaging and tracking. It can also perform calibration processing for the main camera 311 and the tracking camera 312 as described above.

[0060] Although Figure 11 shows one of each device, the main camera 311, tracking camera 312, monitor 313, control device 314, and monitor 315 can all be present in any number, either as a single unit or in multiple units. The number of units may also differ between each device.

[0061] <Control device> Figure 12 is a functional block diagram illustrating the functions of the control device 314. As shown in Figure 12, the control device 314 includes a frame selection unit 351, a calibration processing unit 352, a camera operation instruction estimation unit 353, and an information presentation unit (GUI (Graphical User Interface)) 354.

[0062] The frame selection unit 351 performs processing related to the selection of frames from which feature points are extracted (frames from which feature point matching is performed). In other words, calibration is performed using a portion of the captured images. For example, the frame selection unit 351 acquires images of each frame (first captured images) from the main camera 311. The frame selection unit 351 also acquires images of each frame (second captured images) from the tracking system 331, which includes the tracking camera 312. The frame selection unit 351 may also acquire IMU data from the tracking system 331. The frame selection unit 351 may also acquire camera position and orientation information indicating the position and orientation of the main camera 311 from the tracking system 331. The frame selection unit 351 selects frames from which feature points are extracted from the captured images of each frame and supplies those images (first captured images and second captured images) to the calibration processing unit 352. The frame selection unit 351 may also supply those images (first captured images and second captured images) to the information presentation unit 354.

[0063] The calibration processing unit 352 performs calibration-related processing. For example, the calibration processing unit 352 acquires the first and second captured images supplied from the frame selection unit 351. The calibration processing unit 352 may also acquire IMU data from the tracking system 331. The calibration processing unit 352 uses the acquired information to perform calibration of the position and orientation between the main camera 311 (coordinate system) and the tracking camera 312 (coordinate system) (estimation of the position and orientation offset (external parameters) between the cameras). The calibration processing unit 352 also supplies the camera position and orientation derived from 2D point cloud coordinates, 3D point cloud coordinates, captured images, etc., and estimated external parameters to the information presentation unit 354 as a result of the calibration processing. The calibration processing unit 352 may also supply the 3D point cloud coordinates, camera position and orientation, and external parameters to the camera operation instruction estimation unit 353. The calibration processing unit 352 may also provide the external parameters to other systems, such as the tracking system 331.

[0064] The camera operation instruction estimation unit 353 performs processing related to the estimation of camera operation instructions. For example, the camera operation instruction estimation unit 353 may acquire 3D point cloud coordinates, camera position and orientation, and external parameters supplied from the calibration processing unit 352. The camera operation instruction estimation unit 353 may also estimate the camera operation instructions to be performed on the camera based on this information. The camera operation instruction estimation unit 353 may also supply the estimated camera operation instructions to the information presentation unit 354. Note that this camera operation instruction estimation unit 353 may be omitted.

[0065] The information presentation unit 354 performs processing related to the presentation of information. For example, the information presentation unit 354 acquires 2D point cloud coordinates, 3D point cloud coordinates, camera position and orientation, and external parameters supplied from the calibration processing unit 352. The information presentation unit 354 also acquires the first captured image and the second captured image supplied from the frame selection unit 351. The information presentation unit 354 may also acquire camera operation instructions supplied from the camera operation instruction estimation unit 353. The information presentation unit 354 may also acquire camera position and orientation supplied from the tracking system 331. Based on this information, the information presentation unit 354 generates a display image that presents various information related to calibration and displays the display image on monitor 313 or monitor 315.

[0066] <Application of this technology> The technology described above in <3. Presentation of information regarding calibration> may be applied to the imaging tracking system 300 with the above configuration.

[0067] For example, in a control device 314 (information processing device), the calibration processing unit 352 may perform a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view. This process may involve extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, and performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image. The information presentation unit 354 may then present the matched feature points as a result of the first matching in a manner that allows them to be distinguished from the unmatched feature points.

[0068] The information display unit 354 may present the matched feature points and the unmatched feature points in different ways. The information display unit 354 may also provide guidance on the direction of movement of the second imaging unit (tracking camera 312) in a way that increases the number of matched feature points. The information display unit 354 may also present the matched and unmatched feature points arranged in a virtual three-dimensional space. Furthermore, the information display unit 354 may also present the first and second captured images in which the matched and unmatched feature points are superimposed.

[0069] Furthermore, the calibration processing unit 352 may estimate external parameters as part of the calibration process when the number of matched feature points reaches a predetermined standard. The information presentation unit 354 may then present the results of this estimation. For example, the calibration processing unit 352 may evaluate the accuracy of the calibration. The information presentation unit 354 may then present the results of this evaluation.

[0070] Furthermore, the calibration processing unit 352 may extract feature points from the first captured image in parallel with the imaging by the first imaging unit (main camera 311) as part of the calibration process. The information presentation unit 354 may also superimpose the extracted feature points onto the first captured image and present them. Additionally, as part of the calibration process, the calibration processing unit 352 may terminate imaging by the first imaging unit (main camera 311) and start imaging by the second imaging unit (tracking camera 312) when the number of extracted feature points reaches a predetermined standard. The information presentation unit 354 may also fix the presentation of the first captured image when imaging by the second imaging unit (tracking camera 312) begins. Furthermore, the calibration processing unit 352 may, as part of the calibration process, extract feature points from the second captured image in parallel with the imaging by the second imaging unit (tracking camera 312), and perform a second matching between the feature points extracted from the first captured image and the feature points extracted from the second captured image.

[0071] Furthermore, as part of the calibration process, the calibration processing unit 352 may initially estimate external parameters if the number of matched feature points in the second matching reaches a predetermined standard. For example, as part of the calibration process, if the initial estimation result is normal, the calibration processing unit 352 may extract feature points from the first and second captured images and perform the first matching. Alternatively, as part of the calibration process, if the initial estimation result is not normal, the calibration processing unit may restart the imaging process from the first imaging unit (main camera 311).

[0072] The control device 314, with this function, makes it easier to perform calibration between imaging units that are fixedly connected in a relative orientation without a common field of view.

[0073] <Calibration Process Flow> Next, an example of the calibration process flow performed by this control device 314 will be explained with reference to the flowcharts in Figures 13 and 14.

[0074] When the calibration process is started, in step S101 of Figure 13, the calibration processing unit 352 causes the main camera 311 to take an image and extracts feature points from the captured image.

[0075] In step S102, the calibration processing unit 352 determines whether a sufficient number of feature points have been detected. If it is determined that the number of feature points is insufficient, the process returns to step S101 in Figure 13 and executes the subsequent processes. If it is determined in step S102 that a sufficiently large number of feature points have been detected, the process proceeds to step S103.

[0076] In step S103, the information display unit 354 fixes the display image (first captured image) from the main camera 311.

[0077] In step S104, the calibration processing unit 352 causes the tracking camera 312 to take an image and extracts feature points from the second captured image.

[0078] In step S105, the calibration processing unit 352 performs feature point matching.

[0079] In step S106, the calibration processing unit 352 determines whether a sufficient number of points have matched. If it is determined that the number is insufficient, the process returns to step S104 and executes the subsequent processes. If it is determined in step S106 that a sufficient number of points have matched, the process proceeds to step S107.

[0080] In step S107, the calibration processing unit 352 initializes the external parameters.

[0081] In step S108, the calibration processing unit 352 determines whether the initial setting result is normal or not. If it is determined that the initial estimation result of the external parameters is not normal, the process returns to step S101 in Figure 13 and executes the subsequent processes. If it is determined in step S108 that the initial estimation result of the external parameters is normal, the process proceeds to Figure 14.

[0082] In step S121 of Figure 14, the information display unit 354 releases the lock on the display image of the main camera 311.

[0083] In step S122, the information display unit 354 starts 3D display as shown in the example in Figure 9.

[0084] In step S123, the calibration processing unit 352 extracts feature points.

[0085] In step S124, the calibration processing unit 352 performs feature point matching.

[0086] In step S125, the calibration processing unit 352 determines whether a sufficient number of points have been matched. If it is determined that the number of matched points is insufficient, the process returns to step S123 and the subsequent processes are executed.

[0087] Furthermore, in step S126, the calibration processing unit 352 estimates external parameters.

[0088] In step S127, the information display unit 354 displays the estimation result.

[0089] In step S128, the calibration processing unit 352 determines whether or not to terminate the calibration process. If it is determined not to terminate, the process returns to step S101 in Figure 13, and the subsequent processes are executed. If it is determined in step S128 to terminate the calibration process, the calibration process is terminated.

[0090] The process enclosed by the dotted line frame 401 in Figure 13 corresponds to the Phase 1 process described above. The process enclosed by the dotted line frame 402 in Figure 13 corresponds to the Phase 2 process described above. The process enclosed by the dotted line frame 403 in Figure 14 corresponds to the Phase 3 process described above. The process enclosed by the dotted line frame 404 in Figure 14 corresponds to the Phase 4 process described above. In other words, the process of each phase can be executed by executing each process in the flow shown in Figures 13 and 14.

[0091] By performing each process as described above, calibration between imaging units that are fixedly connected in a relative orientation without a common field of view can be made easier.

[0092] For simplicity, this flow only describes the logic related to UI (User Interface) transitions, and the SfM processing for the main camera 311 and tracking camera 312 is executed independently of this UI flow. Furthermore, the SfM processing can be started from phase 1 or phase 2. From phase 3 onwards, the restoration results of the tracking camera 312 and the main camera 311 can be drawn in a common coordinate system.

[0093] <5. Addendum> <Computer> The series of processes described above may be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0094] Figure 15 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0095] In the computer 900 shown in Figure 15, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0096] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0097] The input unit 911 may consist of, for example, a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 912 may consist of, for example, a display, speaker, output terminal, etc. The storage unit 913 may consist of, for example, a hard disk, RAM disk, non-volatile memory, etc. The communication unit 914 may consist of, for example, a network interface, etc. The drive 915 may drive removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0098] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0099] The program executed by the computer may be recorded and applied on a removable medium 921, such as a package medium. In that case, the program may be installed in the storage unit 913 via the input / output interface 910 when the removable medium 921 is mounted on the drive 915.

[0100] Furthermore, this program may be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program may be received by the communication unit 914 and installed in the storage unit 913.

[0101] In addition, this program may be pre-installed in ROM 902 or storage unit 913.

[0102] <Applicable Subjects of This Technology> Furthermore, this technology can be applied to any configuration. For example, this technology can be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0103] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0104] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0105] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0106] For example, this technology can be applied to systems and devices used to provide entertainment content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0107] <Other> In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0108] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0109] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0110] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0111] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0112] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0113] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0114] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising: a calibration processing unit which performs a calibration process for a first imaging unit and a second imaging unit which are fixedly connected in relative orientations that do not have a common field of view with each other, by extracting feature points from a first image captured by the first imaging unit, extracting feature points from a second image captured by the second imaging unit, and performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and an information presentation unit which, as a result of the first matching, presents the matched feature points in a state that can be distinguished from the unmatched feature points. (2) The information processing device according to (1), wherein the information presentation unit presents the matched feature points and the unmatched feature points in different representations. (3) The information processing device according to (1) or (2), wherein the information presentation unit presents guidance for the direction of movement of the second imaging unit that increases the number of matched feature points. (4) The information display unit presents the matched feature points and the unmatched feature points arranged in a virtual three-dimensional space, as described in any of (1) to (3). (5) The information display unit further presents the first and second captured images in which the matched feature points and the unmatched feature points are superimposed, as described in any of (1) to (4). (6) The calibration processing unit, as part of the calibration process, estimates external parameters when the number of matched feature points reaches a predetermined standard, and the information display unit presents the results of the estimation, as described in any of (1) to (5). (7) The calibration processing unit evaluates the accuracy of the calibration, and the information display unit presents the results of the evaluation, as described in (6). (8) The information processing apparatus according to any one of (1) to (7), wherein the calibration processing unit extracts feature points from the first captured image in parallel with the imaging by the first imaging unit as a calibration process, and the information presentation unit superimposes the extracted feature points onto the first captured image.(9) The information processing device according to (8), wherein the calibration processing unit terminates imaging by the first imaging unit and starts imaging by the second imaging unit when the number of extracted feature points reaches a predetermined standard as part of the calibration processing. (10) The information processing device according to (9), wherein the information presentation unit fixes the presentation of the first image when imaging by the second imaging unit is started. (11) The information processing device according to (9) or (10), wherein the calibration processing unit extracts feature points from the second image in parallel with imaging by the second imaging unit as part of the calibration processing, performs a second matching of feature points extracted from the first image and feature points extracted from the second image, and the information presentation unit presents the results of the second matching superimposed on the first image and the second image. (12) The information processing device according to (11), wherein the calibration processing unit, as a process relating to the calibration, initially estimates external parameters when the number of matched feature points in the second matching reaches a predetermined standard. (13) The information processing device according to (12), wherein the calibration processing unit, as a process relating to the calibration, extracts feature points from the first captured image and the second captured image and performs the first matching when the result of the initial estimation is normal. (14) The information processing device according to (12) or (13), wherein the calibration processing unit, as a process relating to the calibration, restarts the imaging by the first imaging unit when the result of the initial estimation is not normal.(15) An information processing method comprising: a calibration process for a first imaging unit and a second imaging unit fixedly connected in relative orientations that do not share a common field of view with each other, the process comprising: extracting feature points from a first image captured by the first imaging unit; extracting feature points from a second image captured by the second imaging unit; performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and presenting the matched feature points as a result of the first matching in a manner that allows them to be distinguished from the unmatched feature points. (16) The information processing method according to (15), further comprising presenting the matched feature points and the unmatched feature points in different representations of each other. (17) The information processing method according to (15) or (16), further comprising presenting guidance for the direction of movement of the second imaging unit that increases the number of matched feature points. (18) An information processing method according to any one of (15) to (17), further comprising arranging and presenting the matched feature points and the unmatched feature points in a virtual three-dimensional space. (19) An information processing method according to any one of (15) to (18), further comprising presenting the first captured image and the second captured image in which the matched feature points and the unmatched feature points are superimposed. (20) An information processing method according to any one of (15) to (19), which, as a calibration process, comprises estimating external parameters when the number of matched feature points reaches a predetermined standard, and presenting the results of the estimation.

[0115] 300 Imaging tracking system, 310 Cable, 311 Main camera, 312 Tracking camera, 313 Monitor, 314 Control device, 315 Monitor, 331 Tracking system, 351 Frame selection unit, 352 Calibration processing unit, 353 Camera operation instruction estimation unit, 354 Information presentation unit, 900 Computer

Claims

1. Information processing apparatus comprising: a calibration processing unit that performs a calibration process for a first imaging unit and a second imaging unit that are fixedly connected in relative orientations without a common field of view, which include: extracting feature points from a first image captured by the first imaging unit; extracting feature points from a second image captured by the second imaging unit; and performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and an information presentation unit that presents the matched feature points as a result of the first matching in a manner that allows them to be distinguished from the unmatched feature points.

2. The information display unit presents the matched feature points and the unmatched feature points in different ways, as described in claim 1.

3. The information processing apparatus according to claim 1, wherein the information presentation unit provides guidance on the direction of movement of the second imaging unit such that the number of matched feature points increases.

4. The information display unit presents the matched feature points and the non-matched feature points in a virtual three-dimensional space, as described in claim 1.

5. The information processing apparatus according to claim 1, wherein the information presentation unit further presents the first captured image and the second captured image in which the matched feature points and the unmatched feature points are superimposed.

6. The information processing apparatus according to claim 1, wherein the calibration processing unit estimates external parameters when the number of matched feature points reaches a predetermined standard as part of the calibration processing, and the information presentation unit presents the results of the estimation.

7. The information processing apparatus according to claim 6, wherein the calibration processing unit evaluates the accuracy of the calibration, and the information presentation unit presents the results of the evaluation.

8. The information processing apparatus according to claim 1, wherein the calibration processing unit extracts feature points from the first captured image in parallel with imaging by the first imaging unit as a calibration process, and the information presentation unit superimposes the extracted feature points onto the first captured image and presents them.

9. The information processing apparatus according to claim 8, wherein the calibration processing unit, as a calibration process, terminates imaging by the first imaging unit and starts imaging by the second imaging unit when the number of extracted feature points reaches a predetermined standard.

10. The information display unit fixes the display of the first captured image when imaging by the second imaging unit starts, as described in claim 9.

11. The information processing apparatus according to claim 9, wherein the calibration processing unit, as a calibration process, extracts feature points from the second captured image in parallel with imaging by the second imaging unit, performs a second matching of feature points extracted from the first captured image and feature points extracted from the second captured image, and the information presentation unit presents the results of the second matching superimposed on the first captured image and the second captured image.

12. The information processing apparatus according to claim 11, wherein the calibration processing unit, as a process relating to the calibration, initially estimates external parameters when the number of matched feature points in the second matching reaches a predetermined standard.

13. The information processing apparatus according to claim 12, wherein the calibration processing unit, as a process relating to the calibration, extracts feature points from the first captured image and the second captured image and performs the first matching if the result of the initial estimation is normal.

14. The information processing apparatus according to claim 12, wherein the calibration processing unit, as part of the calibration processing, restarts the imaging by the first imaging unit if the result of the initial estimation is not normal.

15. An information processing method comprising: a calibration process for a first imaging unit and a second imaging unit fixedly connected in a relative orientation that does not share a common field of view with each other, the process of: extracting feature points from a first image captured by the first imaging unit; extracting feature points from a second image captured by the second imaging unit; performing a first matching between the feature points extracted from the first image and the feature points extracted from the second image; and presenting the matched feature points as a result of the first matching in a state that can be distinguished from the unmatched feature points.

16. The information processing method according to claim 15, which includes presenting the matched feature points and the unmatched feature points in different representations.

17. The information processing method according to claim 15, which includes providing guidance for the direction of movement of the second imaging unit such that the number of matched feature points increases.

18. The information processing method according to claim 15, which includes arranging and presenting the matched feature points and the non-matched feature points in a virtual three-dimensional space.

19. The information processing method according to claim 15, further comprising presenting the first and second captured images in which the matched feature points and the unmatched feature points are superimposed.

20. The information processing method according to claim 15, which includes, as a calibration process, estimating external parameters when the number of matched feature points reaches a predetermined standard, and presenting the results of the estimation.

Images