Information processing device, information processing method, and program

Abstract

This information processing device comprises a control unit that, in response to the start of SLAM processing, performs processing for executing a behavior guide display for presenting, to a user, an initialization behavior to be given to a sensor unit, which is provided with an IMU and a plurality of monocular cameras, in order to initialize IMU parameters relating to an angular velocity detection value and an acceleration detection value output from the sensor unit, and processing for executing an initialization situation display indicating a situation in which the initialization of the IMU parameters has not yet been completed or has been completed.

Description

Information Processing Apparatus, Information Processing Method, Program

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and particularly relates to the technical field of a user interface for initializing a sensor unit.

[0002] SLAM (Simultaneous Localization and Mapping) is an algorithm that can determine the position and orientation of a device by using various sensors, and is applied to AR (Augmented Reality) applications using an HMD (Head Mounted Display) or a smartphone, and to autonomous driving of vehicles and robots. Examples of sensors include a stereo camera, a monocular camera, an IMU (Inertial Measurement Unit), a LiDAR (Light Detection and Ranging), and a GPS (Global Positioning System), and various ones are selected according to the application.

[0003] Patent Document 1 below discloses a technique related to a UI that promotes SLAM initialization.

[0004] U.S. Patent Application Publication No. 2014 / 0327792

[0005] Here, in particular, in VI-SLAM (Visual Inertial SLAM) using a monocular camera and an IMU, in order to perform stable self-position estimation, it is necessary to stabilize various parameters estimated inside the algorithm immediately after startup. In particular, regarding the IMU, the values of angular velocity (gyro) and acceleration as sensor output values include noise, and the values are accompanied by individual differences due to manufacturing variations and temporal changes due to temperature changes. Therefore, at the time of SLAM startup, the accuracy of SLAM can be improved by estimating this noise amount each time.

[0006] To stabilize the parameters related to this IMU, it is necessary to move or rotate devices such as a monocular camera, a sensor unit equipped with the IMU, or a camera to which the sensor unit is attached, in a specific direction. In this disclosure, this is referred to as the "initialization behavior" for explanatory purposes. This movement is not intuitive for end users and requires user literacy. Furthermore, these parameters themselves require a deep understanding of SLAM, making it difficult for users to understand their stability by looking at the values.

[0007] While Patent Document 1 discloses a User Interface (UI) that assists in camera initialization for AR applications for smartphones, it does not disclose a UI that assists in the initialization of parameters related to the IMU.

[0008] Therefore, this technology provides a UI to assist the user with the initialization behavior when SLAM is started, when using a sensor unit equipped with multiple monocular cameras and an IMU.

[0009] The information processing device related to this technology includes a control unit that performs a behavior guide display to show the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters related to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit, which is equipped with an IMU and multiple monocular cameras, and a control unit that performs an initialization status display to show whether the initialization of the IMU parameters is not yet complete or has been completed. The output values ​​of the IMU (angular velocity detection values, acceleration detection values) contain noise, and these values ​​are subject to individual differences due to manufacturing variations and changes over time due to temperature changes. Therefore, when the self-position estimation process is started, the output values ​​of the IMU are stabilized, and a bias corresponding to the amount of noise is estimated, for example. The status is presented to the user while instructing them on the actions to be taken for initialization.

[0010] This is an explanatory diagram of the system configuration of an embodiment of this technology. This is an explanatory diagram of a sensor unit that can be used in the embodiment. This is a block diagram of the information processing device of the embodiment. This is an explanatory diagram of the initialization of iTc parameters. This is an explanatory diagram of the initialization of iTc parameters. This is an explanatory diagram of the initialization of IMU parameters. This is an explanatory diagram of the initialization of IMU parameters. This is an explanatory diagram of the initialization when using a stereo camera. This is an explanatory diagram of the initialization of the camera scale when using a monocular camera. This is an explanatory diagram of the static instruction on the UI screen of the embodiment. This is an explanatory diagram of the translation instruction on the UI screen of the embodiment. This is an explanatory diagram of the translation instruction on the UI screen of the embodiment. This is an explanatory diagram of the rotation instruction on the UI screen of the embodiment. This is an explanatory diagram of the rotation instruction on the UI screen of the embodiment. This is an explanatory diagram of the initialization completion display on the UI screen of the embodiment. This is a flowchart of the UI processing of the embodiment. This is a flowchart of the translation direction setting of the embodiment. This is a flowchart of the rotation direction setting of the embodiment.

[0011] The embodiments will be described below in the following order: <1. Imaging System> <2. Configuration of Information Processing Device> <3. Parameters> <4. UI Processing> <5. Summary and Modifications>

[0012] In this disclosure, "video" or "image" includes both still images and videos. Furthermore, "video" may refer not only to the state displayed on a display, but also to video data that is not displayed on a display. The same applies to "image."

[0013] <1. Filming System> A filming system to which the technology of this disclosure can be applied will be described. Figure 1 schematically shows a filming system 100. This filming system 100 is a system that performs filming as a virtual production, and the figure shows some of the equipment placed in the filming studio.

[0014] In the filming studio, a performance area 8 is provided where performers 9 perform their acting and other acts. Large display devices are placed on at least the back, left and right sides, and top of this performance area 8. The type of display device is not limited, but the figure shows an example of a large display device using an LED wall 5.

[0015] The LED wall 5 is formed by arranging multiple LED panels vertically and horizontally to create a large panel. The size of the LED wall 5 is not particularly limited, but it should be large enough to display the background when filming the performer 9.

[0016] Near the performance area 8, a camera 1 is positioned for, for example, filming movies or other video content. The cameraman can move the position of camera 1 and control its shooting direction and field of view. Of course, it is also conceivable that camera 1's movement and field of view can be controlled remotely. Alternatively, camera 1 may move and change its field of view automatically or autonomously. For this purpose, camera 1 may be mounted on a tripod or a mobile unit.

[0017] Camera 1 captures both the performer 9 in the performance area 8 and the image displayed on the LED wall 5. For example, if a landscape or other scenery is displayed as background image vB on the LED wall 5, it becomes possible to capture footage that is similar to what it would be like if the performer 9 were actually performing in that location.

[0018] Monitor 6 displays the video footage captured by camera 1 in real time. This allows directors and staff involved in video content production to check the footage being shot.

[0019] Even if we were to display the background image vB on LED wall 5 and film with the performer 9, simply displaying the background image vB would result in an unnatural background in the filmed footage. This is because the background image vB is actually a two-dimensional representation of a three-dimensional background with depth.

[0020] For example, camera 1 can film performer 9 in performance area 8 from various positions and in various directions, and can also zoom. Performer 9 is not standing still in one place. Therefore, the actual appearance of the background behind performer 9 should change depending on the position, shooting direction, and field of view of camera 1, but such changes cannot be obtained with the background image vB as a flat image. Therefore, the background image vB is changed so that the background, including parallax, looks the same as it actually does.

[0021] In other words, in the background video vB, at least the area captured by camera 1 will be an image that represents the scene that would actually be seen when camera 1 is at that position, depending on the position, shooting direction, field of view, etc.

[0022] Specifically, 3D background data, which is a 3D (three-dimensional) model of the background, is prepared, and the background image of the area to be captured based on the viewpoint position of camera 1 is rendered sequentially in real time onto this 3D background data. The image of the captured area rendered in real time is combined with the surrounding image and displayed on LED wall 5. As a result, even if camera 1 is moved forward, backward, left, or right, or if zoom operations are performed, the background of the area to be captured along with the performer 9 will be captured as an image that corresponds to the changes in viewpoint position and FOV (Field of View) that occur with the actual movement of camera 1.

[0023] The reason why only the video within the captured area is rendered is because the entire LED wall 5 is not always captured by camera 1. By inserting a rendered video based on the viewpoint of camera 1, at least for the captured area, a natural background can be obtained in the captured video, and the burden of rendering can be reduced.

[0024] To perform this type of shooting, camera 1 is connected to rendering engine 3 via network switch 2. This allows the captured video from camera 1, as well as camera 1's position information and field of view information, to be transmitted to rendering engine 3.

[0025] The rendering engine 3 uses 3D background data and renders background images of the area captured by camera 1, based on the camera's position and field of view information. The rendered background image and the surrounding background images are then combined to generate the overall display image of the LED wall 5. The surrounding background images are images of areas that are not captured, and therefore the viewpoint position is not changed. The combined overall display image of the LED wall 5 is supplied to the LED processor 4, and the LED processor 4 drives each LED panel that makes up the LED wall 5 to display the background image vB.

[0026] As a result, the monitor image captured by camera 1 and displayed on monitor 6, as well as the so-called main video used in the actual content, will include the performer 9 and the background. However, the background will be a real-time rendered image, meaning it will be an image viewed from a viewpoint (position and shooting direction) corresponding to the position of camera 1. Therefore, this shooting system 100 can not only display the background video vB in a two-dimensional manner, but also capture video that is similar to what would be captured if a landscape were actually photographed.

[0027] In this system, accurately detecting the position and orientation of camera 1 during shooting is crucial. To this end, a camera tracker system using SLAM has been constructed to detect the position and orientation of camera 1 in real time, and to determine where camera 1 is located and in which direction it is facing while shooting.

[0028] For this purpose, a sensor unit 10 is attached to camera 1. The sensor unit 10 is schematically shown in Figure 2, and may be a unit equipped with, for example, five monocular cameras 11a, 11b, 11c, 11d, and 11e, each arranged in a different direction for shooting, and an IMU 12. When describing the monocular cameras 11a, 11b, 11c, 11d, and 11e without distinction, they will be referred to as "monocular camera 11".

[0029] In this example, the sensor unit 10 is equipped with five monocular cameras 11, but this is just one example; in this embodiment, two or more monocular cameras 11 are sufficient. Although multiple monocular cameras 11 are provided, they are not configured as a stereo camera, but rather as unbalanced monocular cameras, each capturing images in a different direction.

[0030] Each monocular camera 11a, 11b, 11c, 11d, and 11e is mounted within the sensor unit 10 with its shooting direction indicated by the dashed arrow. For example, in this example, the monocular cameras 11a, 11b, 11c, and 11d are each oriented in a direction at 90-degree intervals, while the monocular camera 11e is oriented upwards. Note that this is just one example, and the number and orientation of the monocular cameras 11 in the sensor unit 10 can vary.

[0031] This sensor unit 10 outputs video footage from each monocular camera 11, as well as gyro data and acceleration data from the IMU 12. From this output data, the position and orientation of camera 1 are estimated in three-dimensional space, and as described above, the viewpoint for rendering, that is, the position and orientation of the viewpoint (the shooting direction that can be determined from the orientation), is estimated.

[0032] For example, the shooting system 100 is equipped with an information processing device 7, in which SLAM processing is performed using information from the sensor unit 10 to estimate the position and shooting direction of camera 1, and this information is provided to the rendering engine 3. The information processing device 7 also provides a UI for SLAM processing, so that operators and camera operators of camera 1 can view the UI screen.

[0033] For the purposes of describing this embodiment, the information processing device 7 is assumed to perform SLAM processing and UI processing, but it is also possible for separate information processing devices to perform SLAM processing and UI processing. Furthermore, one or both of the SLAM processing and UI processing may be performed by a processor in the camera 1, or by the information processing device acting as the rendering engine 3. In particular, the UI processing will be described later, but it will provide an initialization UI screen when SLAM is started, and it is assumed that this screen will be viewed by staff who can physically move the camera 1 (or the sensor unit 10) to which the sensor unit 10 is attached. Therefore, it is sufficient for the UI processing to be performed by an information processing device that controls the display on a screen that can be viewed by such staff.

[0034] <2. Configuration of Information Processing Device> An example of the configuration of the information processing device 7 in the above-described shooting system 100 is explained in Figure 3. The information processing device 7 is a device capable of information processing, such as a computer. Specifically, this information processing device 7 may be configured as a workstation, a personal computer, a mobile terminal device such as a smartphone or tablet, or a video editing device. The information processing device 7 may also be a computer device configured as a server or computing device in cloud computing.

[0035] The control unit 71 of the information processing device 7 is composed of, for example, a CPU (Central Processing Unit) and performs overall control processing and various calculation processing based on a program. The control unit 71 executes various processes according to a program stored in a non-volatile memory unit 74, such as a ROM (Read Only Memory) 72 or an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from the storage unit 79 into the RAM (Random Access Memory) 73. The RAM 73 also appropriately stores data necessary for the control unit 71 to execute various processes.

[0036] When the information processing device 7 performs SLAM processing and UI processing in the imaging system 100 shown in Figure 1, the control unit 71 includes a SLAM processing unit 71a and a UI processing unit 71b based on the program.

[0037] The SLAM processing unit 71a performs processing to estimate the position and orientation of camera 1 based on detection information from the sensor unit 10, such as images captured by the five monocular cameras 11, and angular velocity and acceleration information from the IMU 12. The UI processing unit 71b provides a UI related to the SLAM processing. In particular, as will be described later, the UI processing unit 71b performs UI processing such as display control for initializing parameters used in the SLAM processing and processing that responds to user input.

[0038] Furthermore, the functions of the SLAM processing unit 71a and the UI processing unit 71b may be implemented by a separate CPU, GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), AI (artificial intelligence) processor, etc., separate from the control unit 71. Also, the processing functions of the SLAM processing unit 71a and the UI processing unit 71b may be implemented by multiple processors.

[0039] The control unit 71, ROM 72, RAM 73, and non-volatile memory unit 74 are interconnected via a bus 83. An input / output interface 75 is also connected to this bus 83.

[0040] An input unit 76, consisting of controls or operating devices, is connected to the input / output interface 75. For example, the input unit 76 can be various controls or operating devices such as a keyboard, mouse, keys, trackball, dial, touch panel, touchpad, or remote controller. User operations are detected by the input unit 76, and the signals corresponding to the input operations are interpreted by the control unit 71. A microphone can also be considered as an input unit 76. User voice can also be input as operation information.

[0041] In addition, to the input / output interface 75, a display unit 77 composed of an LCD (Liquid Crystal Display) or an organic EL (electro-luminescence) panel, etc., and an audio output unit 78 composed of a speaker, etc. are connected integrally or separately. The display unit 77 performs various displays. The display unit 77 is constituted by, for example, a display device provided on the housing of the information processing apparatus 7, or a separate display device connected to the information processing apparatus 7, etc. The display unit 77 performs displays such as various images, operation menus, icons, messages, etc. on the display screen, that is, displays as a GUI (Graphical User Interface) based on the instructions of the control unit 71.

[0042] The input / output interface 75 may also be connected to a storage unit 79 composed of an HDD (Hard Disk Drive), a solid-state memory, etc., and a communication unit 80.

[0043] The storage unit 79 can store various data and programs. A database can also be configured in the storage unit 79.

[0044] The communication unit 80 performs communication processing via a transmission path such as the Internet, and wired / wireless communication and bus communication with various devices such as external databases, editing apparatuses, and information processing apparatuses. In the case of this example, the communication unit 80 communicates with the camera 1 and the rendering engine 3 via, for example, the network switch 2.

[0045] The input / output interface 75 is also connected with a drive 81 as needed, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted. By the drive 81, video data, various computer programs, etc. can be read from the removable recording medium 82. The read data is stored in the storage unit 79, or the video and audio included in the data are output by the display unit 77 and the audio output unit 78. Also, computer programs, etc. read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0046] In this information processing apparatus 7, for example, software for the processing of the present embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in the ROM 72, the storage unit 79, or the like in advance.

[0047] Each device such as the camera 1 and the rendering engine 3 may also have a hardware configuration as shown in FIG. 3.

[0048] <3. Parameters> Here, the parameters used in the SLAM process and their initialization will be described. The following three parameters are listed as the parameters to be initialized at the start of SLAM. - iTc parameter - IMU parameter - Camera scale

[0049] In order to initialize these parameters, as an initialization behavior, it is necessary to keep the sensor unit 10 or the camera 1 attached with the sensor unit 10 stationary, move it in a specific direction, or rotate it. In the present disclosure, "stationary" is also regarded as one of the initialization behaviors. In the UI process described later, the initialization is efficiently completed mainly by presenting the initialization behavior to the user.

[0050] First, the iTc parameter will be described. The iTc parameter is a parameter of the positional relationship between the IMU 12 and each monocular camera 11. "i" represents the IMU, "c" represents the camera, and "T (Translation)" represents the homogeneous transformation matrix. In the sensor unit 10, the positions of the monocular cameras 11a, 11b, 11c, 11d, 11e and the IMU 12 are fixed, but it is a parameter indicating which direction each monocular camera 11 is facing with respect to the IMU 12 at the center of the unit, and specifically, it is a parameter indicating relative position and rotation information.

[0051] For a single IMU 12, there are five iTc parameters representing the relative positions of the monocular cameras 11a, 11b, 11c, 11d, and 11e, as follows: • Position iPca and rotation iRca values ​​for monocular camera 11a as seen from IMU 12 • Position iPcb and rotation iRcb values ​​for monocular camera 11b as seen from IMU 12 • Position iPcc and rotation iRcc values ​​for monocular camera 11c as seen from IMU 12 • Position iPcd and rotation iRcd values ​​for monocular camera 11d as seen from IMU 12 • Position iPce and rotation iRce values ​​for monocular camera 11e as seen from IMU 12

[0052] The position value iPca of the monocular camera 11a is the position offset Px, Py, and Pz in the x, y, and z axes relative to the monocular camera 11 as seen from the IMU 12. The rotation value iRca of the monocular camera 11a is the rotation offset R00, R01, R02, R10, R11, R12, R20, R21, and R22 relative to the monocular camera 11 as seen from the IMU 12 (a rotation matrix with 9 components in a 3x3 grid).

[0053] Similarly, the positions (iPcb, iPcc, iPcd, iPcee) of the other monocular cameras 11b, 11c, 11d, and 11e are the values ​​of the position offsets Px, Py, and Pz in the x, y, and z axes from the IMU 12, and the rotations (iRcb, iRcc, iRcd, iRcee) are the values ​​of the rotation offsets R00, R01, R02, R10, R11, R12, R20, R21, and R22 from the IMU 12.

[0054] These parameters are collectively referred to as iTc parameters. Since the positions of the IMU 12 and each monocular camera 11 within the sensor unit 10 are determined by design, the values ​​of the iTc parameters are ideally fixed values ​​(design values). However, there are assembly tolerances for the monocular camera 11 and IMU 12 in each sensor unit 10. In addition, the position and orientation may shift slightly if the sensor unit 10 is dropped or subjected to impact during use by the user. Furthermore, positional shifts can occur when disassembling or replacing parts during maintenance. For this reason, it is preferable to estimate subtle changes due to assembly tolerances and changes over time by initializing the system each time SLAM processing is performed, in order to improve the accuracy of position estimation by SLAM.

[0055] For initializing the iTc parameters, it is desirable to tilt the entire sensor unit 10 by several tens of degrees in the pan and tilt directions.

[0056] Figure 4 shows the initialization of the iTc parameters for one monocular camera 11 (for example, monocular camera 11a), and Figure 5 shows the initialization of another monocular camera 11 (for example, monocular camera 11b). The horizontal axis in Figures 4 and 5 represents time. The upper part of Figure 4 shows the position offsets Px, Py, and Pz as position iPca, and the lower part shows the rotation offsets as rotation iRca, expressed as Euler angles Ex, Ey, and Ez in the x, y, and z axes. Similarly, the upper part of Figure 5 shows the position offsets Px, Py, and Pz as position iPcb, and the lower part shows the rotation offsets Ex, Ey, and Ez, expressed as Euler angles for rotation iRcb.

[0057] The estimation method involves independently estimating the self-position of both the IMU 12 and the monocular camera 11, and then comparing these estimated values ​​to determine the offset. In Figures 4 and 5, the thick solid lines show the estimated values, and the bands surrounding the thick lines (right-sloping lines, left-sloping lines, and dotted areas) represent the accuracy of the estimated values. Initialization is considered complete when the accuracy converges to a certain value or less. For example, this type of iTc parameter initialization is performed for each of the five monocular cameras 11.

[0058] Next, we will explain the IMU parameters. The output values ​​(angular velocity, acceleration) of the IMU12 contain noise, and these values ​​are subject to individual differences due to manufacturing variations and changes over time due to temperature changes. Therefore, the accuracy of SLAM can be improved by estimating the amount of noise each time and setting the IMU bias value accordingly.

[0059] In other words, the IMU parameters are bias values ​​(IMU bias) that estimate the noise component to the detected angular velocity and acceleration values, and there are bias values ​​for all three axes, x, y, and z, for both angular velocity and acceleration.

[0060] Specifically, the IMU12 outputs gyro detection values ​​(angular velocity detection values) Gx, Gy, Gz in the x, y, and z axes, and acceleration detection values ​​Ax, Ay, Az in the x, y, and z axes. When these values ​​are stable, the gyro bias bGx, bGy, bGz and acceleration bias bAx, bAy, bAz are determined.

[0061] The gyroscopic biases bGx, bGy, and bGz are collectively referred to as gyroscopic parameters. The acceleration biases bAx, bAy, and bAz are collectively referred to as acceleration parameters. IMU parameters are a collective term for gyroscopic parameters and acceleration parameters.

[0062] For initializing the gyro biases bGx, bGy, and bGz, it is desirable to keep the device stationary for about 10 seconds. For initializing the acceleration biases bAx, bAy, and bAz, it is desirable to tilt the device by about 90 degrees in the pan, tilt, and roll directions.

[0063] Figure 6 shows the initialization of the gyro biases bGx, bGy, and bGz. The horizontal axis represents time, and the vertical axis represents the gyro bias value.

[0064] The 3-axis gyroscope sensor in the IMU12 measures the angular velocity in each of the three axes. Therefore, the sensor output should be zero in a completely stationary state. However, gyroscope sensors have various error components due to individual device variations and temperature changes. Among the various errors, the bias error, which adds a certain offset to the sensor output, has a significant impact on SLAM accuracy.

[0065] To eliminate this, the bias error can be removed by keeping the gyro sensor (IMU12) stationary for a certain period and performing zero-point offset calibration. The SLAM filter calculates the estimated value of the bias component and the confidence level of the estimate. The three thick solid lines in Figure 6 are the estimated values ​​of the gyro biases bGx, bGy, and bGz, and the bands around each thick line (right-sloping line, left-sloping line, and dotted area) represent the confidence level of the estimate. The narrower the band becomes, the more confident the estimate is, and initialization is considered complete when the confidence level converges to or below a certain value.

[0066] Figure 7 shows the initialization of the acceleration biases bAx, bay, and bAz. The horizontal axis represents time, and the vertical axis represents the value of the acceleration bias. The three-axis acceleration sensor in the IMU12 measures acceleration in each of the three axes. Acceleration sensors also have various error components due to individual device variations and temperature changes, and bias errors, which superimpose a certain offset on the sensor output, have a significant impact on SLAM accuracy.

[0067] In the case of an acceleration sensor, the bias value can be estimated by tilting the acceleration sensor (IMU12) into various orientations and applying gravitational acceleration in each axis direction. The three thick solid lines in Figure 6 represent the estimated values ​​of the acceleration biases bAx, bAy, and bAz, and the bands around each thick line (right-sloping lines, left-sloping lines, and dotted areas) represent the confidence level of the estimates. The narrower the bands become, the more confident the estimates are, and initialization is considered complete when the confidence level converges to or below a certain value.

[0068] Next, let's explain camera scaling. SLAM estimates the amount of movement by capturing feature points in images taken with a monocular camera 11, but it is not possible to determine the 3D coordinates of feature points from a single image. By first acquiring images from three viewpoints in a time series, it is possible to calculate the 3D coordinates of the feature points, and thereafter, the self-position can be calculated using these points as a clue.

[0069] This point is explained in Figures 8 and 9. Figure 8 shows the case where a stereo camera is used in SLAM. When using a stereo camera with cameras 21 and 22, the depth (distance in the depth direction) of feature points (indicated by ●) can be determined using the parallax of the two cameras in each frame. Since the depth can be determined in one frame, the physical scale (actual scale) of the object 20 on the subject side can be determined. In other words, the depth of each feature point is known, and based on the distance between the two cameras 21 and 22, the distance between feature points and the size of the object 20 can be determined. In this way, the physical size of the feature points can be detected, so no special initialization is required.

[0070] On the other hand, SLAM using the monocular camera 11 cannot detect the depth of feature points with only one frame. Therefore, it is necessary to acquire images of multiple frames with parallax in the time series direction and use them to calculate the depth. This is the initialization of SLAM using the monocular camera 11. For example, as shown in Figure 9A, the camera 23 is translated from time t0 to time t1 and images are taken, and the depth of feature points (indicated by ●) in each frame is calculated. The translational movement distance depends on the size of the studio where the performance area 8 is located, for example as shown in Figure 1, but is approximately several tens of centimeters to 1 meter. Arc-shaped movement as shown in Figure 9B is also acceptable. However, Figure 9C shows rotation at the same position, and the depth of feature points cannot be calculated with this type of behavior.

[0071] Thus, as an initialization behavior, it is desirable to translate the monocular camera 11 in a direction perpendicular to its orientation. This allows for the calculation of the depth of feature points and the determination of their three-dimensional coordinates. In other words, the physical scale of the feature points, such as the depth of feature points and the scale between feature points, in the three-dimensional space where the camera 1 equipped with the sensor unit 10 is located becomes the camera scale.

[0072] Such camera scales can be determined for each of the monocular cameras 11a, 11b, 11c, 11d, and 11e. For explanatory purposes, the camera scales cma, cmb, cmc, cmd, and cme will be used to represent the monocular cameras 11a, 11b, 11c, 11d, and 11e, respectively. The camera scale initialization has two states: completed and not completed.

[0073] <4. UI Processing> The UI processing when initializing the iTc parameters, IMU parameters, and camera scale as part of SLAM initialization will be explained below.

[0074] Figures 10 to 16 show UI screens 30 displayed, for example, on the display unit 77 in the information processing device 7, with each figure showing an example of changes in the displayed content. Figures 17 to 19 are flowcharts of processing examples of the control unit 71 (UI processing unit 71b) for displaying the UI screen 30. The processing examples shown in Figures 17 to 19 will be explained with reference to the UI screen 30.

[0075] Figure 17 shows the UI processing for initialization that the control unit 71 executes in response to the activation of SLAM processing. When SLAM processing is activated by the SLAM processing unit 71a, the control unit 71 starts the processing shown in Figure 17.

[0076] In step S100, the control unit 71 starts displaying the UI screen 30 for initialization in the display unit 77, for example. The UI screen 30 is a screen having a 3D viewer 31 and a text area 32, as shown in Figure 10, for example.

[0077] The 3D viewer 31 displays a camera-equivalent image, which is a representation of the monocular camera 11, on an image that simulates the real-world space where the camera 1 is located, for example, the three-dimensional space within a studio. Here, a triangular polygon 33 is used as an example of the camera-equivalent image. The triangular polygon 33 is displayed in the 3D viewer 31 to show the position and orientation (shooting direction) of the monocular camera 11 in the actual three-dimensional space.

[0078] Note that five triangular polygons 33 are displayed, one for each of the five monocular cameras 11a, 11b, 11c, 11d, and 11e. However, to avoid making the diagrams too complex, only two are shown in Figures 10 to 16: triangular polygon 33a representing monocular camera 11a and triangular polygon 33b representing monocular camera 11b. For example, triangular polygon 33a indicates the position and orientation of monocular camera 11a, and triangular polygon 33b indicates the position and orientation of monocular camera 11b. Triangular polygons 33c, 33d, and 33e for the other monocular cameras 11c, 11d, and 11e are not shown, but it should be understood that they are actually displayed.

[0079] Furthermore, although the triangular polygon 33 is simply shown as a triangle in the diagram, it would be better to represent the shooting direction of the monocular camera 11 by adding a color to one of the sides, a line width, or an additional image.

[0080] In the figure, the triangular polygon 33 is shown with a dashed line, which indicates that the triangular polygon 33 is displayed in the first display mode. It may be displayed with a dashed line as shown in the figure, or it may be a predetermined color. The display mode of the triangular polygon 33 is changed to indicate whether the initialization of the camera scale and iTc parameters is not yet complete or complete. The line type, color, etc. may be changed, or the shape, size, etc. of the camera equivalent image may be changed. In addition, for monocular cameras 11 in which the initialization of all necessary parameters has been completed, the display of the camera equivalent image may be erased. In other words, the display mode of the camera equivalent image indicates the initialization status by changing one or a combination of the line type, color, shape, size, display / hide, etc.

[0081] For example, a radar chart 34 is displayed on the UI screen 30. In the diagram, it is displayed superimposed on the 3D viewer 31, but this is just one example, and it may also be displayed outside of the 3D viewer 31.

[0082] The radar chart 34 is, for example, a hexagonal chart, and each vertex is associated with the gyroscope and acceleration in the x, y, and z axes, displaying Gx, Gy, Gz for the gyroscope and Ax, Ay, Az for the acceleration. This radar chart 34 indicates whether the initialization of the gyroscope biases bGx, bGy, bGz and the acceleration biases bAx, bAy, bAz is incomplete or complete. For example, the display of a vertex position indicates the completion of initialization.

[0083] The text area 32 displays the items "status" and "motion". "status" indicates whether initialization is in progress or complete. At the start, it displays "Initializing" to indicate that initialization is in progress. "motion" indicates the initialization behavior desired by the user.

[0084] From the processing in step S100, the control unit 71 starts displaying the UI screen 30 as described above, and first in step S101, it performs control to display a stationary instruction. Specifically, as shown in Figure 10, it displays a message instructing the system to stay still, such as "Keep stationary" in the "motion" item of the text area 32. The 3D viewer 31 also displays a stationary instruction image 35 that instructs the system to perform initialization, such as a mark indicating that the system is stationary.

[0085] In the control unit 71, SLAM initialization is performed in parallel with UI processing by the function of the SLAM processing unit 71a. When the user recognizes the stationary instruction on the UI screen 30 and stops the sensor unit 10 as an initialization behavior, the initialization of the gyro biases bGx, bGy, and bGz in SLAM processing proceeds. For example, if the sensor unit 10 remains stationary for about 10 seconds, the initialization of each of the gyro biases bGx, bGy, and bGz will be performed.

[0086] In step S102, the control unit 71 checks whether the initialization of at least one of the gyro parameters, which are gyro biases bGx, bGy, and bGz, has been completed. If it is not the timing for completion of initialization, the control unit 71 returns to step S101 and continues to control the display of the static instruction image 35.

[0087] When the initialization of any one of the gyro biases bGx, bGy, and bGz is complete, the control unit 71 proceeds from step S102 to step S103 and updates the display of the UI screen 30. In this case, the display is updated to show the parameter that has been initialized. Specifically, the peaks of the gyro items (Gx, Gy, Gz) in the radar chart 34 are shown. For example, if the initialization of the gyro bias bGx is complete, the control unit 71 updates the display so that the peak of "Gx" in the radar chart 34 is clearly indicated.

[0088] In step S104, the control unit 71 determines whether the initialization of all gyro biases bGx, bGy, and bGz has been completed. If not all have been completed, the control unit 71 returns to step S101 and continues to control the display of the static instruction image 35. Then, once the initialization of one or more other gyro biases has been completed, the system proceeds from step S102 to step S103 accordingly and updates the display of the radar chart 34.

[0089] Therefore, while the user keeps the sensor unit 10 stationary, the gyro parameters (gyro biases bGx, bGy, bGz) are initialized, and the completion of initialization is indicated in the radar chart 34. For example, Figure 10 shows the state in the radar chart 34 where the initialization of all gyro biases bGx, bGy, and bGz has been completed.

[0090] At a certain point in step S104, if the control unit 71 determines that the initialization of the gyro biases bGx, bGy, and bGz is complete, it proceeds to step S111 and sets the translational direction for the initialization of the camera scales cma, cmb, cmc, cmd, and cme.

[0091] For initializing the camera scale, it is desirable to translate the monocular camera 11 in a direction approximately perpendicular to its shooting direction. However, each of the five monocular cameras 11a has a different shooting direction (orientation). Therefore, the translation direction is set sequentially. For example, the control unit 71 selects one monocular camera 11 whose camera scale initialization is not yet complete and sets the direction perpendicular to the shooting direction of that monocular camera 11 as the translation direction.

[0092] However, in this case, if there is a translational direction in which the camera scales of multiple monocular cameras 11 can be initialized simultaneously, that direction may be prioritized for selection. An example of the processing in step S111 in that case is shown in Figure 18.

[0093] The control unit 71 determines whether to maintain the direction instructed in step S140. This is a necessary process after a translation instruction has been issued; if no instruction has been issued yet, the control unit 71 proceeds to step S141. Step S140 will be described later. In step S141, the control unit 71 determines whether there is a translation direction that can initialize multiple camera scales that have not yet been initialized. If there is a translation direction that can initialize multiple camera scales, the control unit 71 proceeds to step S142 and selects, for example, the translation direction that has the greatest potential to initialize the largest number of camera scales that have not yet been initialized.

[0094] For example, assuming the arrangement and orientation of the monocular camera 11 as shown in Figure 2, the direction indicated by the dashed line HP1 is perpendicular to the respective shooting directions of the monocular cameras 11a, 11c, and 11e. In other words, translating in the direction of the dashed line HP1 may allow all three camera scales cma, cmc, and cme to be initialized at once. Similarly, the direction indicated by the dashed line HP2 is perpendicular to the respective shooting directions of the monocular cameras 11a, 11b, 11c, and 11d. In other words, translating in the direction of the dashed line HP2 may allow all four camera scales cma, cmb, cmc, and cmd to be initialized at once. In such cases, the translation direction is set as the direction indicated by the dashed line HP2.

[0095] If in step S141 it is determined that there are no translational directions in which multiple camera scales can be initialized, the control unit 71 proceeds to step S143, selects one monocular camera 11 whose camera scale initialization is not yet complete, and sets the direction perpendicular to the shooting direction of that monocular camera 11 as the translational direction.

[0096] For example, once the translation direction is set using the above process, the control unit 71 proceeds to step S112 in Figure 17, and displays a translation instruction image 36 on the 3D viewer 31 as an image to instruct the initialization behavior, as shown in Figure 11. That is, it makes it possible to instruct the user on the translation direction set in step S111. For example, Figure 11 shows an example where an arrow image is displayed as the translation instruction image 36 to represent the translation direction.

[0097] Following this translation instruction image 36, the user will move the sensor unit 10 in the corresponding direction. In particular, displaying the translation instruction image 36 with arrows on the 3D viewer 31 makes it easy for the user to understand how to move the sensor unit 10.

[0098] In the control unit 71, during SLAM initialization by the SLAM processing unit 71a, while the user recognizes the translation instruction on the UI screen 30 and translates the sensor unit 10 as an initialization behavior, the camera scale initialization proceeds. Depending on the currently instructed translation, it is possible to initialize one or more of the camera scales cma, cmb, cmc, cmd, and cme.

[0099] In step S113, the control unit 71 checks whether the initialization of one or more of the camera scales cma, cmb, cmc, cmd, and cme has been completed. If it is not the initialization completion time, the control unit 71 returns to step S112 and continues to control the display of the translation instruction image 36.

[0100] When at least one of the camera scales cma, cmb, cmc, cmd, or cme has been initialized, the control unit 71 proceeds from step S113 to step S114 and performs a display update process for the UI screen 30. In this case, the triangular polygon 33 representing the monocular camera 11 for which the camera scale has been initialized is updated to the second display mode. Figure 11 shows the state in which the triangular polygon 33a has been changed from a dashed line (first display mode) to a solid line (second display mode) after the camera scale cma has been initialized.

[0101] In step S115, the control unit 71 determines whether the initialization of all camera scales cma, cmb, cmc, cmd, and cme has been completed. If not all have been completed, the control unit 71 returns to step S111 to set the translation direction. For example, the next translation direction is set by the process shown in Figure 18. However, in step S140, the control unit 71 determines whether to maintain the translation instruction image 36 that has been displayed up to this point. For example, if there are camera scales that have the potential to be initialized but have not yet been completed by the translation indicated by the previously displayed translation instruction image 36, and the predetermined time limit has not elapsed, the instruction is maintained, the process shown in Figure 18 is completed, and the process proceeds to step S112. As a result, the previous translation instruction image 36 is maintained. If it is not determined that the indicated direction should be maintained, that is, if all parameters that can be initialized in that translation direction have been initialized, or if time has run out, the process proceeds to step S141 to set a new translation direction.

[0102] If a new translation direction is set, the control unit 71 controls the 3D viewer 31 to display a translation instruction image 36 in step S112 of Figure 17 and to issue a translation instruction. For example, the new direction is displayed as the translation instruction image 36, as shown by the arrow image in Figure 12, prompting the user to translate.

[0103] Subsequently, the initialization completion timing in step S113 is confirmed, and the display update in step S114 is performed when the initialization of one or more camera scales is completed. Figure 12 shows the state in which, at a certain point, the camera scale cmb has been initialized, and the triangular polygon 33b has been changed from a dashed line (first display mode) to a solid line (second display mode).

[0104] If, at a certain point in step S115, the control unit 71 determines that the initialization of all camera scales cma, cmb, cmc, cmd, and cme has been completed, the control unit 71 proceeds to step S121 and sets the rotation direction for the initialization of acceleration parameters (acceleration bias bAx, bAy, bAz) and iTc parameters (position iPca, iPcb, iPcc, iPcd, iPce, rotation iRca, iRcb, iRcc, iRcd, iRce).

[0105] For initializing acceleration parameters, it is desirable to rotate the sensor unit 10 by about 90 degrees in the pan, tilt, and roll directions. For initializing iTc parameters, it is desirable to rotate the sensor unit 10 by about 5 degrees in the pan and tilt directions. Therefore, rotations in various directions are set sequentially. For example, the control unit 71 selects an acceleration parameter and an iTc parameter whose initialization is not yet complete, and sets the rotation direction to one that is suitable for initializing that parameter.

[0106] However, even in this case, if there is a rotation direction in which the initialization of multiple parameters corresponding to the acceleration parameter and iTc parameter can be performed simultaneously, it is preferable to select that direction first. An example of the processing in step S121 in that case is shown in Figure 19.

[0107] The control unit 71 determines whether to maintain the rotation direction instructed in step S150. This is a necessary process after a rotation instruction has been issued; if no instruction has been issued yet, the control unit 71 proceeds to step S151. Step S150 will be described later. In step S151, the control unit 71 determines whether there is a rotation direction in which initialization of multiple parameters among the acceleration parameters and iTc parameters that have not yet been completed can be performed. If there is a rotation direction in which initialization of multiple parameters can be performed, the control unit 71 proceeds to step S152 and selects, for example, the rotation direction that has the greatest potential to initialize the largest number of parameters that have not yet been completed.

[0108] If step S151 determines that there are no rotation directions in which multiple parameters can be initialized, the control unit 71 proceeds to step S153, selects one of the parameters that has not yet been initialized, and sets the rotation direction for initializing that parameter.

[0109] For example, once the rotation direction is set using the above process, the control unit 71 proceeds to step S122 in Figure 17, and displays a rotation instruction image 37 on the 3D viewer 31 as an image to instruct the initialization behavior, as shown in Figure 13. In other words, it allows the user to be instructed on the rotation direction set in step S121. For example, Figure 13 shows an example where a curved arrow image is displayed as the rotation instruction image 37 to represent the rotation direction.

[0110] Following this rotation instruction image 37, the user rotates the sensor unit 10 in the corresponding direction. In particular, displaying the rotation instruction image 37 with arrows on the 3D viewer 31 makes it easy for the user to understand how to move the sensor unit 10.

[0111] In the control unit 71, during SLAM initialization by the SLAM processing unit 71a, while the user recognizes the rotation instruction on the UI screen 30 and rotates the sensor unit 10 as an initialization behavior, the initialization of acceleration parameters and iTc parameters proceeds. Depending on the currently instructed rotation, it becomes possible to initialize one or more parameters corresponding to the acceleration parameters and iTc parameters.

[0112] In step S123, the control unit 71 checks whether the initialization of one or more of the acceleration parameters and iTc parameters has been completed. If it is not the timing for completion of initialization, the control unit 71 returns to step S122 and continues to control the display of the rotation instruction image 37.

[0113] When at least one of the parameters corresponding to the acceleration parameter and iTc parameter has been initialized, the control unit 71 proceeds from step S123 to step S124 and performs the display update process for the UI screen 30.

[0114] For example, when the initialization of any of the acceleration parameters, namely acceleration biases bAx, bAy, and bAz, is completed, the control unit 71 proceeds from step S123 to step S124 and updates the display so that the completion of initialization is represented in the radar chart 34. For example, when the initialization of acceleration bias bAx is completed, as shown in Figure 13, the control unit 71 performs a display update process so that the peak of "Ax" in the radar chart 34 is clearly indicated.

[0115] Furthermore, the iTc parameters determine the completion timing for each monocular camera 11. For example, as mentioned above, the iTc parameters for monocular camera 11a are position iPca and rotation iRca. When the control unit 71 has finished initializing position iPca and rotation iRca, it determines that the initialization of the iTc parameters for monocular camera 11a is complete, and proceeds from step S123 to step S124 to update the display. In this case, the display mode of the triangular polygon 33a representing the monocular camera 11a for which the iTc parameters have been initialized is updated. Figure 13 shows the state in which the triangular polygon 33a has been changed from a solid line (second display mode) to a black fill (third display mode).

[0116] In step S125, the control unit 71 determines whether all initialization of acceleration parameters and iTc parameters has been completed. If not all are completed, the control unit 71 returns to step S121 to set the rotation direction. For example, the next rotation direction is set by the process shown in Figure 19. However, in step S150, the control unit 71 determines whether to maintain the rotation instruction image 37 that has been displayed up to this point. For example, if there are parameters that could be initialized but have not yet been completed by the rotation indicated by the previously displayed rotation instruction image 37, and the predetermined time limit has not elapsed, the instruction is maintained, the process shown in Figure 19 is completed, and the process proceeds to step S122. As a result, the rotation instruction image 37 up to this point is maintained. If it is not determined that the indicated direction should be maintained, that is, if all parameters that can be initialized in that rotation direction have been initialized, or if time has run out, the process proceeds to step S151 to set a new rotation direction.

[0117] If a new rotation direction is set, the control unit 71 controls the 3D viewer 31 to display a rotation instruction image 37 in step S122 of Figure 17 and issue a rotation instruction. For example, the rotation instruction image 37 displays the new rotation direction, as shown by the arrow image in Figure 14, prompting the user to rotate.

[0118] Subsequently, the initialization completion timing in step S123 is confirmed, and the display update in step S124 is performed when the initialization of one or more camera scales is completed. Figure 14 shows the state in which the display mode of the triangular polygon 33b is updated to black (third display mode) because the initialization of the iTc parameter for the monocular camera 11b has been completed at a certain point. It also shows the state in which the vertex of "Ay" in the radar chart 34 is explicitly shown in accordance with the completion of the initialization of the acceleration bias bAy as an acceleration parameter.

[0119] Furthermore, Figure 15 shows the state in which the peak of "Az" in the radar chart 34 is explicitly displayed, corresponding to the completion of the initialization of the acceleration bias bAz as an acceleration parameter.

[0120] Figure 15 shows that the initialization of the gyro parameters and acceleration parameters has been completed in the radar chart 34, and that all triangular polygons 33a and 33b are in the third display mode, indicating that the initialization of all IMU parameters, camera scale, and iTc parameters has been completed.

[0121] If the control unit 71 determines at step S126 that the initialization of all acceleration parameters and iTc parameters is complete, the process proceeds to step S130, where display control is performed so that, for example, "Initialization Complete" is displayed as shown in Figure 16. In Figure 16, all triangular polygons 33 are in the third display mode, and all radar charts 34 indicate that the initialization is complete. In addition, the "status" item in the text area 32 is updated to "Initialize Done," indicating that the initialization is complete.

[0122] <5. Summary and Modifications> The following effects can be obtained according to the above embodiments.

[0123] The control unit 71 of the information processing device 7 in this embodiment performs a process to display a behavior guide that presents to the user the initialization behavior to be given to the sensor unit 10 for initializing the IMU parameters related to the angular velocity detection value and acceleration detection value output from the sensor unit 10, which is equipped with an IMU 12 and a plurality of monocular cameras 11. For example, as a behavior guide display, it displays a static instruction image 35 that shows the initialization behavior for initializing the gyro parameters, and a rotation instruction image 37 that shows the initialization behavior for initializing the acceleration parameters. The control unit 71 also performs a process to display, for example, a radar chart 34 as an initialization status display that shows whether the initialization of the IMU parameters is not yet complete or has been completed. The user just needs to move the sensor unit 10 and the camera 1 to which the sensor unit 10 is attached according to the behavior guide display displayed on the UI screen 30 when the SLAM process is started. The user can also visually confirm the initialization status as a result. This makes it possible to easily complete the initialization of the output values ​​of the IMU 12 and enable highly accurate self-position estimation.

[0124] In this embodiment, the control unit 71 performs a process to present to the user, as a behavior guide display, the initialization behavior to be given to the sensor unit 10 for initializing positional relationship parameters (iTc parameters) that indicate the positional relationship between the IMU 12 and each of the multiple monocular cameras 11. For example, as a behavior guide display, it displays a rotation instruction image 37 that shows the initialization behavior for initializing the iTc parameters. The control unit 71 also performs a process to present, as an initialization status display, the status of whether the initialization of the positional relationship parameters (iTc parameters) is not yet complete or has been completed. For example, the first or second display mode of the triangular polygon 33 indicates that initialization is not yet complete, and the third display mode indicates that initialization is complete. On the UI screen 30, the initialization behavior related to the iTc parameters is also displayed, and the initialization status is also displayed, for example, in the display mode of the triangular polygon 33. This allows the user to perform the appropriate initialization behavior without being particularly aware of the iTc parameters. As an indicator of whether the initialization of iTc parameters is not yet complete or has been completed, for example, when the initialization of iTc parameters for all monocular cameras 11 has been completed, the completion of initialization may be indicated by some mark, symbol, character, etc.

[0125] In this embodiment, the control unit 71 performs a process to present to the user, as a behavior guide display, the initialization behavior to be given to the sensor unit 10 for initializing the camera scale used for calculating the three-dimensional coordinates of feature points of the subject for each of the multiple monocular cameras 11. For example, as a behavior guide display, it displays a translation instruction image 36 that shows the initialization behavior for initializing the camera scale. The control unit 71 also performs a process to present, as an initialization status display, whether the initialization of the camera scale is not yet complete or has been completed. For example, the first display mode of the triangular polygon 33 indicates that initialization is not yet complete, and the second display mode indicates that initialization is complete. On the UI screen 30, the initialization behavior related to the camera scale (cma, cmb, cmc, cmd, cme) is also displayed, and the initialization status is also displayed, for example, by the display mode of the triangular polygon 33. This allows the user to perform the appropriate initialization behavior without being particularly aware of the camera scale. As an initialization status display indicating whether the initialization of the camera scale is not yet complete or has been completed, for example, when the initialization of the camera scale for all monocular cameras 11 is completed, some mark, symbol, character, etc. may be used to indicate that initialization is complete.

[0126] In this embodiment, the IMU parameters are bias values ​​corresponding to the amount of noise in the angular velocity and acceleration detection values ​​output from the IMU 12. In other words, the IMU parameters are gyro biases bGx, bGy, bGz and acceleration biases bAx, bAy, bAz. By calculating these parameters while the gyro detection values ​​Gx, Gy, Gz and acceleration detection values ​​Ax, Ay, Az are stable, the influence of noise on the IMU 12's detection values ​​can be eliminated, thereby improving SLAM accuracy.

[0127] In this embodiment, an example was given in which the control unit 71 performs processing to display, as a behavior guide display, multiple camera-equivalent images representing each of the multiple monocular cameras 11, and behavior instruction images indicating the direction or state of the initialization behavior given by the user to the sensor unit 10. For example, as shown in Figures 10 to 16, the control unit 71 displays triangular polygons 33 representing each monocular camera 11 as camera-equivalent images, and displays a stationary instruction image 35, a translation instruction image 36, and a rotation instruction image 37 as behavior instruction images. This allows the user to intuitively understand the initialization behavior and also makes it easier to understand the relationship between each monocular camera 11 and the initialization behavior.

[0128] In this example, the UI screen 30 displays all the triangular polygons 33 corresponding to the multiple monocular cameras 11 simultaneously. However, it is also possible to display only the triangular polygons 33 of the monocular cameras 11 that can be initialized by that translation while, for example, one translation instruction image 36 is being displayed. In other words, it is possible to always display all the triangular polygons 33 of the monocular cameras 11, but it is also possible to use a UI method that temporarily displays some of the triangular polygons 33, or displays them one by one. Furthermore, while triangular polygons 33 were given as an example of a camera-equivalent image, other display modes are also possible.

[0129] In this embodiment, an example was given in which the control unit 71 processes the camera equivalent image and behavior instruction image to display them on an image that mimics a three-dimensional space. As a 3D viewer 31, by displaying triangular polygons 33, translation instruction images 36, rotation instruction images 37, etc., arranged in an overhead view on a background that mimics a three-dimensional space, it becomes easier for the user to grasp the direction of the initialization behavior.

[0130] In this embodiment, an example is given in which the control unit 71 prioritizes selecting an initialization behavior that allows for the initialization of multiple parameters and displays it using an initialization instruction image. For example, as shown in Figures 18 and 19, if there is an initialization behavior that allows for the initialization of multiple parameters with a single translation or rotation, the control unit prioritizes selecting such a translation or rotation and instructs the user using a translation instruction image 36 or a rotation instruction image 37. This streamlines the series of initialization behaviors and reduces the burden on the user.

[0131] In this embodiment, an example was given in which the control unit 71 performs a process to display the initialization status of each of the multiple IMU parameters individually, indicating whether the initialization is not yet complete or has been completed. For example, the radar chart 34 shown in Figures 10 to 16 indicates whether the initialization of each of the multiple IMU parameters, such as IMU biases bGx, bGy, bGz, bAx, bAy, and bAz, is not yet complete or has been completed. This allows the user to recognize the progress of initialization for each individual IMU parameter. Furthermore, the user can understand which IMU parameters are being initialized in relation to the initialization behavior.

[0132] In this embodiment, an example is given in which the control unit 71 performs a process to display the initialization status for each of the multiple monocular cameras 11, indicating whether the initialization of the camera scale is not yet complete or has been completed. For example, as shown in Figures 10 to 12, the display mode of the triangular polygon 33 changes to individually indicate whether the initialization of the camera scales cma, cmb, cmc, cmd, and cme for each monocular camera 11 is not yet complete or has been completed. This allows the user to recognize the progress of initialization for each camera scale of each monocular camera 11. The user can also understand which camera scale of which monocular camera 11 is being initialized in relation to the initialization behavior.

[0133] In this embodiment, the control unit 71 performs a process to display the initialization status for each of the multiple monocular cameras 11, indicating whether the initialization of the positional relationship parameters (iTc parameters) is not yet complete or has been completed. For example, as shown in Figures 12 to 14, the display pattern of the triangular polygon 33 changes to individually indicate whether the initialization of the iTc parameters (iPca and iRca, iPcb and iRcb, iPcc and iRcc, iPcd and iRcd, iPce and iRce) for each monocular camera 11 is not yet complete or has been completed. This allows the user to recognize the progress of initialization for each iTc parameter of each monocular camera 11. The user can also understand which monocular camera 11's iTc parameters are being initialized in relation to the initialization behavior.

[0134] In this embodiment, an example is given in which the control unit 71 performs a process to display characters indicating the direction or state of the initialization behavior that the user provides to the sensor unit 10 as a behavior guide display. For example, in the character area 32 shown in Figures 10 to 14, characters indicating the direction or state of the initialization behavior, such as "Keep stationary" or "Translation left and right," are displayed as "motion." This makes it easier for the user to understand the content of the requested initialization behavior.

[0135] In this embodiment, an example is given in which the control unit 71 performs a process to display text indicating that the initialization of all parameters has been completed as an initialization status indicator. For example, as shown in Figure 16, the text area 32 displays text indicating the completion of initialization, such as "Initialize Done," as "status." This makes it easier for the user to recognize that the initialization is complete. In addition, the fact that all triangular polygons 33 in the 3D viewer 31 are in the third display mode and that all vertices in the radar chart 34 have reached their peaks also functions as an indicator that the initialization of all parameters has been completed, but for example, text or marks indicating the completion of initialization may be displayed on the 3D viewer 31.

[0136] In this embodiment, the control unit 71 provides an example of an initialization behavior to the sensor unit 10 for initializing the angular velocity parameters included in the IMU parameters, namely by indicating a stationary state. By keeping the sensor unit 10 stationary for about 10 seconds, the gyro sensor output (Gx, Gy, Gz) can be stabilized, and the gyro parameters (gyro bias bGx, bGy, bGz) can be appropriately initialized.

[0137] In this embodiment, the control unit 71 provides an example in which rotation is given to the sensor unit 10 as an initialization behavior for initializing the acceleration-related parameters included in the IMU parameters. By rotating the sensor unit 10 by about 90° in the pan, tilt, and roll directions, the acceleration sensor output (Ax, Ay, Az) can be stabilized and the acceleration parameters (acceleration bias bAx, bAy, bAz) can be appropriately initialized.

[0138] In this embodiment, an example was given in which the control unit 71 provides translation as the initialization behavior to the sensor unit 10 for initializing the camera scale. By translating the sensor unit 10 perpendicular to the orientation of the monocular camera 11, it is possible to obtain captured images from different viewpoints and appropriately initialize the camera scales cma, cmb, cmc, cmd, and cme.

[0139] In this embodiment, an example was given in which the control unit 71 provides rotation as an initialization behavior to the sensor unit 10 for initializing the positional relationship parameters. By rotating the sensor unit 10 relative to the orientation of the monocular camera 11, the positional relationship parameters (iTc parameters) can be appropriately initialized.

[0140] In this embodiment, the control unit 71 provides a behavior guide display to the user, first presenting stationary movement, then translation, and finally rotation. For example, as shown in Figure 17, the behavior guide display sequentially instructs stationary movement, translation, and rotation. This allows for the efficient initialization of all parameters with minimal burden on the user. In particular, by first bringing the sensor unit 10 to a stationary position, the gyro parameters can be initialized, and accurate gyro output can be used when initializing each parameter thereafter.

[0141] In this embodiment, the sensor unit 10 is attached to a camera 1 for image capture. The UI for initializing IMU parameters and other settings for the sensor unit attached to the camera 1 includes behavior guide displays and initialization status displays. This provides a suitable UI for applications such as a camera tracker in an imaging system 100.

[0142] In this embodiment, an example of the initialization UI was described using a sensor unit 10 equipped with an IMU 12 in the imaging system 100. However, this technology is not limited to such systems and can be broadly applied as a UI method for any device or system that performs some kind of processing using a sensor unit 10 equipped with an IMU 12.

[0143] The program of the embodiment is a program that causes a processor such as a CPU or DSP, or a device including such a processor, to execute the processes shown in Figures 17, 18, and 19 described above. Specifically, the program of the embodiment is a program that causes an information processing device to execute a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit 10 for initializing the IMU parameters related to the angular velocity detection value and acceleration detection value output from the sensor unit 10 equipped with an IMU 12 and a plurality of monocular cameras 11, and an initialization status display that shows whether the initialization of the IMU parameters is not yet complete or has been completed.

[0144] With such a program, the information processing device 7 described above can be implemented using various computer devices.

[0145] Such programs can be pre-recorded on storage media such as HDDs (hard disk drives) built into computer devices, or on ROM (remote-controlled memory) within microcomputers with CPUs. Furthermore, such programs can be temporarily or permanently stored (recorded) on removable storage media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. These removable storage media can be provided as so-called packaged software. In addition to being installed from removable storage media to personal computers, such programs can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.

[0146] Furthermore, such a program is suitable for providing a wide range of information processing devices 7 that perform UI processing as in the embodiment. For example, by downloading the program to personal computers, communication devices, mobile terminal devices such as smartphones and tablets, mobile phones, game devices, video devices, PDAs (Personal Digital Assistants), etc., these devices can be made to function as the information processing devices 7 of this disclosure.

[0147] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0148] The technology can also be configured as follows: (1) An information processing device comprising a control unit that performs the following: a process of causing the control unit to perform a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters related to angular velocity detection values ​​and acceleration detection values ​​output from a sensor unit equipped with an IMU and a plurality of monocular cameras; and a process of causing the control unit to perform an initialization status display that shows whether the initialization of the IMU parameters is not yet complete or has been completed. (2) The information processing device according to (1) above, wherein the control unit performs the following processes: as the behavior guide display, it presents to the user the initialization behavior to be given to the sensor unit for initializing positional relationship parameters that show the positional relationship between the IMU and each of the plurality of monocular cameras; and as the initialization status display, it presents whether the initialization of the positional relationship parameters is not yet complete or has been completed. (3) The information processing device according to (1) or (2) above, wherein the control unit performs the process of displaying to the user the initialization behavior to be given to the sensor unit for initializing the camera scale used for calculating the three-dimensional coordinates of feature points of the subject for each of the plurality of monocular cameras as the behavior guide display, and displays the status of whether the initialization of the camera scale is not yet finished or has been completed as the initialization status display. (4) The information processing device according to any one of (1) to (3) above, wherein the IMU parameter is a bias value corresponding to the amount of noise of the angular velocity detection value and acceleration detection value output from the IMU. (5) The information processing device according to any one of (1) to (4) above, wherein the control unit performs the process of displaying a plurality of camera equivalent images showing each of the plurality of monocular cameras and a behavior instruction image showing the direction or state of the initialization behavior given by the user to the sensor unit as the behavior guide display. (6) The information processing device according to (5) above, wherein the control unit performs the process of displaying the camera equivalent images and the behavior instruction image on an image that mimics a three-dimensional space. (7) The information processing apparatus according to (5) or (6) above, wherein the control unit prioritizes selecting an initialization behavior that allows for the initialization of multiple parameters and performs a process of displaying it using the behavior instruction image.(8) The information processing apparatus according to any one of (1) to (7) above, wherein the control unit performs the process of displaying, as the initialization status display, the status of whether initialization is not yet completed or completed for each of the multiple IMU parameters individually. (9) The information processing apparatus according to any one of (1) to (8) above, wherein the control unit performs the process of displaying, as the behavior guide display, a plurality of camera equivalent images showing each of the plurality of monocular cameras, and a behavior instruction image showing the direction or state of initialization behavior given to the sensor unit for initializing the camera scale used for calculating the three-dimensional coordinates of feature points of the subject, and performs the process of displaying, as the initialization status display, the status of whether initialization of the camera scale is not yet completed or completed for each of the plurality of monocular cameras. (10) The control unit performs the process of displaying, as the behavior guide display, a plurality of camera equivalent images showing each of the plurality of monocular cameras, and a behavior instruction image showing the direction or state of initialization behavior to be given to the sensor unit for initialization of positional relationship parameters showing the positional relationship between the IMU and each of the plurality of monocular cameras, and as the initialization status display, the information processing device according to any one of (1) to (9) above, which performs the process of displaying, for each of the plurality of monocular cameras, the status of whether the initialization of the positional relationship parameters is not yet finished or has been finished. (11) The control unit performs the process of displaying, as the behavior guide display, characters showing the direction or state of initialization behavior given to the sensor unit by the user. (12) The control unit performs the process of displaying, as the initialization status display, characters indicating that the initialization of all parameters has been completed. (13) The information processing apparatus according to any one of (1) to (12) above, wherein the control unit presents a stationary state as an initialization behavior to be given to the sensor unit for initializing the angular velocity parameter included in the IMU parameter.(14) The information processing device according to any one of (1) to (13) above, wherein the control unit presents rotation as an initialization behavior to be given to the sensor unit for initializing the acceleration parameters included in the IMU parameters. (15) The information processing device according to (3) or (9) above, wherein the control unit presents translation as an initialization behavior to be given to the sensor unit for initializing the camera scale. (16) The information processing device according to (2) or (10) above, wherein the control unit presents rotation as an initialization behavior to be given to the sensor unit for initializing the positional relationship parameters. (17) The information processing device according to any one of (1) to (16) above, wherein the control unit presents stationary first, then translation, and finally rotation as a behavior guide display to present to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters, positional relationship parameters indicating the positional relationship between the IMU and each of the plurality of monocular cameras, and camera scales used for calculating the three-dimensional coordinates of feature points of a subject for each of the plurality of monocular cameras. (18) The information processing device according to any one of (1) to (16) above, wherein the sensor unit is attached to a camera for capturing images. (19) An information processing method which causes the information processing device to perform a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters relating to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit equipped with an IMU and a plurality of monocular cameras, and an initialization status display that shows whether the initialization of the IMU parameters is not yet complete or has been completed. (20) A program which causes the information processing device to perform a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters relating to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit equipped with an IMU and a plurality of monocular cameras, and an initialization status display that shows whether the initialization of the IMU parameters is not yet complete or has been completed.

[0149] 1 Camera 7 Information Processing Unit 10 Sensor Unit 11, 11a, 11b, 11c, 11d, 11e Monocular Camera 12 IMU 30 UI Screen 31 3D Viewer 32 Text Area 33, 33a, 33b Triangular Polygon 34 Radar Chart 35 Static Instruction Image 36 Translational Instruction Image 37 Rotational Instruction Image 71 Control Unit 71a SLAM Processing Unit 71b UI Processing Unit 100 Shooting System

Claims

1. An information processing device comprising a control unit that performs the following: a process of causing the sensor unit, which is equipped with an IMU and multiple monocular cameras, to execute a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters related to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit; and a process of causing the sensor unit to execute an initialization status display that indicates whether the initialization of the IMU parameters is not yet complete or has been completed.

2. The information processing apparatus according to claim 1, wherein the control unit causes the user to be presented with initialization behavior to be given to the sensor unit for initializing positional relationship parameters that indicate the positional relationship between the IMU and each of the plurality of monocular cameras, as the behavior guide display, and performs the processing to show whether the initialization of the positional relationship parameters is not yet completed or has been completed, as the initialization status display.

3. The information processing apparatus according to claim 1, wherein the control unit causes the user to see, as a behavior guide display, the initialization behavior to be given to the sensor unit for initializing the camera scale used for calculating the three-dimensional coordinates of feature points of the subject for each of the plurality of monocular cameras, and the control unit performs a process to show, as an initialization status display, whether the initialization of the camera scale is not yet complete or has been completed.

4. The information processing apparatus according to claim 1, wherein the IMU parameter is a bias value corresponding to the amount of noise in the angular velocity detection value and acceleration detection value output from the IMU.

5. The information processing apparatus according to claim 1, wherein the control unit performs a process of displaying, as the behavior guide display, a plurality of camera equivalent images representing each of the plurality of monocular cameras and a behavior instruction image indicating the direction or state of the initialization behavior given by the user to the sensor unit.

6. The information processing apparatus according to claim 5, wherein the control unit performs a process of displaying the camera equivalent image and the behavior instruction image on an image that simulates a three-dimensional space.

7. The information processing apparatus according to claim 5, wherein the control unit prioritizes selecting an initialization behavior that allows for the initialization of multiple parameters and performs a process of displaying it using the behavior instruction image.

8. The information processing apparatus according to claim 1, wherein the control unit performs a process to display, as an initialization status display, the status of whether initialization is not yet complete or has been completed for each of the plurality of IMU parameters individually.

9. The information processing apparatus according to claim 1, wherein the control unit performs the process of displaying, as the behavior guide display, a plurality of camera equivalent images representing each of the plurality of monocular cameras, and a behavior instruction image for each of the plurality of monocular cameras that indicates the direction or state of the initialization behavior given to the sensor unit for initializing the camera scale used for calculating the three-dimensional coordinates of feature points of the subject, and performs the process of displaying, as the initialization status display, the status for each of the plurality of monocular cameras, whether the initialization of the camera scale is not yet completed or has been completed.

10. The information processing apparatus according to claim 1, wherein the control unit performs the following processing as the behavior guide display: a plurality of camera equivalent images showing each of the plurality of monocular cameras, and a behavior instruction image for each of the plurality of monocular cameras showing the direction or state of initialization behavior given to the sensor unit for initialization of positional relationship parameters showing the positional relationship between the IMU and each of the plurality of monocular cameras; and the information processing apparatus according to claim 1, wherein the control unit performs the following processing as the initialization status display: whether the initialization of the positional relationship parameters has not yet been completed or has been completed for each of the plurality of monocular cameras.

11. The information processing apparatus according to claim 1, wherein the control unit performs a process to display characters indicating the direction or state of the initialization behavior given by the user to the sensor unit as the behavior guide display.

12. The information processing apparatus according to claim 1, wherein the control unit performs a process to display characters indicating that the initialization of all parameters has been completed as an initialization status display.

13. The information processing apparatus according to claim 1, wherein the control unit presents a stationary state as an initialization behavior to be given to the sensor unit for initializing the angular velocity parameter included in the IMU parameter.

14. The information processing apparatus according to claim 1, wherein the control unit presents rotation as an initialization behavior to be given to the sensor unit for initializing the acceleration parameters included in the IMU parameters.

15. The information processing apparatus according to claim 3, wherein the control unit presents translation as an initialization behavior to be given to the sensor unit for initializing the camera scale.

16. The information processing apparatus according to claim 2, wherein the control unit presents rotation as an initialization behavior to be given to the sensor unit for initializing the positional relationship parameters.

17. The information processing apparatus according to claim 1, wherein the control unit presents to the user an initialization behavior to be given to the sensor unit for initializing the IMU parameters, positional relationship parameters indicating the positional relationship between the IMU and each of the plurality of monocular cameras, and the camera scale used for calculating the three-dimensional coordinates of feature points of a subject for each of the plurality of monocular cameras, by first presenting static, then translation, and finally rotation.

18. The information processing apparatus according to claim 1, wherein the sensor unit is attached to a camera for capturing images.

19. An information processing method comprising: causing an information processing device to perform a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters related to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit, which is equipped with an IMU and multiple monocular cameras; and causing the device to perform an initialization status display that indicates whether the initialization of the IMU parameters is not yet complete or has been completed.

20. A program that causes an information processing device to execute a behavior guide display that presents to the user the initialization behavior to be given to the sensor unit for initializing the IMU parameters related to angular velocity detection values ​​and acceleration detection values ​​output from the sensor unit equipped with an IMU and multiple monocular cameras, and a program that causes an initialization status display that indicates whether the initialization of the IMU parameters is not yet complete or has been completed.

Images