Information processing device and method, and information processing system

Abstract

The present disclosure relates to an information processing device and method, and an information processing system, with which it is possible to highly accurately calculate the camera position of event data and to improve the quality of 3D reconstruction. This information processing device comprises: an acquisition unit that acquires event data obtained by detecting, as an event, a change in luminance of an optical signal emitted by a marker, and an RGB image generated by capturing an image of a predetermined object; a decoding unit that demodulates the optical signal on the basis of the event data; an event camera pose calculation unit that calculates, from the demodulated optical signal, the camera pose of the camera when the predetermined event data is acquired; an RGB camera pose calculation unit that calculates, using the camera pose of the camera when the predetermined event data is acquired, a camera pose of the camera when the RGB image is acquired; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, the RGB image, and the camera pose at that time. The technology of the present disclosure can be applied to, for example, an information processing device or the like that generates a 3D reconstruction model of a predetermined object.

Description

Information processing apparatus and method, and information processing system

[0001] This disclosure relates to an information processing device and method, and an information processing system, and more particularly to an information processing device and method, and an information processing system that can calculate the camera position of event data with high accuracy and improve the quality of 3D reconstruction.

[0002] In recent years, many 3D reconstruction technologies, such as NeRF (Representing Scenes as Neural Radiance Fields for View Synthesis) and 3D Gaussian Splatting, which capture three-dimensional space to create free-viewpoint video, have been proposed, and the number of consumer-friendly applications is increasing.

[0003] In 3D reconstruction using NeRF, methods that use only low-frame-rate RGB cameras suffer from reduced accuracy in scenes where the RGB cameras move rapidly, due to the lack of inter-frame information from the RGB cameras. In contrast, Non-Patent Document 1 proposes a method that uses event data acquired by an event camera with high temporal resolution, which can improve upon the above-mentioned drawbacks.

[0004] Non-patent document 2 discloses a 3D reconstruction technique using NeRF, in which, when the object to be 3D reconstructed is photographed with a rolling shutter RGB camera, processing is performed to take into account the difference in the shooting time for each row.

[0005] Klenk, Simon, et al. "E-nerf: Neural radiance fields from a moving event camera." IEEE Robotics and Automation Letters 8.3 (2023): 1587-1594.Niu, Muyao, et al. "RS-NeRF: Neural Radiance Fields from Rolling Shutter Images." arXiv preprint arXiv:2407.10267 (ECCV2024).

[0006] In the technology described in Non-Patent Document 1, the camera position of event data is determined by interpolation of the camera position of an RGB camera, which is prone to errors when the actual camera movement is complex. A decrease in the accuracy of camera position estimation leads to a decrease in the quality of 3D reconstruction.

[0007] This disclosure is made in light of these circumstances and aims to improve the quality of 3D reconstruction by accurately calculating the camera position of event data.

[0008] The information processing device in the first aspect of this disclosure includes: an acquisition unit that acquires event data, which is obtained by detecting a change in the brightness of an optical signal emitted by a marker as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the optical signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated optical signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

[0009] The second aspect of the information processing method of this disclosure includes: an information processing device acquiring event data in which a change in the brightness of a light signal emitted by a marker is detected as an event, and an RGB image generated by photographing a predetermined object; demodulating the light signal based on the event data; calculating the camera pose of the camera at the time the predetermined event data was acquired from the demodulated light signal; calculating the camera pose of the camera at the time the RGB image was acquired using the camera pose of the camera at the time the predetermined event data was acquired; and generating a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

[0010] The information processing system in the third aspect of this disclosure comprises one or more markers having two or more light-emitting elements, one or more cameras capable of outputting at least one of event data or an RGB image, and an information processing device, the information processing device having an acquisition unit that acquires event data, which is obtained by detecting a change in the brightness of a light signal emitted by the one or more markers as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the light signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated light signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

[0011] In the first to third aspects of this disclosure, event data is obtained by detecting a change in the brightness of a light signal emitted by a marker as an event, and an RGB image is obtained by photographing a predetermined object. Based on the event data, the light signal is demodulated, the camera pose of the camera at the time the predetermined event data was obtained is calculated from the demodulated light signal, the camera pose of the camera at the time the RGB image was obtained is calculated using the camera pose of the camera at the time the predetermined event data was obtained, and a 3D reconstruction model of the predetermined object is generated using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

[0012] The information processing device of the first aspect of this disclosure can be realized by having a computer execute a program. The program to be executed by the computer in order to realize this information processing device can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

[0013] An information processing device may be an independent device or an internal block that constitutes a single device.

[0014] This is a diagram illustrating NeRF. This is a diagram illustrating NeRF. This is a diagram illustrating the method of Non-Patent Literature 1. This is a diagram showing an example of the comparison result between the camera pose obtained by the method of Non-Patent Literature 1 and the actual camera pose. This is a diagram illustrating the outline of this method. This is a block diagram showing an example configuration of an information processing system to which the technology of this disclosure is applied. This is a diagram illustrating optical wireless communication of a marker. Positive events and negative events and marker position information IDpos n This is a conceptual diagram showing (tj). Marker position information ID pos ０ (tk0) to IDpos3 (tk3) are used to determine the camera pose at time Tk. VLC This diagram illustrates the process of calculating (Tk). Each event data EV(tj) corresponds to a camera pose Pose. E This diagram illustrates the process of calculating (tj). It shows the camera pose of the RGB camera. RGB This is a diagram illustrating the process of calculating (ti). This is a diagram illustrating the removal of event data EV(tj) from the training data. This is a flowchart illustrating the 3D reconstruction model generation process by the information processing system. This is a diagram illustrating the first modified version of the information processing system. This is a diagram illustrating the first modified version of the information processing system. This is a diagram illustrating the second modified version of the information processing system. This is a diagram illustrating the third modified version of the information processing system. This is a diagram illustrating an example of applying the information processing system of Figure 6 to a drone photography system. This is a diagram illustrating an example of applying the information processing system of Figure 6 to a drone photography system. This is a block diagram illustrating an example of the configuration of a computer to which the technology of this disclosure is applied.

[0015] Hereinafter, embodiments for carrying out the technology of this disclosure (hereinafter referred to as "embodiments") will be described with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals to avoid redundant explanations. The explanation will proceed in the following order: 1. Background of the Method 2. Overview of the Method 3. Block Diagram of the Information Processing System 4. Flowchart of the 3D Reconstruction Model Generation Process 5. Modified Examples of the Information Processing System 6. Application Examples of the Information Processing System 7. Computer Configuration Examples

[0016] <1. Background of this method> In recent years, many 3D reconstruction technologies such as NeRF and 3D Gaussian Splatting have been proposed, that is, technologies that capture three-dimensional space and create free-viewpoint images, and the number of consumer-friendly applications is also increasing.

[0017] NeRF is a method that generates radiance fields corresponding to the space containing objects and approximates those radiance fields using a neural network.

[0018] A radiance field represents a scene in space by the color of each direction at each location in that space, and the opacity of each location in that space. Here, opacity is an indicator of the presence of some object, and can be considered a density. In other words, by obtaining a radiance field where the density of coordinates where objects exist is high, the shape of a 3D space can be represented. NeRF approximates such a radiance field with a neural network. That is, NeRF is a technology that generates a radiance field corresponding to a 3D object, approximates that radiance field with a neural network, and performs rendering using that neural network.

[0019] In NeRF, first, as shown in Figure 1, a predetermined object 10, which is the target of 3D reconstruction, is sequentially photographed from different positions using an RGB camera. This process yields multiple RGB images 11-1, 11-2, 11-3, ... and the camera poses CAMpos 11-1, 11-2, 11-3, ... at that time. The camera pose CAMpos 11 represents the orientation of the RGB camera when it took the photograph and includes the position and orientation of the RGB camera. The rectangular prism surrounding the predetermined object 10, which is the target of 3D reconstruction, in Figure 1 is for convenience only and represents the space of the object to be 3D reconstructed (the same applies to the figures described later). Information indicating the pose of the RGB camera is also called pose information.

[0020] Then, as shown in Figure 2A, a neural network 21 is generated that approximates the radiance field 20 representing the three-dimensional shape of object 10 by training the neural network using multiple RGB images 11 and camera pose CAMpos 11. During inference, as shown in Figure 2B, a rendering image (new viewpoint image) 22 of the new viewpoint is obtained by inputting pose information (also called viewpoint information) of a desired new viewpoint into this neural network 21.

[0021] However, using only a standard low-frame-rate RGB camera results in poor accuracy in scenes where the RGB camera is moving rapidly, because there is no information between frames from the RGB camera.

[0022] In contrast, a method using event data by employing an event camera in addition to an RGB camera has been proposed in Non-Patent Document 1 mentioned above, which can improve upon the aforementioned shortcomings.

[0023] An event camera is a camera (sensor) that detects brightness changes as events. It generates and outputs only the net change (difference) of the position coordinates, polarity, and time information of the brightness change that occurred. Therefore, there is no redundancy in the amount of data information, and it has a high temporal resolution on the order of microseconds. As shown in Figure 3, for example, the brightness change of object 10 between the position of RGB image 11-1 and the position of RGB image 11-2 can be acquired at high density with high temporal resolution, as event data 12-1, 12-2, ... 12-5. If the frame rate of the RGB camera is 30 fps, the temporal resolution of the event camera can be as high as, for example, 1000 fps, but in Figure 3, for clarity, it is shown as the output of five event data 12.

[0024] In the method described in Non-Patent Document 1, the camera pose CAMpos12 (not shown in Figure 3) for each event data 12 is estimated from the camera pose CAMpos11 of each RGB image 11 calculated from COLMAP using spherical linear interpolation and spline interpolation. That is, the camera pose CAMpos12-1 for event data 12-1, the camera pose CAMpos12-2 for event data 12-2, the camera pose CAMpos12-3 for event data 12-3, and so on are estimated.

[0025] Although event data 12 can be acquired at high density, the method described in Non-Patent Document 1 is prone to errors because the camera pose CAMpos 12 of each event data 12 is determined by interpolation from the camera pose CAMpos 11 of each RGB image 11, which is a sparsely plotted image.

[0026] Figure 4 shows an example of a comparison between the camera pose CAMpos12 of each event data 12 obtained using the method described in Non-Patent Document 1 and the actual camera pose.

[0027] In Figure 4, each point indicated by a black circle represents the coordinates of the camera pose CAMpos11 of the RGB camera corresponding to each RGB image 11 calculated from COLMAP. The event camera estimated path 25 shows the path of the camera pose CAMpos12 of the event camera corresponding to each event data 12, estimated from each camera pose CAMpos11 using spherical linear interpolation and spline interpolation. On the other hand, the event camera actual path 26 represents the actual movement path of the event camera, and the × marks on the event camera actual path 26 represent the actual position of the event camera corresponding to the event data 12. Note that in Figure 4, the number of × marks indicating the actual position of the event camera has been reduced for easier viewing.

[0028] In the method described in Non-Patent Document 1, when the camera movement becomes complex, such as the path shown by the dashed ellipse in Figure 4, the error between the estimated event camera path 25 and the actual event camera path 26 becomes large. The error in the estimated event camera path 25 leads to a reduction in the accuracy of the NeRF's radiance field. In other words, a reduction in the accuracy of camera pose estimation may reduce the quality of the learning results (the accuracy of the radiance field approximated by the neural network). Consequently, the quality of 3D reconstruction obtained by inference using the learned neural network and the rendering quality using it may be reduced.

[0029] <2. Overview of this Method> Below, we will describe an information processing device that accurately calculates the camera pose of event data, thereby enabling improved quality of 3D reconstruction.

[0030] Referring to Figure 5, the method for calculating the camera pose of each event data executed by the information processing device described later (hereinafter referred to as "this method") will be explained. In Figure 5, the generation of the RGB image 11 and event data 12 will be explained in the same manner as in Figures 1 and 3.

[0031] In this method, a marker 30 is placed near a predetermined object 10 that is the target of 3D reconstruction. The marker 30 has multiple light-emitting elements 31, which are composed of LEDs or the like. The number of light-emitting elements 31 can be two or more depending on the conditions, but the example in Figure 5 shows an example in which four light-emitting elements 31, 31-1 to 31-4, are provided.

[0032] Each light-emitting element 31 of the marker 30 transmits an optical signal (a signal for optical wireless communication) modulated with an ID (ID: one of 0 to 3), which is identification information that identifies itself (light-emitting element 31), using a predetermined modulation scheme. For example, digital data of "0" or "1" corresponding to the transmission information, consisting of a start code indicating the start of the transmission information, an ID (one of 0 to 3), and an end code indicating the end of the transmission information, is modulated at a predetermined blinking timing and emitted from each light-emitting element 31. Each light-emitting element 31 repeatedly transmits (emits light) the transmission information at a predetermined cycle. The relative positions of the four light-emitting elements 31-1 to 31-4 are known and stored as setting information in the information processing device.

[0033] The RGB camera and the event camera each sequentially capture images of a predetermined object 10 from different positions, generating multiple RGB images 11 consisting of RGB images 11-1, 11-2, 11-3, ... and multiple event data 12 consisting of event data 12-1, 12-2, 12-3, ...

[0034] The information processing device acquires each event data 12 generated by the event camera, demodulates the optical signal based on the multiple event data 12, and obtains the IDs and relative positions of the four light-emitting elements 31. The information processing device calculates the camera pose CAMpos12' of the event camera corresponding to the event data 12 by solving a PnP problem using the position information of the four light-emitting elements 31 acquired based on the event data 12. Therefore, while the camera pose CAMpos12 in the method of Non-Patent Literature 1 described in Figure 3 is calculated by interpolation of the camera pose CAMpos11 of the RGB camera, the camera pose CAMpos12' in this method is calculated using the position information of the four light-emitting elements 31 acquired by optical wireless communication of the marker 30, and the camera pose CAMpos of the event camera can be determined with higher accuracy.

[0035] As described above, this method enables high-frequency and high-precision self-localization of an event camera by using optical wireless communication with an event camera having high temporal resolution. This improves the quality of 3D reconstruction such as NeRF and 3D Gaussian Splatting, and the rendering quality using them.

[0036] <3. Block Diagram of the Information Processing System> Figure 6 is a block diagram showing an example configuration of an information processing system that implements the method described above. The configuration of the information processing system in Figure 6 will be explained with reference to Figures 7 to 12 as appropriate.

[0037] The information processing system 50 in Figure 6 includes an RGB camera 61, an event camera 62, and an information processing device 63.

[0038] The RGB camera 61 is a color camera having an image sensor that generates a two-dimensional color image, and it photographs a predetermined object 10, which is the target of 3D reconstruction, and a marker 30. The RGB camera 61 photographs the subject at a predetermined frame rate such as 30 fps and generates an RGB image PIC(ti) obtained from that image, which is output to the information processing device 63. The RGB image PIC(ti) represents the RGB image PIC generated at time ti (i = 1, 2, 3, ...). In the following, when time ti is omitted and it is written as RGB image PIC, it represents the RGB image PIC at any time without specifying time ti.

[0039] The event camera 62 detects changes in brightness of a predetermined object 10, which is the target of 3D reconstruction, as an event. The event camera 62 also detects changes in brightness of the light signal emitted by a marker 30 placed near the object 10 as an event. The event camera 62 has a pixel array in which multiple pixels are arranged in a matrix, and each pixel detects the temporal change of the light signal by the temporal change of the electrical signal obtained by photoelectric conversion of the light signal. Such an event camera 62 is also called an event sensor or EVS (event-based vision sensor). The event camera 62 outputs the detected temporal change of the light signal as event data EV(tj) to the information processing device 63. Event data EV(tj) represents the event data EV generated at time tj.

[0040] The event data EV(tj) outputs event data that includes the time tj (j = 1, 2, 3, ...) when the event occurred, the coordinates (xj, yj) representing the position of the pixel where the event occurred, and the polarity pj of the brightness change as an event.

[0041] The event time tj is a timestamp representing the time when the event occurred, and can be represented, for example, by the count value of a counter based on a predetermined clock signal within a sensor. The timestamp corresponding to the timing of the event can be said to be time information representing the (relative) time when the event occurred, as long as the interval between events remains the same as when the events occurred.

[0042] Polarity pj represents the direction of a change in brightness (change in light intensity) when an event occurs that exceeds a predetermined threshold (event threshold EV_TH). It indicates whether the change in brightness is positive or negative. For example, the polarity pj of an event is represented as "1" for a positive change and "0" for a negative change. In the following, events in which the change in brightness is positive will also be called positive events, and events in which the change in brightness is negative will also be called negative events.

[0043] In event data, the interval between the time tj of an event and the time tj+1 of an adjacent event is not necessarily constant. That is, the times tj and tj+1 of an event may be the same or they may be different. However, the relationship between the event times tj and tj+1 is given by the equation tj <= tj+1.

[0044] Unlike frame-format image data (frame data) which is output at a predetermined frame cycle, event data is output each time an event occurs. The RGB camera 61 captures images in synchronization with the vertical synchronization signal and outputs frame data, which is image data of one frame (screen) at the cycle of that vertical synchronization signal. However, the event camera 62 outputs event data only when an event occurs, and can therefore be described as an asynchronous or address-controlled sensor.

[0045] In the following, when the time tj is omitted and referred to as event data EV, it represents a single event data EV at any time without specifying time tj.

[0046] The information processing device 63 uses the RGB image PIC output by the RGB camera 61 and the event data EV output by the event camera 62 to perform a process that generates a 3D reconstruction model of a predetermined object 10 that has been photographed as a subject. The information processing device 63 is composed of, for example, a smartphone, a personal computer, a server device, etc.

[0047] The information processing device 63 includes an acquisition unit 71, a storage unit 72, a decoding unit 73, an event camera pose calculation unit 74, an event data pose calculation unit 75, an RGB camera pose calculation unit 76, and a 3D reconstruction learning unit 77.

[0048] The acquisition unit 71 acquires the RGB image PIC output by the RGB camera 61 and the event data EV output by the event camera 62, and supplies them to the storage unit 72. Furthermore, when the information processing device 63 performs a real-time 3D reconstruction model generation process based on the acquired RGB image PIC and event data EV, the acquisition unit 71 also supplies the acquired RGB image PIC and event data EV to the decoding unit 73.

[0049] The storage unit 72 stores the RGB image PIC and event data EV supplied from the acquisition unit 71. The storage unit 72 also stores the size information of the marker 30 and camera offset information indicating the relative positional relationship between the RGB camera 61 and the event camera 62, which are input by the user via an input unit (not shown). The size information of the marker 30 is positional information indicating the relative positional relationship of the four light-emitting elements 31, and is expressed as a relative position with respect to the center position of the marker 30, for example, the three-dimensional position coordinates of the light-emitting element 31-1 with ID:0 (-2,2,0) and the three-dimensional position coordinates of the light-emitting element 31-2 with ID:1 (2,2,0). In addition to the positional information of the four light-emitting elements 31, the absolute position coordinates of the center position of the marker 30 may also be stored in the size information of the marker 30. The storage unit 72 supplies the RGB image PIC and event data EV to the decoding unit 73 and supplies the size information of the marker 30 to the event camera pose calculation unit 74.

[0050] The decoding unit 73 demodulates the optical signals emitted by each of the four light-emitting elements 31 based on one or more event data EV supplied from the storage unit 72, using a demodulation method corresponding to the modulation method. By demodulating using a predetermined demodulation method, the decoding unit 73 obtains marker position information IDpos for each ID (ID: n=0 to 3) of the four light-emitting elements 31 (31-1 to 31-4). n (tj) = (x, y, tj) is generated and supplied to the event camera pose calculation unit 74. Marker position information IDpos n(tj) = (x, y, tj) represents the (x, y) coordinates of the pixel array where the light-emitting element 31 with ID: n is recognized and the time tj.

[0051] Here, the marker position information IDpos n The (x, y) coordinates and the time tj of (tj) = (x, y, tj) are representative values of a plurality of event data EV that recognized the light-emitting element 31 with ID: n. That is, as shown in FIG. 7, the optical signal emitted by the light-emitting element 31 is a signal in which digital data of "0" or "1" corresponding to the transmission information composed of a start code, an ID (any one of 0 to 3), and an end code is modulated at a predetermined blinking timing. On the other hand, since the event camera 62 captures only the change in the light intensity and outputs it as event data EV, a plurality of event data EV are required until the ID emitted by the light-emitting element 31 is recognized, and it has a predetermined time width. The decoding unit 73 determines a predetermined time within the time width from the start code to the end code, for example, the first time when the start code is detected, or the middle time of the time width from the start code to the end code, etc., as the time tj when generating the marker position information IDpos n (tj), generates the marker position information IDpos n (tj). Also, when photographing the object 10 while moving the event camera 62, the time from the start code to the end code and the (x, y) coordinates on the pixel array also change. The decoding unit 73 tracks the transition of the (x, y) coordinates from the start code to the end code, and sets the (x, y) coordinates corresponding to the time tj as the (x, y) coordinates of the marker position information IDpos n (tj).

[0052] FIG. 8 shows the positive events and negative events detected on the pixel array of the event camera 62 over time, and the marker position information IDpos n (tj) recognized for each ID of the light-emitting elements 31-1 to 31-4 (ID: n = 0 to 3).

[0053] Returning to FIG. 6, the event camera pose calculation unit 74 uses the marker position information IDpos of the four light-emitting elements 31-1 to 31-4 recognized in the vicinity of the time Tk (k = 1, 2, 3,...)n By solving the PnP problem using (tj), we can determine the camera pose of event camera 62 at time Tk. VLC (Tk) is calculated. The PnP problem can be solved geometrically if three corresponding points are given, but in this embodiment, the marker position information IDpos of the four light-emitting elements 31 is calculated. n Use (tj) to set the camera pose at time Tk. VLC (Tk) is calculated. Specifically, the event camera pose calculation unit 74 obtains the rotation matrix R and translation vector T of the event camera 62 using the following equation (1). ｃ = K[R|T]p w .....(1) In equation (1), s, p ｃ , K, p w is known, p ｃ This is the coordinate of a 2D point in the image coordinate system of the event camera 62, and the marker position information IDpos of the four light-emitting elements 31. n (TJ) corresponds to p w is the coordinate of a 3D point in the world coordinate system, and corresponds to the 3D position coordinates of the four light-emitting elements 31 in the world coordinate system. s is the scale factor, and K is an intrinsic parameter of the event camera 62.

[0054] Figure 9 shows the marker position information IDpos of the light-emitting elements 31-1 to 31-4 recognized in the vicinity of time Tk. ０ (tk0) to IDpos3 (tk3) are used to determine the camera pose at time Tk. VLC (Tk) = r ｋ This is a conceptual diagram showing the process of calculating the camera pose at time Tk. VLC (Tk) ｋ This is how it is expressed.

[0055] In Figure 9, for example, the camera pose r1 at time T1 corresponds to the marker position information IDpos of the light-emitting elements 31-1 to 31-4 recognized in the vicinity of time T1. ０ (t10) is calculated using IDpos3 (t13). Camera pose r at time T2 ２ This is the marker position information IDpos of the light-emitting elements 31-1 to 31-4 recognized in the vicinity of time T2.０ This is calculated using (t20) to IDpos3 (t23).

[0056] Camera pose at time Tk ｋ Within a predetermined time width as a nearby time range, the marker position information IDpos of the four light-emitting elements 31 is displayed. n If (tj) is detected, the marker position information IDpos of those four light-emitting elements 31 n This can be calculated using (tj). In this case, camera pose r ｋ (k = 1, 2, 3, ...) is calculated sequentially at irregular intervals, and time Tk is the marker position information IDpos of the four light-emitting elements 31 used in the calculation. n This is considered a representative value (e.g., the median) for time tk0 to tk3 of (tj).

[0057] Or, also, camera pose r ｋ (k = 1, 2, 3, ...) can also be calculated sequentially so that they are generated at a predetermined frame rate such as 500 fps. In this case, the marker position information IDpos of the light-emitting elements 31-1 to 31-4 is generated in the vicinity of time Tk. ０ If none of (tk0) to IDpos3 (tk3) are detected, the marker position information IDpos is detected before time Tk. n By extrapolating from (tj), we obtain the marker position information IDpos at time Tk. n After calculating (tj), the camera pose r ｋ This is calculated.

[0058] For example, the event camera pose calculation unit 74 calculates the camera pose r ｋ Assuming that it is generated at a constant frame rate, the camera pose r in Figure 9 ５Let's consider the case of calculating the marker position information IDpos2 (t52) for ID:2 (light-emitting element 31-3) and IDpos3 (t53) for ID:3 (light-emitting element 31-4) are detected within the vicinity of time T5, but IDpos0 (t50) for ID:0 (light-emitting element 31-1) and IDpos1 (t51) for ID:1 (light-emitting element 31-2) are not detected. In this case, the event camera pose calculation unit 74 calculates the marker position information IDpos0 (t50) for ID:0 and IDpos1 (t51) for ID:1 at time T5 by extrapolating from the marker position information IDpos0 (tj) and IDpos1 (tj) before time T5. Then, the marker position information IDpos ０ Using (t50) to IDpos3 (t53), set the camera pose r ５ This is calculated.

[0059] Returning to Figure 6, the event camera pose calculation unit 74 calculates the camera pose r of the event camera 62 at time Tk sequentially as described above. ｋ = Pose VLC (Tk) is supplied to the event data pause calculation unit 75.

[0060] The event data pose calculation unit 75 receives the camera pose r of the event camera 62 at time Tk from the event camera pose calculation unit 74. ｋ Using this, camera pose for each event data EV(tj) E (tj) is calculated by interpolation.

[0061] That is, as shown in Figure 10, camera pose r ｋ The marker position information IDpos of the four light-emitting elements 31 n (tj) is calculated in accordance with a predetermined time range in which it is detected, and is discrete information. The event data pose calculation unit 75 calculates the discrete camera pose r ｋ Interpolation using this method determines the camera pose corresponding to each event data EV(tj) detected by the event camera 62. E Calculate (tj). For example, the camera pose r at time T2. ２ And, the camera pose at time T3 r３ Interpolation using this method generates the camera pose corresponding to each event data EV(tj) between time T2 and time T3. E (tj) is calculated. The interpolation method is not particularly limited, but for example, spherical linear interpolation or spline interpolation can be used. Camera pose r ｋ If the time Tk at which the calculation was made is the same as the time tj of the event data EV(tj), then the camera pose Pose E (tj) = Pose VLC (Tk) This is the camera pose. E Since (tj) is calculated by interpolation, it can be calculated one-to-one with the event data EV(tj) even if the light-emitting element 31 is not visible due to occlusion or the like.

[0062] Returning to Figure 6, the event data pose calculation unit 75 calculates the camera pose for each event data EV(tj) as described above. E (tj) is supplied to the RGB camera pose calculation unit 76.

[0063] The RGB camera pose calculation unit 76 uses the camera offset information obtained from the storage unit 72 to calculate the camera pose when the RGB camera 61 generates the RGB image PIC(ti). RGB (ti) = Ri is the camera pose of each event data EV(tj) E Calculate from (tj).

[0064] Figure 11 shows the camera pose for each event data EV(tj). E (tj) represents the camera pose when the RGB camera 61 generates the RGB image PIC(ti). RGB This is a conceptual diagram showing the process of calculating (ti) = Ri.

[0065] The RGB camera pose calculation unit 76 calculates the camera pose of the event camera 62 at the same time (ti = tj) as the time ti when the RGB image PIC(ti) was generated. E The camera pose when generating the RGB image PIC(ti) using (tj) RGB Calculate (ti) = Ri.

[0066] More specifically, the camera pose of event camera 62 at the same time (ti = tj) as the time ti when the RGB image PIC(ti) was generated. E If (tj) exists, the RGB camera pose calculation unit 76 calculates the relative vector R, which is camera offset information obtained from the storage unit 72. relative Use camera pose RGB (ti) = Pose E (tj) + R relative This allows the RGB camera 61 to perform a camera pose. RGB (ti) is calculated. Meanwhile, the camera pose of event camera 62 at the same time (ti = tj) as the time ti when the RGB image PIC(ti) was generated is calculated. E If (tj) does not exist, first the RGB camera pose calculation unit 76 calculates the camera pose of the event camera 62 at the same time (ti = tj) as the time ti when the RGB image PIC(ti) was generated. E (tj) is calculated by interpolation similar to that of the event data pose calculation unit 75. Next, the RGB camera pose calculation unit 76 calculates the relative vector R relative Use camera pose RGB (ti) = Pose E (tj) + R relative This allows the RGB camera 61 to perform a camera pose. RGB Calculate (ti).

[0067] Furthermore, because the time resolution of the event data EV is extremely high compared to the time resolution of the RGB image PIC, there may be multiple event data EVs corresponding to the time ti when the RGB image PIC(ti) was generated. In this case, the RGB camera pose calculation unit 76 may determine the event data EV corresponding to the time ti when the RGB image PIC(ti) was generated according to a predetermined rule, such as the median or minimum value of the time width consisting of multiple event data EVs.

[0068] Returning to Figure 6, the RGB camera pose calculation unit 76 calculates the camera pose of each RGB image PIC(ti) as described above. RGB(ti) is the camera pose of each event data EV(tj) supplied from the event data pose calculation unit 75. E (tj) is supplied to the 3D reconstruction learning unit 77.

[0069] The 3D reconstruction learning unit 77 processes each RGB image PIC(ti) supplied from the RGB camera pose calculation unit 76 and its camera pose Pose. RGB (ti), and each event data EV(tj) and its camera pose Pose E Using (tj), a 3D reconstruction model of a predetermined object 10 photographed as a subject is generated by machine learning. For example, the 3D reconstruction learning unit 77 learns the 3D reconstruction model using the E-nerf method described in Non-Patent Literature 1. A machine learning model using event data other than that described in Non-Patent Literature 1 may also be used for the 3D reconstruction model. For example, a machine learning model using 3D Gaussian Splatting technology may be used. The 3D reconstruction model (and its parameters) generated by the 3D reconstruction learning unit 77 is supplied to, for example, the storage unit 72 for storage and used in the free-viewpoint image generation mode. In the free-viewpoint image generation mode, the inference unit (not shown) of the information processing device 63 inputs pose information (viewpoint information) of a desired new viewpoint to the 3D reconstruction model obtained by machine learning, thereby generating an image of the object 10 viewed from that new viewpoint (new viewpoint image). Alternatively, the 3D reconstruction model (and its parameters) generated by the 3D reconstruction learning unit 77 may be output to another information processing device that performs the process of generating and displaying a new viewpoint image of the object 10 from a desired new viewpoint.

[0070] Furthermore, the event data EV(tj) and its camera pose Pose are used in the machine learning of the 3D reconstruction model. E Regarding (tj), as shown in Figure 12, it is preferable to remove the event data EV(tj) generated by optical wireless communication, in other words, the event data EV(tj) generated by the blinking of each light-emitting element 31 of the marker 30, from the training data before training.

[0071] Figure 12 is a conceptual diagram showing the event data EV(tj) to be removed from the event data EV(tj) output from the event camera 62.

[0072] The information processing system 50 is configured as described above.

[0073] <4. Flowchart of 3D Reconstruction Model Generation Process> Next, the 3D reconstruction model generation process performed by the information processing system 50 will be described with reference to the flowchart in Figure 13. This process is started, for example, when an operation to instruct the information processing device 63 to start the 3D reconstruction model generation process is performed.

[0074] First, in step S1, the RGB camera 61 and the event camera 62 photograph a predetermined object 10 and marker 30, which are the targets for 3D reconstruction, and output the RGB image PIC and event data EV obtained from the photography to the information processing device 63. More specifically, the RGB camera 61 photographs the object 10 at a predetermined frame rate such as 30 fps and outputs the resulting RGB image PIC to the information processing device 63. The event camera 62 detects changes in brightness of the object 10 and marker 30 and outputs them to the information processing device 63 as event data EV. The acquisition unit 71 of the information processing device 63 acquires the RGB image PIC and event data EV output from the RGB camera 61 and the event camera 62 and stores them in the storage unit 72. The storage unit 72 stores the RGB image PIC and event data EV.

[0075] In step S2, the decoding unit 73 demodulates the optical signal based on multiple event data EVs, thereby generating marker position information IDpos for each of the four light-emitting elements 31 (ID: n=0 to 3). nGenerate (tj) = (x, y, tj) and supply it to the event camera pose calculation unit 74. More specifically, the decoding unit 73 acquires a plurality of event data EV sequentially stored in the storage unit 72, and demodulates the optical signals emitted by each of the four light emitting elements 31 based on the plurality of event data EV by a predetermined demodulation method. The decoding unit 73 recognizes the ID (ID: n = 0 to 3) of each of the four light emitting elements 31 by demodulating the optical signal, and for each ID of the light emitting element 31, the marker position information IDpos n Generate (tj) = (x, y, tj) and supply it to the event camera pose calculation unit 74.

[0076] In step S3, the event camera pose calculation unit 74 uses the marker position information IDpos of the four light emitting elements 31-1 to 31-4 recognized in the vicinity of the time Tk (k = 1, 2, 3,...) n (tj) to solve the PnP problem, thereby calculating the camera pose Pose of the event camera 62 at time Tk VLC (Tk). Thereby, the rotation matrix R and the translation vector T of the event camera 62 in the above-described equation (1) are obtained. The event camera pose calculation unit 74 supplies the camera pose Pose of the event camera 62 at time Tk calculated sequentially VLC (Tk) to the event data pose calculation unit 75.

[0077] In step S4, the event data pose calculation unit 75 uses the camera pose Pose of the event camera 62 at time Tk supplied from the event camera pose calculation unit 74 VLC (Tk) to calculate the camera pose Pose for each event data EV(tj) by interpolation. The event data pose calculation unit 75 supplies the calculated camera pose Pose for each event data EV(tj) E (tj)を、RGBカメラポーズ計算部76に供給する。 E (tj) to the RGB camera pose calculation unit 76.

[0078] In step S5, the RGB camera pose calculation unit 76 uses the camera pose Pose of each event data EV(tj) EUsing (tj) and the camera offset information, the camera pose Pose RGB (ti) when the RGB camera 61 generates the RGB image PIC(ti) is calculated. More specifically, the camera pose Pose E (tj) of the event camera 62 at the same time (ti = tj) as the time ti when the RGB image PIC(ti) is generated, and the relative vector R which is the camera offset information relative are used to calculate the camera pose Pose RGB (ti) = Pose E (tj) + R relative Thus, the camera pose Pose RGB (ti) of the RGB camera 61 is calculated. The RGB camera pose calculation unit 76 supplies the calculated camera pose Pose RGB (ti) of each RGB image PIC(ti) to the 3D reconstruction learning unit 77 together with the camera pose Pose E (tj) of each event data EV(tj) supplied from the event data pose calculation unit 75.

[0079] In step S6, the 3D reconstruction learning unit 77 uses each RGB image PIC(ti) supplied from the RGB camera pose calculation unit 76, its camera pose Pose RGB (ti), each event data EV(tj), and its camera pose Pose E (tj) to generate a 3D reconstruction model of a predetermined object 10 photographed as a subject by machine learning. For example, the 3D reconstruction learning unit 77 learns the 3D reconstruction model by the E-nerf method of Non-Patent Document 1. The 3D reconstruction model (parameters) obtained by learning is stored in, for example, the storage unit 72, and the 3D reconstruction model generation process in FIG. 13 ends.

[0080] As described above, in the information processing system 50, the information processing apparatus 63 generates a 3D reconstruction model of the predetermined object 10 photographed by the RGB camera 61 and the event camera 62 as a subject. The camera pose Pose E(tj) can be estimated with high accuracy by using optical wireless communication via marker 30. Also, the camera pose of each RGB image PIC(ti) RGB (ti) is also an estimated camera pose with high accuracy and frequency. E By calculating from (tj), it is possible to estimate with high accuracy. Each RGB image PIC(ti) and its camera pose Pose are estimated with high accuracy. RGB (ti), and each event data EV(tj) and its camera pose Pose E By using (tj), the quality of the generated 3D reconstructed model can be improved.

[0081] <5. Modifications of the Information Processing System> Next, various modifications of the information processing system 50 described above will be explained.

[0082] <First Modification> Figures 14 and 15 illustrate the first modification of the information processing system 50.

[0083] The information processing system 50 described above had an RGB camera 61 and an event camera 62, and was configured to have a separate camera that outputs an RGB image PIC (first camera) and a camera that outputs event data EV (second camera).

[0084] Alternatively, a configuration may be used in which the camera that outputs the RGB image PIC and the camera that outputs the event data EV are implemented in the same camera. For example, a hybrid camera can be used which has a pixel array in which RGB pixels that generate R, G, or B luminance signals of gradation levels according to the amount of incident light and event pixels that detect changes in the luminance of the incident light as events and generate event signals are arranged together. When a hybrid camera is used instead of the RGB camera 61 and the event camera 62, the hybrid camera detects changes in the luminance of the light signal emitted by the marker 30 as an event and outputs the event data EV to the information processing device 63, and also outputs the RGB image PIC of the object 10 to be 3D reconstructed to the information processing device 63. The acquisition unit 71 of the information processing device 63 acquires the RGB image PIC and event data EV output by the hybrid camera and supplies them to the storage unit 72.

[0085] A hybrid camera has a pixel array in which 16 pixels in a 4x4 arrangement, as shown in Figure 14, are repeatedly arranged in the row and column directions. Figure 14 shows an example of the pixel configuration of a hybrid camera.

[0086] Figure 14 shows a 4x4 pixel array of 16 pixels, where pixels 100R, 100Gr, 100Gb, and 100B are arranged in a 2x2 grid of 4 pixels each, in a Bayer array of R, Gr, Gb, and B. Two pixels of pixel 100R and two pixels of pixel 100B are replaced with pixel 100EV. Pixels 100R, 100Gr, 100Gb, and 100B are RGB pixels that generate R (red) luminance signals, G (green) luminance signals, G (green) luminance signals, and B (blue) luminance signals, respectively, while pixel 100EV is an event pixel that detects changes in visible light luminance as events.

[0087] Furthermore, the hybrid camera may also be a pixel array in which 16 pixels in a 4x4 configuration are repeatedly arranged in the row and column directions, as shown in Figure 15. Figure 15 shows another example of a pixel configuration for a hybrid camera.

[0088] Figure 15 shows a pixel array in which one of the two 100EV pixels in the pixel array shown in Figure 14 is replaced with a 100EV-IR pixel, which is an event pixel that detects changes in infrared light brightness as an event. In this way, the hybrid camera may have two event pixels: one 100EV pixel that detects visible light events and one 100EV-IR pixel that detects invisible light (infrared light) events. In this case, the light signal emitted by the light-emitting element 31 of the marker 30 is invisible light, which is infrared light. The event data EV of the 100EV pixel that detects visible light events is used for 3D reconstruction, and the event data EV of the 100EV-IR pixel that detects infrared light events is used for camera pose. VLC Since the use of event data EV can be separated for use in the calculation of (Tk), the process of removing event data EV generated by optical wireless communication from the training data, as explained in Figure 12, becomes unnecessary. In other words, it becomes easier to process the data so that the emission of marker 30 does not affect the machine learning process of the 3D reconstruction model.

[0089] Even in a configuration where a camera for photographing the object 10 to be 3D reconstructed and a camera for detecting optical wireless communication signals as events are provided separately, for example, by using two event cameras 62, one event camera 62 (RGB) for detecting visible light events and one event camera 62 (IR) for detecting non-visible light (infrared light) events, event data EV and camera pose Pose used for 3D reconstruction can be obtained. VLC It is possible to configure the system to separate the use of event data EV, which is used in the calculation of (Tk).

[0090] When using the hybrid cameras shown in Figures 14 and 15, in step S5 of the 3D reconstruction model generation process in Figure 13, the camera offset information is used to determine the camera pose of the event camera 62. E (tj) Camera pose for RGB camera 61 RGB The process for calculating (ti) will be changed. The process in step S5 after the change will be to set the camera pose of the event camera 62. E(tj) is the same as the camera pose of RGB camera 61. RGB (ti) is considered (Pose RGB (ti) = Pose E (tj)).

[0091] <Second Modification> Figure 16 is a diagram illustrating a second modification of the information processing system 50.

[0092] As a second modification of the information processing system 50, for example, a configuration is possible in which multiple markers 30 are provided.

[0093] Figure 16 shows an example in which two markers 30 are placed near a predetermined object 10 that is the target of 3D reconstruction. The camera CAM in Figure 16 corresponds to a set of an RGB camera 61 and an event camera 62, or a hybrid camera having the configuration described in Figure 14 or Figure 15.

[0094] The two markers 30-1 and 30-2 shown in Figure 16 have the same configuration as marker 30 shown in Figure 5, differing only in the identification information transmitted as an optical signal by the four light-emitting elements 31. For example, the four light-emitting elements 31 of marker 30-1 transmit IDs 0 to 3 as identification information. The four light-emitting elements 31 of marker 30-2 transmit IDs 4 to 7 as identification information.

[0095] If there is only one marker 30 and the camera CAM moves around the object 10 to be 3D reconstructed while taking pictures, depending on the movement path, the object 10 may become an obstacle, and there may be scenes where the optical signal of the marker 30 cannot be detected. By placing multiple markers 30 around the object 10 to be 3D reconstructed, the camera CAM can reliably detect the optical signal of at least one marker 30.

[0096] Figure 16 shows an example where two markers 30 are placed, but three or more markers 30 may also be placed.

[0097] <Third Modification> Figure 17 is a diagram illustrating a third modification of the information processing system 50.

[0098] As a third modification of the information processing system 50, for example, a configuration is possible in which multiple cameras (CAMs) are provided.

[0099] Figure 17 shows an example in which two cameras CAM-1 and CAM-2 are provided in the information processing system 50. Each of the cameras CAM-1 and CAM-2 in Figure 17 corresponds to a set of an RGB camera 61 and an event camera 62, or a hybrid camera having the configuration described in Figure 14 or Figure 15.

[0100] Cameras CAM-1 and CAM-2 are equipped with a marker 121. Like marker 30, marker 121 has four light-emitting elements 31-1 to 31-4.

[0101] When a camera CAM moves around an object 10 to be 3D reconstructed and takes images, depending on the camera CAM's movement path, the object 10 may become an obstacle, and there may be scenes where the optical signal of marker 30 cannot be detected. In such scenes, a camera CAM that cannot detect the optical signal of marker 30 can calculate its own camera pose by detecting the optical signal of marker 121 of another camera CAM that can detect the optical signal of marker 30. For example, in the example in Figure 17, suppose camera CAM-2 is in a position where it cannot detect the optical signal of marker 30. Camera CAM-1 can detect the optical signal of marker 30, so its camera pose can be calculated. The camera pose of camera CAM-2 can be calculated by detecting the optical signal of marker 121-1 of camera CAM-1, and using its relative position with respect to the camera pose of camera CAM-1 as the reference. The number of light-emitting elements 31 on the camera CAM's marker 121 is two or more, depending on the conditions, similar to the marker 30.

[0102] Figure 17 shows an example where two camera CAMs capture the object 10 to be 3D reconstructed and a marker 30, but three or more camera CAMs may also be used. If at least one of the multiple camera CAMs detects the optical signal of the marker 30 placed near the object 10, the camera pose of each camera CAM can be calculated from its relative position to that camera CAM.

[0103] It is also possible to combine the second modified example described in Figure 16 with the third modified example described in Figure 17. That is, a configuration in which multiple markers 30 are placed near the object 10 to be 3D reconstructed, and the object 10 and the multiple markers 30 are photographed by multiple cameras CAM.

[0104] <Other variations> <Rolling shutter camera> When the RGB camera 61 of the information processing system 50 described above takes a picture of the object 10 to be 3D reconstructed using global shutter operation, the exposure timing of all pixels is the same. On the other hand, when the RGB camera 61 takes a picture of the object 10 using rolling shutter operation, the exposure timing of each row of the pixel array is different, so strictly speaking, the self-position estimation result will be different for each row. When the RGB camera 61 takes a picture of the object 10 using rolling shutter operation, the actual shooting time of the RGB image PIC is calculated for each row of the pixel array, and the camera pose of the RGB camera 61 is set. RGB (ti) = Ri and event data EV(tj) camera pose Pose E By calculating the correspondence with (tj) on a row-by-row basis, highly accurate self-position estimation is possible even in rolling shutter operation.

[0105] <Marker Transmission Information> In the information processing system 50 described above, the marker 30 placed near the object 10 to be 3D reconstructed transmits only the ID, which is the identification information of the light-emitting element 31, as substantial transmission information, excluding the start code and end code. However, in addition to the identification information of the light-emitting element 31, the size information of the marker 30 may also be transmitted as transmission information. This eliminates the need for the information processing device 63, which is on the side that detects the optical signal of the marker 30, to register the size information of the marker 30 in advance. Without prior information, it becomes possible to estimate the camera pose by detecting and demodulating the optical signal of the marker 30.

[0106] <Number of light-emitting elements in the marker> In the example above, the marker 30 has four light-emitting elements 31-1 to 31-4, and the marker position information IDpos of the four light-emitting elements 31 n Use (tj) to set the camera pose at time Tk. VLC(Tk) was calculated. However, in PnP problems, generally, if three corresponding points are given, a predetermined three-dimensional position coordinate can be determined, so the number of light-emitting elements 31 should be three or more. Also, for example, if the RGB camera 61 and event camera 62 move in two dimensions in the XY direction, such as on the ceiling of a factory, and it is sufficient to determine the two-dimensional position coordinate, then the number of light-emitting elements 31 should be two or more.

[0107] <6. Examples of Information Processing System Applications> Figures 18 to 20 show examples of applying the above-described information processing system 50 to a drone's shooting system.

[0108] As shown in Figure 18, the drone 141 has the RGB camera 61 and event camera 62 described above, or the hybrid camera described in Figures 14 and 15, and moves at high speed around the object 10 to be 3D reconstructed to take images, and outputs the RGB image PIC and event data EV obtained from the images to the information processing device 63. The information processing device 63 (not shown) acquires the RGB image PIC and event data EV and can estimate the camera pose of the RGB camera 61 and event camera 62 with high accuracy by using optical wireless communication via the marker 30. The information processing device 63 may be installed inside the drone 141.

[0109] By applying the information processing system 50 to the drone's shooting system and using optical wireless communication, self-position estimation can be performed at a faster speed compared to estimating the camera pose using methods such as COLMAP, making it suitable for shooting drones moving at high speeds.

[0110] Figure 19 shows an example of applying the coordinated shooting method described in Figure 17, in which multiple cameras CAM work together to compensate for the blind spots of marker 30, to a drone 141.

[0111] Two drones 141-1 and 141-2 move at high speed around the object 10 and take photographs. If one of the drones 141-1 or 141-2 (the first drone 141) cannot detect the light signal of the marker 30, the other drone 141 (the second drone 141) detects the light signal of the marker 121, calculates its own camera pose (Pose), and takes photographs of the object 10 while estimating its own position. There may be three or more drones 141.

[0112] Figure 20 shows an example of drone 141 capturing a dynamic scene.

[0113] For example, as shown in Figure 20, if the object 10 to be 3D reconstructed is a person performing a predetermined movement, and the drone 141 is capturing a dynamic scene, the information processing device 63 can estimate the camera pose in real time from the acquired RGB image PIC and event data EV using optical wireless communication. The information processing device 63 may also perform movement control such that the drone 141 moves along a path that allows for the generation of a highly accurate 3D reconstructed model of the object 10.

[0114] <7. Example of Computer Configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on the computer. Here, the term "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0115] Figure 21 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0116] In the computer 200, the processing circuit 201, ROM (Read Only Memory) 202, and RAM (Random Access Memory) 203 are interconnected by a bus 204.

[0117] An input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

[0118] The input unit 206 may include physical or virtual means of operation that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means of inputting information by the user through voice, gaze, etc. Furthermore, the input unit 206 may include sensors for inputting various physical quantities to the computer. For example, the input unit 206 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 206 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 207 may include means of presenting information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 208 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 209 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 210 drives removable media 211 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0119] The processing circuit 201 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 201 (its processor) performs the series of processes described above by loading the program stored in the memory unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executing it. The processing circuit 201 can output the processing results of the series of processes from the output unit 207, for example, via the bus 204 and the input / output interface 205, as needed. The processing circuit 201 can also store the processing results in the memory unit 208 or transmit them from the communication unit 209.

[0120] The program executed by the computer 200 (processing circuit 201) can be provided by recording it on a removable medium 211, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0121] In the computer 200, programs can be installed in the storage unit 208 via the input / output interface 205 by inserting the removable media 211 into the drive 210. Alternatively, programs can be received by the communication unit 209 from other devices such as servers via wired or wireless transmission media and installed in the storage unit 208. Furthermore, programs can be pre-installed in the ROM 202 or the storage unit 208.

[0122] The program executed by the computer 200 may be a program that is processed chronologically in the order described herein, or it may be a program that is processed in parallel or at necessary times, such as when a call is made.

[0123] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0124] The program may be processed by a single computer 200 (processor), or it may be processed in a distributed manner by multiple computers 200. Furthermore, the program may be transferred to a remote computer and executed there.

[0125] When the computer 200 executes a program and the above-described series of processes are performed, the input unit 206 or the communication unit 209 functions as the acquisition unit 71 in Figure 6, and the storage unit 208 functions as the storage unit 72 in Figure 6. When the processing circuit 201 (its processor) executes a program, it functions as the decoding unit 73, event camera pose calculation unit 74, event data pose calculation unit 75, RGB camera pose calculation unit 76, and 3D reconstruction learning unit 77 in Figure 6.

[0126] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of a processor and memory (for example, only the processing circuit 201, or a combination of the processing circuit 201 to the bus 204, etc.), constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0127] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0128] The embodiments of this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of the technology of this disclosure.

[0129] For example, the technology disclosed herein can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

[0130] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0131] The technology disclosed herein may adopt the following configurations: (1) An information processing apparatus comprising: an acquisition unit that acquires event data, which is obtained by detecting a change in the brightness of a light signal emitted by a marker as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the light signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated light signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time. (2) The information processing apparatus according to (1), wherein the decoding unit generates position information of a plurality of light-emitting elements of the marker by demodulating the light signal, and the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the position information of the plurality of light-emitting elements. (3) The information processing apparatus according to (2), wherein the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the position information of the plurality of light-emitting elements detected within a predetermined time width. (4) The information processing apparatus according to (3), wherein the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data at a predetermined frame rate. (5) The information processing apparatus according to any one of (1) to (4), further comprising an event data pose calculation unit that calculates the camera pose of the camera at each event data acquisition between the time of acquisition of the first event data and the time of acquisition of the second event data by interpolation using the camera pose at the time of acquisition of the first event data and the camera pose at the time of acquisition of the second event data. (6) The information processing apparatus according to any one of (1) to (5), wherein the RGB camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the event data at the same time as the RGB image.(7) The information processing apparatus according to any one of (1) to (6), wherein the event data includes data generated by detecting a change in the brightness of the marker's optical signal and data generated by detecting a change in the brightness of the predetermined object, and the 3D reconstruction learning unit removes the event data generated by detecting a change in the brightness of the marker's optical signal from the event data acquired by the acquisition unit to generate a 3D reconstruction model of the predetermined object. (8) The information processing apparatus according to any one of (1) to (7), wherein the camera used when acquiring the predetermined event data and the camera used when acquiring the RGB image are different cameras. (9) The information processing apparatus according to any one of (1) to (8), wherein the camera used when acquiring the predetermined event data and the camera used when acquiring the RGB image are different cameras, and the camera used when acquiring the predetermined event data consists of two cameras: one that detects visible light events and one that detects non-visible light events. (10) The information processing apparatus according to any one of (1) to (9) above, wherein the camera used when acquiring the predetermined event data and the camera used when acquiring the RGB image are the same camera having a pixel array in which event pixels and RGB pixels are mixed and arranged. (11) An information processing method comprising: an information processing apparatus acquiring event data in which a change in brightness of a light signal emitted by a marker is detected as an event, and an RGB image generated by photographing a predetermined object; demodulating the light signal based on the event data; calculating the camera pose of the camera used when acquiring the predetermined event data from the demodulated light signal; calculating the camera pose of the camera used when acquiring the RGB image using the camera pose of the camera used when acquiring the predetermined event data; and generating a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.(12) An information processing system comprising: one or more markers having two or more light-emitting elements; one or more cameras capable of outputting at least one of event data or an RGB image; and an information processing device, wherein the information processing device includes: an acquisition unit that acquires event data obtained by detecting a change in brightness of an optical signal emitted by the one or more markers as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the optical signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated optical signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time. (13) The information processing system according to (12), comprising a plurality of the markers. (14) The information processing system according to any one of (12) to (13), comprising a plurality of cameras, wherein each camera has a marker having two or more light-emitting elements. (15) The information processing system according to any one of (12) to (13), comprising a plurality of drones each having a camera, wherein each drone has a marker having two or more light-emitting elements, and the event camera pose calculation unit calculates the camera pose of the camera of the first drone based on the result of detecting the marker of a second drone, which is different from the first drone, if the camera of the first drone among the plurality of drones cannot detect the marker placed near the predetermined object. (16) The information processing system according to any one of (12) to (15), wherein the one or more cameras is a single camera capable of outputting the event data and the RGB image. (17) The information processing system according to any one of (12) to (16), wherein the one or more cameras are two cameras: an event camera that outputs the event data and an RGB camera that outputs the RGB image.(18) The information processing system according to any one of (12) to (17), wherein the one or more cameras are three cameras: an event camera that detects visible light events and outputs event data, an event camera that detects non-visible light events and outputs event data, and an RGB camera that outputs an RGB image. (19) The information processing system according to (17) or (18), wherein the RGB camera that outputs the RGB image is a camera that photographs the predetermined object in global shutter operation. (20) The information processing system according to (17) or (18), wherein the RGB camera that outputs the RGB image is a camera that photographs the predetermined object in rolling shutter operation.

[0132] 10 Object, 11 RGB image, 12 Event data, 20 Radiance field, 21 Neural network, 30 Marker, 31-1 to 4 Light-emitting element, 50 Information processing system, 61 RGB camera, 62 Event camera, 63 Information processing device, 71 Acquisition unit, 72 Storage unit, 73 Decode unit, 74 Event camera pose calculation unit, 75 Event data pose calculation unit, 76 RGB camera pose calculation unit, 77 3D reconstruction learning unit, 100B pixel, 100EV pixel, 100Gb pixel, 100Gr pixel, 100R pixel, 121 Marker, 141 Drone, 200 Computer, 202 ROM, 204 Bus, 205 Input / Output interface, 206 Input unit, 207 Output unit, 208 Memory unit, 209 Communication unit, 210 Drive, 211 Removable media

Claims

1. An information processing device comprising: an acquisition unit that acquires event data, which is obtained by detecting a change in the brightness of a light signal emitted by a marker as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the light signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated light signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

2. The information processing apparatus according to claim 1, wherein the decoding unit generates position information of a plurality of light-emitting elements of the marker by demodulating the optical signal, and the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the position information of the plurality of light-emitting elements.

3. The information processing apparatus according to claim 2, wherein the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the position information of the plurality of light-emitting elements detected within a predetermined time width.

4. The information processing apparatus according to claim 3, wherein the event camera pose calculation unit calculates the camera pose of the camera at the time of acquisition of the predetermined event data at a predetermined frame rate.

5. The information processing apparatus according to claim 1, further comprising an event data pose calculation unit that calculates the camera pose of the camera at each event data acquisition between the time of the first event data acquisition and the time of the second event data acquisition by interpolation using the camera pose at the time of the first event data acquisition and the camera pose at the time of the second event data acquisition.

6. The information processing apparatus according to claim 1, wherein the RGB camera pose calculation unit calculates the camera pose of the camera at the time the RGB image was acquired using the camera pose of the event data at the same time as the RGB image.

7. The information processing apparatus according to claim 1, wherein the event data includes data generated by detecting a change in the brightness of the marker's optical signal and data generated by detecting a change in the brightness of the predetermined object, and the 3D reconstruction learning unit removes the event data generated by detecting a change in the brightness of the marker's optical signal from the event data acquired by the acquisition unit to generate a 3D reconstruction model of the predetermined object.

8. The information processing apparatus according to claim 1, wherein the camera used when acquiring the predetermined event data and the camera used when acquiring the RGB image are different cameras.

9. The information processing apparatus according to claim 1, wherein the camera used to acquire the predetermined event data and the camera used to acquire the RGB image are different cameras, and the camera used to acquire the predetermined event data consists of two cameras: one for detecting visible light events and one for detecting non-visible light events.

10. The information processing apparatus according to claim 1, wherein the camera used when acquiring the predetermined event data and the camera used when acquiring the RGB image are the same camera having a pixel array in which event pixels and RGB pixels are arranged in a mixed manner.

11. An information processing method comprising: an information processing device detecting a change in the brightness of a light signal emitted by a marker as an event, acquiring event data and an RGB image generated by photographing a predetermined object; demodulating the light signal based on the event data; calculating the camera pose of the camera at the time of acquiring the predetermined event data from the demodulated light signal; calculating the camera pose of the camera at the time of acquiring the RGB image using the camera pose of the camera at the time of acquiring the predetermined event data; and generating a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

12. An information processing system comprising: one or more markers having two or more light-emitting elements; one or more cameras capable of outputting at least one of event data or an RGB image; and an information processing device, wherein the information processing device includes: an acquisition unit that acquires event data obtained by detecting a change in the brightness of a light signal emitted by the one or more markers as an event, and an RGB image generated by photographing a predetermined object; a decoding unit that demodulates the light signal based on the event data; an event camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the predetermined event data from the demodulated light signal; an RGB camera pose calculation unit that calculates the camera pose of the camera at the time of acquisition of the RGB image using the camera pose of the camera at the time of acquisition of the predetermined event data; and a 3D reconstruction learning unit that generates a 3D reconstruction model of the predetermined object using the event data, the camera pose at that time, and the RGB image and the camera pose at that time.

13. The information processing system according to claim 12, comprising a plurality of the markers.

14. The information processing system according to claim 12, comprising a plurality of cameras, wherein each camera has a marker having two or more light-emitting elements.

15. The information processing system according to claim 12, comprising a plurality of drones each having a camera, wherein each drone has a marker having two or more light-emitting elements, and the event camera pose calculation unit calculates the camera pose of the camera of the first drone based on the result of detecting the marker by a second drone, which is different from the first drone, if the camera of the first drone among the plurality of drones cannot detect the marker placed near the predetermined object.

16. The information processing system according to claim 12, wherein the one or more cameras are one camera capable of outputting the event data and the RGB image.

17. The information processing system according to claim 12, wherein the one or more cameras consist of two cameras: an event camera that outputs the event data and an RGB camera that outputs the RGB image.

18. The information processing system according to claim 12, wherein the one or more cameras are three cameras: an event camera that detects visible light events and outputs the event data, an event camera that detects non-visible light events and outputs the event data, and an RGB camera that outputs the RGB image.

19. The information processing system according to claim 17, wherein the RGB camera that outputs the RGB image is a camera that photographs the predetermined object in global shutter operation.

20. The information processing system according to claim 17, wherein the RGB camera that outputs the RGB image is a camera that photographs the predetermined object using a rolling shutter operation.

Images