3D data reproduction system

Abstract

Provided is a 3D data reproduction system comprising: a 3D data measurement device including a plurality of LiDARs and one camera; and a time synchronization device that transmits the same time information to the plurality of LiDARs and the one camera and performs synchronization control between the plurality of LiDARs and the one camera. The synchronization control synchronizes point cloud data from the plurality of LiDARs and the one camera to achieve measurement of a colored point cloud at a high frame rate.

Description

3D Data Reconstruction System

[0001] The present invention relates to a 3D data reconstruction system capable of measuring high-quality colored real-time point clouds with little shaking even when using a low frame rate LiDAR, by performing synchronous control between a plurality of LiDARs and a camera, and improving a method for combining point cloud data.

[0002] In entertainment fields such as sports and music live events, there are services that capture athletes, performers, entire spaces and the like as 3D data for live streaming, so as to provide real-time content to viewers through immersive display equipment such as 3D displays and head-mounted displays.

[0003] In the above services, low latency is required for 3D data measurement, particularly in order to realize "interactive content" that is intended to feed back viewers' reaction after experiencing the content back to athletes or performers.

[0004] "Professional Baseball Broadcast Using Volumetric Video Technology", by Takayuki Shinoda et al., Journal of the Institute of Image Information and Television Engineers Vol. 78, No. 2, pp. 247-251 (2024) "Intel RealSense SR305, D415 and L515: Experimental Evaluation and Comparison of Depth Estimation", by Francisco Lourenço, Helder Araujo, VISAPP 2021-16th International Conference on Computer Vision Theory and Applications "Multimedia at Work", by Wenjun Zeng, 1070-986X / 12 / $31.00 © 2012 IEEE, IEEE Computer Society

[0005] It is generally known that combining a single LiDAR with a camera allows for real-time measurement of a wide 3D space as a colored point cloud. However, many rotary LiDARs capable of measuring large amounts of point cloud data with high precision have a measurement frame rate of around 10 Hz, and the low frame rate of the resulting colored point cloud has been a problem.

[0006] For example, as shown in Non-Patent Document 1, volumetric video technology is known as a method for 3D measurement of a wide area using multiple cameras. Volumetric video involves capturing a subject (person, object, space) from multiple directions using cameras, and generally involves applying chroma key to separate the subject from the background, and then matching corresponding points from multiple cameras to estimate depth information. This technology requires the use of a large number of cameras, ranging from tens to hundreds, for measurement, and even with high-speed processing techniques like those in Non-Patent Document 1, it still takes about 3 seconds from measurement to display. 3 seconds is too long a processing time to achieve natural two-way interaction, and in this respect, this technology needs improvement.

[0007] On the other hand, as disclosed in Non-Patent Documents 2 and 3, Time-of-Flight (ToF) cameras are known as a method for 3D measurement of space at a high frame rate of 30 fps. ToF cameras work by irradiating the object to be measured with light pulses such as infrared light, and calculating the distance between the camera and the object by measuring the time it takes for the irradiated light to reflect and be remeasured. ToF cameras emit light for measurement themselves, but they can be divided into two types, scan type and flash type, depending on the irradiation method.

[0008] Scanning-type imaging involves irradiating a point with a laser beam and tracing the space using rotating mirrors or MEMS mirrors. Flash-type imaging, on the other hand, illuminates a wide area at once with a surface rather than a point, and uses an image sensor to detect the reflected light in parallel to generate a depth map.

[0009] While scanning-type cameras offer high accuracy and can measure space over very long distances, ranging from tens to over a hundred meters, they suffer from a lower frame rate. Therefore, most ToF cameras use flash-type sensors, which can achieve high frame rates of 30 fps or more, but their shooting distance is mostly limited to within 10 meters, resulting in larger distance errors at distant measurement points.

[0010] This invention has been made in view of these circumstances, and aims to provide a 3D data reproduction system that can measure high-quality, colored, real-time point clouds with minimal shaking at a high frame rate of 30 fps or more.

[0011] A first aspect of the present invention for achieving the above objective is a 3D data reproduction system comprising a 3D data measurement device including a plurality of LiDARs and one camera, and a time synchronization device that sends the same time information to the plurality of LiDARs and one camera and performs synchronization control between the plurality of LiDARs and one camera, thereby synchronizing point cloud data from the plurality of LiDARs and one camera through synchronization control and realizing high frame rate colored point cloud measurement.

[0012] A second aspect of the present invention is a 3D data reproduction system of the first aspect, further comprising: a 3D data merging device that, when there is a pair of LiDAR data and camera data output from a camera in which the time data matches, converts the point cloud data for the matching pair into a camera coordinate system, projects the converted point cloud data onto a two-dimensional image to obtain a depth image, and sends out data including an RGBD image obtained based on the depth image, corresponding time information, pre-prepared camera internal parameters, and pre-prepared camera external parameters to reference coordinates; and a 3D data reproduction device that receives the data sent out from the 3D data merging device and converts the received data into a colored 3D point cloud of reference coordinates.

[0013] A third aspect of the present invention is a 3D data reproduction system of the second aspect, comprising a plurality of equal numbers of 3D data measurement devices and 3D data merging devices, with one 3D data measurement device and one 3D data merging device connected in a one-to-one relationship, a time synchronization device provided in common to the plurality of 3D data measurement devices to send the same time information to all LiDARs and cameras included in the plurality of 3D data measurement devices, and a 3D data reproduction device provided in common to the plurality of 3D data merging devices to receive data transmitted from the plurality of 3D data merging devices.

[0014] A fourth aspect of the present invention is a 3D data reproduction system of the second aspect, in which the 3D data merging device includes a data merging unit that converts point cloud data to a camera coordinate system and a LiDAR external parameter storage unit that stores LiDAR external parameters used by the data merging unit when converting point cloud data to a camera image system, and in the 3D data measurement device, one of the multiple LiDARs included in the 3D data measurement device is predetermined as a reference LiDAR, and the data merging unit uses the LiDAR external parameters of the reference LiDAR from among the LiDAR external parameters stored in the LiDAR external parameter storage unit as LiDAR external parameters used when converting point cloud data to a camera coordinate system.

[0015] According to the 3D data reproduction system of the present invention, by combining multiple scanning LiDARs and cameras, and by devising a method for combining point cloud data, it becomes possible to measure high-quality, colored real-time point clouds with minimal shaking at a high frame rate of 30 fps or more, thereby solving the above-mentioned problems.

[0016] Figure 1 is a configuration diagram showing an example of a 3D data reproduction system according to the first embodiment. Figure 2 is a diagram illustrating the synchronization control of each LiDAR. Figure 3 is a diagram illustrating the synchronization control between the camera and the LiDAR. Figure 4 is a diagram illustrating the LiDAR external parameters stored in the LiDAR external parameter storage unit. Figure 5 is a configuration diagram showing an example of a 3D data reproduction system according to the second embodiment. Figure 6 is a configuration diagram showing an example of a 3D data reproduction system according to the third embodiment. Figure 7 is a flowchart showing the processing flow by the LiDAR data receiving unit of the 3D data merging device. Figure 8 is a flowchart showing the processing flow by the camera data receiving unit. Figure 9 is a flowchart showing the processing flow by the LiDAR / camera data merging unit. Figure 10 is a flowchart showing the processing flow by the LiDAR / camera combined data receiving unit. Figure 11 is a flowchart showing the processing flow by the LiDAR / camera combined data restoration unit.

[0017] Embodiments of the present invention will be described below with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes of the parts, etc., are not necessarily the same as those of reality. Furthermore, even when representing the same part, the dimensions and ratios may be represented differently in the drawings. In this specification and each drawing, elements similar to those described in previously shown drawings are denoted by the same reference numerals, and detailed explanations and redundant explanations are omitted as appropriate.

[0018] (First Embodiment) Figure 1 is a configuration diagram showing an example of a 3D data reproduction system according to the first embodiment.

[0019] The 3D data reproduction system 10 according to the first embodiment includes a 3D data measurement device 20, a time synchronization device 30, a 3D data merging device 40, and a 3D data reproduction device 50.

[0020] The 3D data measurement device 20 consists of multiple scanning LiDARs (hereinafter simply referred to as "LiDARs") and one camera 4. In Figure 1, three LiDARs, LiDAR1, LiDAR2, and LiDAR3, are shown as examples of multiple LiDARs, but this is just one example, and the number of LiDARs is not limited to three; it could be two, four or more.

[0021] Here, the frame rate of LiDAR is L ＦＰＳ The number of LiDAR units is n, and the frame rate of camera 4 is C ＦＰＳ Therefore, nL ＦＰＳ or C ＦＰＳ The larger of these values ​​will be the maximum frame rate for 3D measurement in the 3D data reproduction system 10.

[0022] Usually L ＦＰＳ is C ＦＰＳ Often smaller than C ＦＰＳ = 60, L ＦＰＳ When the value is 10, as shown in Figure 1, combining three LiDAR units (LiDAR1, LiDAR2, and LiDAR3) enables measurement at 30 fps, and combining six LiDAR units (not shown) enables measurement at 60 fps.

[0023] The time synchronization device 30 sends time information jj to each of the LiDAR1, LiDAR2, LiDAR3 and the camera 4 that constitute the 3D data measurement device 20, thereby achieving synchronized control between each of the LiDAR1, LiDAR2, LiDAR3 and the camera 4.

[0024] Figure 2 illustrates the synchronization control of each LiDAR.

[0025] As shown in Figure 2, the synchronization control of each LiDAR1, LiDAR2, and LiDAR3 is performed such that, assuming that the scanning direction of each LiDAR1, LiDAR2, and LiDAR3 is moving at a constant angular velocity, which is true for many scanning LiDARs, the scanning direction at the same time is shifted by 2π / n, depending on the number of LiDARs n.

[0026] For example, when the 3D data measurement device 20 is configured by three LiDARs, LiDAR 1, LiDAR 2, and LiDAR 3, the scanning direction of (a) LiDAR 1 at time t is θ Ｌ１ (t), the scanning direction of (b) LiDAR 2 is θ Ｌ２ (t), and the scanning direction of (c) LiDAR 3 is θ Ｌ３ (t), then θ Ｌ１ (t)=θ Ｌ２ (t)+2π / 3=θ Ｌ３ (t)+2*2π / 3 is controlled to hold.

[0027] Further, when n LiDARs are used, θ Ｌ１ (t)=θ Ｌ２ (t)+2π / n=...=θ Ｌｎ (t)+(n−1)*2π / n is controlled such that the scanning direction corresponding to time satisfies the above relationship.

[0028] FIG. 3 is a diagram explaining synchronous control between a camera and LiDAR 1.

[0029] For the synchronous control of the camera 4, as shown in FIG. 3, at time t １ , the angular direction of LiDAR 1

[0030] matches the optical axis direction θ of camera 4 ｃａｍ , then the shooting time of each frame of the moving image captured by camera 4 satisfies t １ +(1 / (nL ＦＰＳ ))*N (where N is any integer). It is further assumed that

[0031] is satisfied.

[0032] Returning to FIG. 1, as illustrated, the 3D data combining device 40 includes a LiDAR data receiving unit 41, a camera data receiving unit 42, a LiDAR / camera data combining unit 43, a LiDAR external parameter storage unit 44, and a camera internal and external parameter storage unit 45.

[0033] The LiDAR data receiving unit 41 receives, from the n-th LiDAR, data that the optical axis direction of the n-th LiDAR is θ Ｌｎ (t ｎ )=θ ｃａｍThe time t is when this occurs. ｎ Based on, time t ｎ -0.5 (1 / L) ＦＰＳ ) from t ｎ +0.5 (1 / L) ＦＰＳ Point cloud frame data up to (reference time t) ｎ The point cloud frame data fd(#n) is received.

[0034] L is the nth LiDAR. ｎ Each point cloud frame data fd (#n) is t ｎ Using this as the reference time, L ｎ Use as the reference coordinate system

[0035] Each point is represented by, at time t ｎ -0.5 (1 / L) ＦＰＳ ) from t ｎ +0.5 (1 / L) ＦＰＳ ) All point information measured

[0036] It is composed of, for example.

[0037] When it is composed of m points

[0038] and each point

[0039] is coordinate system

[0040] It has coordinate values ​​represented as follows. At this time, the point cloud frame data of each LiDAR1 to 3 is time 1 / L ＦＰＳ Each time, a new point cloud frame data is acquired, and the point cloud frame data between LiDARs is time 1 / nL. ＦＰＳ A phase difference will occur with each step.

[0041] The LiDAR data receiving unit 41 receives the reference time t ｎ and all point cloud information

[0042] The measured LiDAR number n is then transmitted to the LiDAR / camera data merging unit 43.

[0043] Based on the constraints and conditions already mentioned, the shooting time t of each frame of the video captured by camera 4 ｊThis will always coincide with one of the reference times of all point cloud frame data fd for all LiDAR1 to 3.

[0044] The camera data receiving unit 42 receives 2D image data captured in each frame.

[0045] , 1 / nL ＦＰＳ Each time the shooting time t ｊ Acquired as 2D image data. 2D image data

[0046] It is composed of pixels with dimensions (u, v), and each pixel holds 3 channels of RGB data.

[0047] The camera data receiving unit 42 receives the shooting time t ｊ and 2D image data

[0048] This is transmitted to the LiDAR / camera data merging unit 43. Note that if it is not an ideal pinhole camera, 2D image data is transmitted.

[0049] This data represents the result after correction for distortions in the camera lens itself and during mounting, such as radial distortion and tangential distortion.

[0050] The camera internal and external parameter storage unit 45 consists of two parts: camera internal parameters and camera external parameters.

[0051] Camera internal parameters

[0052] The horizontal and vertical focal lengths f ｘ , f ｙ and the principal point c, which is the optical center in the image plane. ｘ , c ｙ It is a 3x3 matrix composed of the following: camera coordinates

[0053] against

[0054] This holds true.

[0055] Camera external parameter K is the reference coordinate of each LiDAR.

[0056] and camera coordinates

[0057] A rotation matrix that connects them.

[0058] and translation vector

[0059] When there are three LiDARs, LiDAR1, LiDAR2, and LiDAR3, the camera's external parameters are

[0060] There are three types.

[0061] In the camera's external parameters, for all LiDARs, the corresponding LiDAR number n is set.

[0062] Save it.

[0063] Camera internal and external parameters are estimated through a measurement process known as calibration.

[0064] Figure 4 illustrates the LiDAR external parameters stored in the LiDAR external parameter storage unit 44.

[0065] The LiDAR external parameters stored in the LiDAR external parameter storage unit 44 are, as shown in Figure 4, the overall reference coordinates.

[0066] and the reference coordinates of each LiDAR1 to 3

[0067] A rotation matrix that connects them.

[0068] and translation vector

[0069] Composed of,

[0070] Let the following hold true. Generally speaking

[0071] The reference coordinate L ｎ These are referred to as external parameters of LiDAR1-3.

[0072] LiDAR external parameters are also estimated through a measurement process known as calibration, and these estimates contain errors.

[0073] When there are three LiDARs, LiDAR1, LiDAR2, and LiDAR3, the external parameters are:

[0074] There are three types. However, the conversion from camera coordinates to reference coordinates, as described later, is performed according to each LiDAR1 to 3.

[0075] When using these three external parameters, the errors present in each of them influence the coloring of the 3D data acquired frame by frame, resulting in fluctuations.

[0076] When observing continuous data of a colored point cloud, people are more sensitive to the shift in the colored position over time than to the shift in the position of the points over time. Therefore, in the 3D data reproduction system 10, in order to reduce visual fluctuations, one of the three LiDARs 1 to 3 is designated as the reference LiDAR, and the reference LiDAR number i and the external parameters corresponding to number i are defined.

[0077] We will devise a method to use only that.

[0078] The LiDAR / camera data merging unit 43 receives data from the LiDAR data receiving unit 41.

[0079] And, received from the camera data receiving unit 42

[0080] Of these, t ｎ and t ｊ Search for pairs where and match.

[0081] The following process will be carried out regarding this matter.

[0082] The LiDAR / camera data merging unit 43 first receives the number i from the LiDAR external parameter storage unit 44 and

[0083] The data is read, and then the camera's internal parameters are retrieved from the camera's internal and external parameter storage unit 45.

[0084] And, the camera external parameters that match the n of the matched pair.

[0085] Furthermore, camera external parameters that match number i

[0086] Load the information.

[0087] Next, from the loaded data

[0088] camera image

[0089] First, convert it to a depth image that matches.

[0090] LiDAR reference coordinates

[0091] From camera coordinate P ｃ Converted

[0092] The following calculation

[0093] We will find it using this method. Furthermore,

[0094] Each point

[0095] Apply the following equation to convert it into coordinates (u, v) of a 2D image.

[0096] When converted to 2D image coordinates (u, v), the depth value of one channel of each coordinate is

[0097] Apply this. However,

[0098] In that case, remove the relevant point and round (u, v) to an integer.

[0099] When we find the same coordinate values

[0100] If there are multiple points with the same value, only the single point with the smallest value that is greater than 0 is selected.

[0101] Also, if the vertical and horizontal resolutions of the image to be converted are X and Y,

[0102] Remove any pixels that do not match the specified value.

[0103] This process

[0104] The process was performed on all points, and the resulting 1-channel depth image was obtained.

[0105] and 3ch RGB image

[0106] Combine them to create a 4-channel RGBD image.

[0107] The LiDAR / camera data merging unit 43 obtains the following.

[0108] and measurement time t ｎ ,moreover

[0109] Camera internal parameters to return to the reference coordinates

[0110] The set is sent to the 3D data reproduction device 50.

[0111] At this time

[0112] Any one point on

[0113] teeth

[0114] By converting it to reference coordinates (X, Y, Z) using the following formula, it can be converted to a single point (X, Y, Z, r, g, b) on the reference coordinates that has color information.

[0115] At this time

[0116] This was read from the LiDAR external parameter storage unit 44.

[0117] And the camera external parameter that matches the number i read from the LiDAR external parameter storage unit 44

[0118] twist

[0119] It is expressed as follows: At this time, from the properties of the inverse matrix...

[0120] That is. Also, ｃ teeth

[0121] It can be expressed as follows.

[0122] Next, as shown in Figure 1, the 3D data reproduction device 50 consists of a LiDAR / camera combined data receiving unit 51 and a LiDAR / camera combined data restoration unit 52.

[0123] The LiDAR / camera data receiving unit 51 receives data transmitted by the LiDAR / camera data merging unit 43 of the 3D data merging device 40.

[0124] It receives these five elements. The received data is then sent to the LiDAR / camera combined data reconstruction unit 52.

[0125] The LiDAR / camera combined data recovery unit 52 uses the aforementioned formula (a) to recover the data received from the LiDAR / camera combined data receiving unit 51.

[0126] Convert to reference coordinates. The point converted to reference coordinates is at time t. ｎ This will be colored point data (X, Y, Z, r, g, b).

[0127] The point cloud obtained by converting all of them

[0128] However, the time t was reconstructed as 3D data. ｎ This is a colored point cloud.

[0129] Received from LiDAR / camera combined data receiving unit 51

[0130] Regarding t ｎ If there are multiple values ​​with the same value, they are converted to (X, Y, Z, r, g, b) as colored point data for the same time, and then...

[0131] It will be joined to

[0132] According to the 3D data reproduction system 10 of the first embodiment described above, by combining multiple scanning LiDARs 1 to 3 and a camera 4, and by devising a method for combining point cloud data, it becomes possible to measure high-quality, colored real-time point clouds with minimal shaking at a high frame rate of 30 fps or more.

[0133] (Second Embodiment) Next, a 3D data reproduction system according to the second embodiment will be described. Figure 5 is a configuration diagram showing an example of the 3D data reproduction system according to the second embodiment.

[0134] In Figure 5, the same reference numerals are used for the same parts as in Figure 1, and in this embodiment, redundant explanations of parts that were described in the first embodiment are avoided.

[0135] The 3D data reproduction system 10A according to the second embodiment illustrated in Figure 5 comprises a plurality of 3D data measurement devices 20 (in this example, three: 20(#1), 20(#2), and 20(#3)), a single time synchronization device 30 provided in common to the plurality of 3D data measurement devices 20, a plurality of 3D data merging devices 40 (in this example, three: 40(#1), 40(#2), and 40(#3)) connected one-to-one to each of the plurality of 3D data measurement devices 20, and a single 3D data reproduction device 50 provided in common to the plurality of 3D data merging devices 40.

[0136] The time synchronization device 30 provides the same time information jj to all LiDARs 1 to 3 and cameras 4 of each 3D data measurement device 20 (#1) to (#3).

[0137] The 3D data measurement device 20 and the 3D data merging device 40 are connected on a one-to-one basis. That is, the 3D data measurement device 20 (#1) is connected to the 3D data merging device 40 (#1), the 3D data measurement device 20 (#2) is connected to the 3D data merging device 40 (#2), and the 3D data measurement device 20 (#3) is connected to the 3D data merging device 40 (#3).

[0138] In each pair, the connection relationship between the 3D data measurement device 20 and the 3D data merging device 40, as well as the data flow, are as described in the first embodiment.

[0139] Note that while Figure 5 shows an example with three pairs of 3D data measurement devices 20 and 3D data merging devices 40, the number of pairs can be any number and is not limited to three.

[0140] The LiDAR external parameter storage unit 44 and the camera internal and external parameter storage units 45 of each 3D data merging device 40 shall store the values ​​of the paired 3D data measurement device 20. The data to be stored is the same as described above.

[0141] Furthermore, each of the 3D data merging devices 40 (#1) to (#3) is connected to a single 3D data reproduction device 50 that is provided in common. As a result, each LiDAR / camera data merging unit 43 of all 3D data merging devices 40 (#1) to (#3) transmits a signal to the LiDAR / camera merged data receiving unit 51 of the common 3D data reproduction device 50.

[0142] By performing the process described in the first embodiment using a 3D data reproduction system 10A equipped with multiple pairs of 3D data measurement devices 20 and 3D data merging devices 40, as in this embodiment, it becomes possible to measure the subject from multiple positions. The LiDAR / camera combined data restoration unit 52 of the 3D data reproduction device 50 then processes the measurement results of 3D data with less occlusion and 3D data over a wider range, while maintaining a high frame rate, at each time step t. ｎ Colored point cloud

[0143] It can be reproduced as follows.

[0144] As described above, the 3D data reproduction system 10A according to the second embodiment allows for measurement of the subject from more positions compared to the 3D data reproduction system 10 according to the first embodiment illustrated in Figure 1. Therefore, it is possible to measure 3D data with less occlusion and 3D data over a wider range while maintaining a high frame rate.

[0145] Furthermore, since the processing according to the first and second embodiments is lightweight, it is entirely possible to achieve the entire process from measurement to playback in a short time of less than one second. Also, although outside the scope of the present invention,

[0146] By viewing the data from any viewpoint through a virtual camera, it becomes possible to enjoy it as 3D content with a free viewpoint and movement.

[0147] (Third Embodiment) Next, a 3D data reproduction system according to the third embodiment will be described.

[0148] Figure 6 is a configuration diagram showing an example of a 3D data reproduction system according to the third embodiment.

[0149] In Figure 6, the same reference numerals are used for the same parts as in Figure 1, and in this embodiment, redundant explanations of parts that were described in the first embodiment are avoided.

[0150] The 3D data reproduction system 10B illustrated in Figure 6 comprises two 3D data measurement devices 20 (#1) and (#2), one time synchronization device 30 provided in common to the two 3D data measurement devices 20 (#1) and (#2), two 3D data merging devices 40 (#1) and (#2) connected one-to-one to each of the two 3D data measurement devices 20 (#1) and (#2), and one 3D data reproduction device 50 provided in common to the two 3D data merging devices 40 (#1) and (#2).

[0151] Both 3D data measurement devices 20 (#1) and (#2) consist of two LiDARs and one camera 4. Specifically, 3D data measurement device 20 (#1) comprises two LiDARs 11 and 12 and one camera 4 (#1), while 3D data measurement device 20 (#2) comprises two LiDARs 21 and 22 and one camera 4 (#2).

[0152] Both the 3D data merging devices 40 (#1) and (#2) are equipped with a LiDAR data receiving unit 41, a camera data receiving unit 42, a LiDAR / camera data merging unit 43, a LiDAR external parameter storage unit 44, and a camera internal and external parameter storage unit 45, similar to those described with reference to Figure 1. Specifically, the 3D data merging device 40 (#1), provided in conjunction with the 3D data measurement device 20 (#1), comprises a LiDAR data receiving unit 41 (#1), a camera data receiving unit 42 (#1), a LiDAR / camera data merging unit 43 (#1), a LiDAR external parameter storage unit 44 (#1), and a camera internal and external parameter storage unit 45 (#1). The 3D data merging device 40 (#2), provided in conjunction with the 3D data measurement device 20 (#2), comprises a LiDAR data receiving unit 41 (#2), a camera data receiving unit 42 (#2), a LiDAR / camera data merging unit 43 (#2), a LiDAR external parameter storage unit 44 (#2), and a camera internal and external parameter storage unit 45 (#2).

[0153] An example of the operation of a 3D data reproduction system 10B with this configuration will be described.

[0154] The time synchronization device 30 can use equipment that can acquire time with high precision, such as a PTP grandmaster clock. In this embodiment, the frame rate of all LiDARs is set to 10 fps, and the frame rate of all cameras 4 is set to 20 fps. At this time, all LiDARs 11, 12, 21, and 22 of the 3D data measurement devices 20 (#1) and (#2) are used that can control the rotation direction based on the PTP time. The scan direction of LiDAR 11 and LiDAR 12, and LiDAR 21 and LiDAR 22 at each time is set to θ Ｌ１ (t) and θ Ｌ２ Let (t) be θ Ｌ１ (t) = θ Ｌ２ The rotation direction of each LiDAR should be controlled based on the PTP time so as to satisfy (t) + π.

[0155] Furthermore, the cameras 4(#1) and (#2) of the 3D data measurement devices 20(#1) and (#2) are equipped with cameras that can control the shutter timing based on the PTP time. The optical axis direction of cameras 4(#1) and 4(#2) is set to θ ｃａｍ Therefore, the rotation direction of LiDAR11 and LiDAR21 at t=0 is

[0156] This is the same as camera 4(#1) and camera 4(#2), and camera 4(#1) and camera 4(#2) are controlled so that images are taken at the timings {0, 0.05, 0.10, 0.15…} based on 0 + (1 / (2*10))*N from t=0.

[0157] Figure 7 is a flowchart showing the processing flow by the LiDAR data receiving units 41 (#1) and (#2) of the 3D data merging device 40 (#1) and (#2).

[0158] The 3D data merging device 40 (#1) receives data transmitted from LiDAR 11 and LiDAR 21 of the 3D data measurement device 20 (#1), and the 3D data merging device 40 (#2) receives data transmitted from LiDAR 12 and LiDAR 22 of the 3D data measurement device 20 (#2).

[0159] LiDAR11, LiDAR12, LiDAR21, and LiDAR22 have LiDAR numbers 11, 12, 21, and 22, respectively, and the reference time t ｎ And, time t ｎ -0.5 (1 / L) ＦＰＳ =10) from t ｎ All point information measured at +0.5 (1 / 10)

[0160] The following is transmitted. At this time, LiDAR11 and LiDAR21 are assumed to have a reference time of {0, 0.10, 0.20, 0.30...}, and LiDAR12 and LiDAR22 are assumed to have a reference time of {0.05, 0.15, 0.25, 0.35...}.

[0161] In process S411, the reference time t ｎ , all point information with each LiDAR as the reference coordinate.

[0162] The system receives three pieces of data: the data itself and the Lidar number. Once received, the process proceeds to step S412.

[0163] In process S412, the reference time t ｎ , full information

[0164] The three data points, including LiDAR number n, are sent to the LiDAR / camera data merging unit 43. Once the transmission is complete, the process ends and returns to the start, repeating the process.

[0165] Figure 8 is a flowchart showing the processing flow by the camera data receiving units 42 (#1) and (#2).

[0166] Camera data receiving unit 42 (#1) receives data transmitted from camera 4 (#1) of 3D data measurement device 20 (#1), and camera data receiving unit 42 (#2) receives data transmitted from camera 4 (#2) of 3D data measurement device 20 (#2).

[0167] The time of shooting t for camera 4 (#1) and camera 4 (#2) ｊ Both values ​​are {0, 0.05, 0.10, 0.15...}, and each shooting time t ｊ RGB image

[0168] The images transmitted from each of the cameras 4(#1) and (#2) are sent to the camera data receiving units 42(#1) and (#2), respectively. In this embodiment, for simplicity, cameras 4(#1) and 4(#2) are ideal pinhole cameras without distortion. Then, in processing S421, the images transmitted from each camera 4(#1) and (#2) are sent to the camera data receiving units 42(#1) and (#2), respectively.

[0169] and shooting time t ｊ The system receives the data and then proceeds to process S422.

[0170] In processing S422, the image

[0171] and shooting time t ｊ The two sets of data are sent to the LiDAR / camera data merging unit 43. Once the transmission is complete, the process ends, and the process returns to the start and is repeated.

[0172] In the LiDAR external parameter storage units 44 (#1) and (#2), the reference LiDAR number i and the external parameter corresponding to number i are stored.

[0173] The data is stored. The rotation matrix is ​​a 3x3 matrix, and specific parameters for each matrix can be set using quaternions, etc. The translation vector is a three-dimensional vector, and each parameter is determined by the translational displacement of each axis.

[0174] In the LiDAR external parameter storage unit 44 (#1), the reference LiDAR number is set to 11.

[0175] This is an external parameter of LiDAR11 relative to the reference coordinates.

[0176] In the LiDAR external parameter storage unit 44 (#2), the reference LiDAR number is set to 21.

[0177] These are external parameters of LiDAR21 relative to the reference coordinates.

[0178] The camera internal and external parameter storage units 45 (#1) and (#2) store two sets of parameters: camera internal parameters and camera external parameters.

[0179] Camera internal parameters

[0180] The values ​​are set according to camera 4 (#1) and camera 4 (#2), and the horizontal and vertical focal length f ｘ , f ｙ and the principal point c, which is the optical center in the image plane. ｘ , c ｙ This results in a 3x3 matrix composed of the elements.

[0181] Next, the camera external parameters consist of a rotation matrix and a translation vector that connect the reference coordinates of each LiDAR to the camera coordinates.

[0182] In the camera's internal and external parameter storage unit 45 (#1), the camera's external parameters are stored for LiDAR 11 and LiDAR 12.

[0183] It consists of LiDAR number 11 and

[0184] As a pair, LiDAR number 12 and

[0185] These are saved as a pair.

[0186] Furthermore, in the camera's internal and external parameter storage unit 45 (#2), the camera's external parameters are stored for LiDAR 21 and LiDAR 22.

[0187] It consists of LiDAR number 21 and

[0188] As a pair, LiDAR number 22 and

[0189] These are saved as a pair.

[0190] Figure 9 is a flowchart showing the processing flow by the LiDAR / camera data merging units 43 (#1) and (#2).

[0191] In process S431, data is received from the LiDAR data receiving unit 41 or the camera data receiving unit 42. The LiDAR data receiving unit 41 receives the reference time t ｎ , full information

[0192] It consists of three datasets: , and Lidar number n.

[0193] It receives the shooting time t from the camera data receiving unit 42. ｊ and image

[0194] It consists of two datasets.

[0195] The data is received. In the LiDAR / camera data merging unit 43, once one of the data is received, the data is saved and processing proceeds to the next step.

[0196] Next, in process S432, the reference time t of the data received from the LiDAR data receiving unit 41 is included in the data received in process 431. ｎAnd the time of shooting t of the data received from the camera data receiving unit 42 ｊ A comparison is performed to see if there are any matches. If there are no matches (S432: No), the process returns to S431 and waits for new data to be received. If there are matches (S432: Yes), the matching data pairs are joined.

[0197] This is sent to the next process, S433.

[0198] In process 433, the LiDAR external parameter storage unit 44 retrieves the LiDAR number i and

[0199] The LiDAR external parameter storage unit 44 (#1), which is read by the LiDAR / camera data merging unit 43 (#1), has i=11.

[0200] Only is stored there, and the LiDAR external parameter storage unit 44 (#2) read by the LiDAR / camera data merging unit 43 (#2) has i=21.

[0201] Since only the LiDAR number i and

[0202] This is uniquely determined for each 3D data merging device 40.

[0203] Once the data has been read, the process proceeds to the next step, S434.

[0204] In processing S434, the camera internal parameters are stored from the camera internal and external parameter storage unit 45.

[0205] And, camera external parameters that match n combined in process 432.

[0206] Furthermore, camera external parameters that match the LiDAR number i read in process 433

[0207] The data is loaded. Once the data is loaded, the process proceeds to the next step, S435.

[0208] In process S435, the combined in process S432

[0209] The camera external parameters read in processing S434

[0210] Therefore, in the following calculation

[0211] Convert to [this format].

[0212] In this way

[0213] Once the conversion to is complete, the process proceeds to S436.

[0214] In process S436,

[0215] Each point

[0216] Apply the aforementioned equation (1) to convert it into coordinates (u, v) of a two-dimensional image.

[0217] Next, when converted to the coordinates (u, v) of a 2D image, the depth value of one channel of each coordinate is

[0218] Apply this. However,

[0219] In that case, remove the relevant point and round (u, v) to an integer.

[0220] When we find the same coordinate values

[0221] When there are multiple points with the same value, only the single point with the smallest value greater than 0 is selected. Also, when the vertical and horizontal resolutions of the image to be transformed are X and Y,

[0222] Remove pixels that do not match the value.

[0223] This was performed on all points, and the 1-channel depth image

[0224] Obtain a 1-channel depth image.

[0225] Once obtained, proceed to process S437.

[0226] In processing S437, the 1-channel depth image

[0227] And, combined in process S432

[0228] By combining these, a 4-channel RGB image is created.

[0229] This is obtained. Then, the data read in processes S433 and S434

[0230] Based on the following calculation, R ｃ and t ｃ We seek.

[0231] Then, in process S432, the combined t ｎ And, read in process S434

[0232] Together

[0233] The data is sent to the 3D data reproduction device 50. Once the transmission is complete, the process ends and returns to process S431.

[0234] Figure 10 is a flowchart showing the processing flow by the LiDAR / camera combined data receiving unit 51.

[0235] In process 611, from the 3D data merging device 40 (#1) or the 3D data merging device 40 (#2)

[0236] The system receives the data. Once the data is received, the process proceeds to step S612.

[0237] In process S612

[0238] The data is sent to the LiDAR / camera combined data restoration unit 52. Once the data transmission is complete, the process is terminated and the process returns to process S611.

[0239] Figure 11 is a flowchart showing the processing flow by the LiDAR / camera combined data recovery unit 52.

[0240] In processing S621, the LiDAR / camera combined data receiving unit 51 transmitted

[0241] The system receives the data. If the received data is the first data received from the start, the time the first data was received is recorded. If no data has been received, a certain time interval of 1 / 2C is recorded from the time the first data was received. ＦＰＳ In other words, in this embodiment, if 0.025 seconds or more have elapsed, the process proceeds to S622.

[0242] In process S622, a certain time interval of 1 / 2C is taken from the time the first data was received in process S621. ＦＰＳ In other words, in this embodiment, if 0.025 seconds or more have elapsed (S622: Yes), the process proceeds to S623. If not enough time has elapsed (S622: No), the process returns to S621.

[0243] In process S623, the entire dataset received in process S621 is processed.

[0244] Applying equation (a) mentioned above,

[0245] The RGBD data is expanded as colored point data (X, Y, Z, r, g, b) of the reference coordinates. In this embodiment, the dataset received in processing S623 is at the same time t ｎ There are two sets of data: one from a 3D data merging device 40 (#1) and one from a 3D data merging device 40 (#2), and by merging the two sets of colored point data, time t ｎ Colored point cloud of reference coordinates

[0246] It can be reproduced as follows.

[0247] This is reproduced in process S623.

[0248] In this embodiment, C per second ＦＰＳ =It is generated in 20 steps,

[0249] The data can be viewed from any viewpoint through a virtual camera, allowing it to be enjoyed as 3D content with a free viewpoint.

[0250] As described above, the 3D data reproduction system according to each embodiment of the present invention, by combining multiple scanning LiDARs and cameras and devising a method for combining point cloud data, makes it possible to measure high-quality, colored real-time point clouds with minimal shaking at a high frame rate of 30 fps or more.

[0251] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate as possible, and in that case, the combined effects can be obtained. Moreover, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple constituent elements disclosed.

[0252] 1-3 Lidar 4 Camera 10, 10A, 10B 3D Data Reproduction System 20 3D Data Measurement Device 30 Time Synchronization Device 40 3D Data Combining Device 41 Lidar Data Receiving Unit 42 Camera Data Receiving Unit 43 Lidar / Camera Data Combining Unit 44 Lidar External Parameter Storage Unit 45 Camera Internal and External Parameter Storage Unit 50 3D Data Reproduction Device 51 Lidar / Camera Combined Data Receiving Unit 52 Lidar / Camera Combined Data Restoration Unit

Claims

1. A 3D data reproduction system comprising: a 3D data measurement device including multiple LiDARs and one camera; and a time synchronization device that sends the same time information to the multiple LiDARs and the one camera and performs synchronization control between the multiple LiDARs and the one camera, thereby synchronizing point cloud data from the multiple LiDARs and the one camera and achieving high frame rate color point cloud measurement.

2. The 3D data merging system according to claim 1, further comprising: a 3D data merging device that, when there is a pair of LiDAR data and camera data with matching time data among the LiDAR data output from the plurality of LiDARs and the camera data output from the camera, converts the point cloud data for the matching pair into a camera coordinate system, projects the converted point cloud data onto a two-dimensional image to obtain a depth image, and sends out data including an RGBD image obtained based on the depth image, corresponding time information, pre-prepared camera internal parameters, and pre-prepared camera external parameters to reference coordinates; and a 3D data reproduction device that receives the data sent out from the 3D data merging device and converts the received data into a colored 3D point cloud of the reference coordinates.

3. The 3D data reproduction system according to claim 2, comprising a plurality of equal numbers of the 3D data measurement devices and the 3D data merging devices, wherein one 3D data measurement device and one 3D data merging device are connected in a one-to-one relationship, the time synchronization device is provided in common to the plurality of 3D data measurement devices and sends the same time information to all LiDARs and cameras included in the plurality of 3D data measurement devices, and the 3D data reproduction device is provided in common to the plurality of 3D data merging devices and receives data transmitted from the plurality of 3D data merging devices.

4. The 3D data merging device includes a data merging unit that converts the point cloud data into a camera coordinate system, and a LiDAR external parameter storage unit that stores LiDAR external parameters used by the data merging unit when converting the point cloud data into a camera image system, wherein the 3D data measuring device predetermines one of a plurality of LiDARs included in the 3D data measuring device as a reference LiDAR, and the data merging unit uses the LiDAR external parameters of the reference LiDAR from among the LiDAR external parameters stored in the LiDAR external parameter storage unit as LiDAR external parameters used when converting the point cloud data into a camera coordinate system, the 3D data reproduction system according to claim 2.

Images