3D data processing device and 3D data processing method

Abstract

A 3D data processing device according to one embodiment of the present invention acquires 3D data from each of a plurality of measurement devices, calculates a weight for each of the plurality of measurement devices on the basis of a first direction indicating the optical axis center direction of a virtual camera and a second direction indicating the optical axis center direction for each of the plurality of measurement devices, performs rendering processing at the viewpoint of the virtual camera on the basis of the 3D data and the weights, and generates display data.

Description

3D data processing device and 3D data processing method

[0001] The present invention relates to a 3D data processing device and a 3D data processing method.

[0002] In the entertainment fields such as sports and music concerts, a technology that acquires 3D data of athletes, performers, and the entire space, and then live streams this acquired 3D data, is attracting attention. This technology makes it possible for viewers (hereinafter referred to as users) to enjoy 3D content in real time through immersive display devices such as 3D displays and head-mounted displays.

[0003] On the other hand, 3D data measured by 3D data measurement devices contains unique noise. For example, when combining 3D data measured by multiple 3D data measurement devices, simply combining the noisy 3D data can result in the noise in the combined data impairing the user's experience.

[0004] Mai Saito, et al., "Interpolation Method for Long-Range Sparse Point Clouds by Sensor Fusion Using LiDAR and Camera," Transactions of the Society of Automotive Engineers of Japan, Vol. 53, No. 3, May 2022.

[0005] In prior art, such as the technology disclosed in Non-Patent Document 1, 3D data measured by a device combining LiDAR and a camera contains unique noise. Therefore, when 3D data measured using multiple such devices are simply combined and rendered through a virtual camera, the noise contained in the data impairs the user's experience when viewing the 3D data.

[0006] The present invention aims to improve the user experience by reducing the impact of noise in 3D data, which contains noise, measured by multiple 3D data measurement devices, by devising a processing method during rendering based on the pose of a virtual camera operated by the user.

[0007] A 3D data processing device according to one aspect of the present invention acquires 3D data from each of a plurality of measuring devices, calculates a weight for each of the plurality of measuring devices based on a first direction indicating the direction of the optical axis center of a virtual camera and a second direction indicating the direction of the optical axis center for each of the plurality of measuring devices, and generates display data by performing rendering processing at the viewpoint of the virtual camera based on the 3D data and the weights.

[0008] According to the present invention, a technology is provided that reduces the influence of noise in 3D data and improves the user experience in displayed data.

[0009] Figure 1 is a block diagram showing an example of a specific configuration of a 3D data processing system including a 3D data processing device according to an embodiment. Figure 2 is a block diagram showing an example of the hardware configuration of the 3D data processing device in Figure 1. Figure 3 is a block diagram showing an example of the functional configuration of the 3D data processing device in Figure 2. Figure 4 is a table showing an example of virtual camera information in an embodiment. Figure 5 is a table showing an example of measurement device information in an embodiment. Figure 6 is a flowchart showing an example of 3D data acquisition processing in a 3D data processing device according to an embodiment. Figure 7 is a flowchart showing an example of virtual camera position control processing in a 3D data processing device according to an embodiment. Figure 8 is a flowchart showing an example of 3D data rendering processing in a 3D data processing device according to an embodiment. Figure 9 is a perspective view illustrating the positional relationship between the measurement device, the subject, and the background object. Figure 10 is a top view of Figure 9. Figure 11 is a perspective view illustrating the positional relationship between the measurement device, the virtual camera, the subject, and the background object. Figure 12 is a diagram showing an example of a displayed image from the viewpoint of the virtual camera in Figure 11. Figure 13 is a perspective view illustrating another positional relationship between the measurement device, the virtual camera, the subject, and the background object. Figure 14 shows an example of a display image from the viewpoint of the virtual camera in Figure 13. Figure 15 is a perspective view illustrating a different positional relationship between multiple measuring devices, a virtual camera, a subject, and background objects. Figure 16 shows an example of a display image from the viewpoint of the virtual camera in Figure 15. Figure 17 shows an example of a pixel of interest in the display image of Figure 16. Figure 18 shows an example where a noise point is preferred over a non-noise point in a pixel of interest. Figure 19 shows an example where a weight related to the measuring instrument is used to prioritize a non-noise point over a noise point in a pixel of interest. Figure 20 is a diagram illustrating the angle between the optical axis center direction of the virtual camera and the optical axis center direction of the measuring device.

[0010] Before describing embodiments of the present invention, we will explain the problems caused by and solutions to the specific noise contained in 3D data.

[0011] The unique noise contained in 3D data is, for example, erroneous data points resulting from measurement errors. When rendering 3D data containing these erroneous data points, the erroneous data points are also made visible in the 2D display data, which can impair the user's experience (which may also be called the "visual experience"). Below, we will explain the first problem concerning noise contained in 3D data measured with a single measuring device, the second problem concerning noise contained in multiple 3D data sets measured with multiple measuring devices, and solutions to the problem concerning noise contained in multiple 3D data sets.

[0012] (Problem 1) In Problem 1, we consider a case where 3D data of an object is acquired using a single measuring device, and the acquired 3D data is rendered from the viewpoint of a virtual camera.

[0013] Figure 9 is a perspective view illustrating the positional relationship between the measuring device, the subject, and the background object. Figure 10 is a top view of Figure 9. In the following perspective views, the shooting direction of the measuring device 10 in Figures 9 and 10 is defined as the Y direction, the direction perpendicular to the horizontal plane including the Y direction is defined as the Z direction, and the direction perpendicular to the YZ plane including the Y and Z directions is defined as the X direction.

[0014] In the direction of the measurement device 10, the subject OBJ and the background object BO are arranged in order. The subject OBJ is cylindrical, with its central axis coinciding with the Z direction. The background object BO is rectangular, with the largest surface area corresponding to the XZ plane, which includes the X and Z directions. Furthermore, the subject OBJ and the background object BO are positioned and sized to fit within the field of view of the measurement device 10. The arrangement of the subject OBJ and the background object BO is the same in subsequent examples.

[0015] As shown in Figure 10, in the positional relationship between the measuring device 10 and the subject OBJ, a measurement line ML is assumed to extend radially from the measurement center point CP of the measuring device 10. In this case, if a part of the measurement line ML is tangent to the subject OBJ, the 3D data relating to the tangent measurement line ML cannot be uniquely determined to represent the depth of the subject OBJ or the depth of the background object BO. As a result, noise data (which may also be called "measurement noise") extending like a tail from the point of contact C between the measurement line ML and the subject OBJ towards the background object BO may be measured. Note that this noise data does not represent the accurate depth. Furthermore, from now on, the range of data points in which the accurate depth can be measured will be referred to as the normal range NR, and the range of data points in which the accurate depth cannot be measured will be referred to as the abnormal range AR.

[0016] Next, we consider the placement of a virtual camera corresponding to the user's viewpoint in the positional relationship between the measuring device 10, the subject OBJ, and the background object BO shown in Figure 9.

[0017] Figure 11 is a perspective view illustrating the positional relationship between the measuring device, virtual camera, subject, and background object. Figure 12 is a diagram showing an example of the displayed image from the viewpoint of the virtual camera in Figure 11. In Figure 11, the measuring device 10 and the virtual camera VC are shown offset from each other, but they are assumed to be located in the same position and have the same shooting direction (which may also be called "orientation").

[0018] As shown in Figure 11, the virtual camera VC is positioned in the same location and orientation as the measuring device 10. When the 3D data measured by the measuring device 10 is rendered from the viewpoint of the virtual camera VC, the user will see the display image IMG1 shown in Figure 12.

[0019] The displayed image IMG1 shows the side view of the subject OBJ and the background object BO. Noise data N1 is also visualized and displayed around the edge (outer boundary) of the subject OBJ. This noise data N1 can occur in a small area around the subject OBJ. Therefore, even if the user views the displayed image IMG1, the impact of the noise data N1 on their perception is small.

[0020] Next, we consider placing the virtual camera VC in a different position relative to the measuring device 10, virtual camera VC, subject OBJ, and background object BO shown in Figure 11.

[0021] Figure 13 is a perspective view illustrating an alternative positional relationship between the measuring device, virtual camera, subject, and background objects. Figure 14 is a diagram showing an example of the displayed image from the viewpoint of the virtual camera in Figure 13.

[0022] As shown in Figure 13, the virtual camera VC is positioned to photograph the subject OBJ from a shooting direction 90 degrees different from the shooting direction of the measuring device 10. When the 3D data measured by the measuring device 10 is rendered from the viewpoint of the virtual camera VC, the user will see the display image IMG2 shown in Figure 14.

[0023] The displayed image IMG2 shows a side view of the subject OBJ. However, the subject OBJ contains both a normally rendered portion within the normal range NR and an abnormally rendered portion within the abnormal range AR. The normally rendered portion and the abnormally rendered portion look significantly different. Therefore, when a user views the displayed image IMG2, the impact of the abnormally rendered portion on their perception is significant (i.e., it impairs the perception).

[0024] (Second Problem) In the second problem, we consider a case where 3D data of an object is acquired using multiple measuring devices, and the acquired 3D data is rendered from the viewpoint of a virtual camera.

[0025] Figure 15 is a perspective view illustrating an alternative positional relationship between multiple measuring devices, a virtual camera, a subject, and background objects. Figure 16 is a diagram showing an example of the displayed image from the viewpoint of the virtual camera in Figure 15. Note that three measuring devices (first measuring device 10-1, second measuring device 10-2, and third measuring device 10-3) are assumed to be the multiple measuring devices. Also, although measuring device 10-1 and virtual camera VC are shown offset from each other in Figure 15, they are assumed to be located in the same position and have the same shooting direction.

[0026] As shown in Figure 15, the three measuring devices are oriented in different shooting directions relative to the subject OBJ. Specifically, the first measuring device 10-1 is positioned with the Y direction as its shooting direction relative to the subject OBJ, similar to the measuring device 10 in Figure 9. The second measuring device 10-2 is positioned with the X direction as its shooting direction relative to the subject OBJ. The third measuring device 10-3 is positioned with the -X direction as its shooting direction relative to the subject OBJ. The virtual camera VC is positioned in the same position and orientation as the first measuring device 10-1. When the three 3D data measured by the three measuring devices are combined and rendered from the viewpoint of the virtual camera VC, the user will see the display image IMG3 shown in Figure 16.

[0027] The displayed image IMG3 shows the side view of the subject OBJ and the background object BO. Noise data N1 is visualized and displayed on the outer edge of the subject OBJ, and noise data N2, which originates from the 3D data of the second measuring device 10-2, and noise data N3, which originates from the 3D data of the third measuring device 10-3, are also visualized and displayed on the subject OBJ.

[0028] These noise data N2 and noise data N3 are located closer to the virtual camera VC than the 3D data with normal depth acquired by the first measuring device 10-1. As a result, the abnormal rendering portion (which corresponds to the noise data N2 and noise data N3) is made visible overlapping the normal rendering portion. As mentioned above, the normal rendering portion and the abnormal rendering portion look significantly different. Therefore, when a user views the displayed image IMG3, the impact of the abnormal rendering portion on their perception is considerable.

[0029] From the above, as shown in Figure 11, rendering 3D data using a single measuring device may have little impact on the user's experience if the position and orientation of the measuring device and the virtual camera are the same. On the other hand, as shown in Figure 15, increasing the number of measuring devices may impair the user's experience even if the position and orientation of any of the multiple measuring devices and the virtual camera are the same.

[0030] (Solution) Regarding the multiple 3D data measured by multiple measuring devices in the second problem described above, we will explain the method for processing point cloud data using Figures 17 to 19.

[0031] Figure 17 shows an example of a pixel of interest in the image displayed in Figure 16. Figure 18 shows an example of a case where noise points are prioritized over non-noise points in a pixel of interest.

[0032] In Figure 17, we focus on any one pixel P that makes up the subject OBJ containing noise data N2. When rendering from the viewpoint of the virtual camera VC, the problem of noise being displayed can be attributed to the different distances from the virtual camera VC to multiple points (measurement points). Specifically, as shown in Figure 18, this is because the distance d1 from the virtual camera VC to a measurement point p1 with a normal depth (which may also be called a "correct point" or "non-noise point") is greater than the distance d2 from the virtual camera VC to a measurement point p2 with an abnormal depth (which may also be called a "noise point"). Therefore, since the pixel value of the measurement point that is closer to the viewpoint takes precedence, noise data is displayed on pixel P.

[0033] As a solution to the above problem, we will consider a method of weighting 3D data measured by multiple measuring devices when determining the pixel values ​​of measurement points when generating a rendered image from the viewpoint of a virtual camera. For example, in Figure 18, since weighting is not considered, the distance from the virtual camera VC is d1 > d2, and the pixel value of the noise point p2 measured by the second measuring device 10-2 corresponding to the distance d2 was determined to be the pixel value of pixel P.

[0034] Figure 19 shows an example of prioritizing non-noise points over noise points in a pixel of interest using weights related to the measuring instrument. In Figure 19, weight w1 is assigned to the depth evaluation of the non-noise point p1, and weight w2 is assigned to the depth evaluation of the noise point p2. As a result, the distance from the virtual camera VC becomes w1d1 < w2d2, and the pixel value of the non-noise point p1' measured by the first measuring device 10-1 corresponding to the distance w1d1 is determined as the pixel value of pixel P.

[0035] To calculate the aforementioned weights, we will consider a method that uses the shooting directions of both the virtual camera and the measurement device. The measurement noise, such as the noise points mentioned above, is related to the angle between the optical axis center direction of the virtual camera and the optical axis center direction of the measurement device.

[0036] Figure 20 is a diagram illustrating the angle between the optical axis center direction of the virtual camera and the optical axis center direction of the measuring device. In Figure 20, four measuring devices (first measuring device 10-1, second measuring device 10-2, third measuring device 10-3, and fourth measuring device 10-4) are shown with shooting directions different from the shooting direction of the virtual camera VC relative to the subject OBJ.

[0037] The optical axis center direction of the first measuring device 10-1 is offset by an angle θ1 with respect to the optical axis center direction of the virtual camera VC. Similarly, the second measuring device 10-2 is offset by an angle θ2, the third measuring device 10-3 by an angle θ3, and the fourth measuring device 10-4 by an angle θ4. These relationships are θ1 < θ2 < θ3 < θ4.

[0038] For example, the first measuring device 10-1 photographs the subject OBJ from approximately the same direction as the virtual camera VC. On the other hand, the fourth measuring device 10-4 photographs the subject OBJ from a direction approximately opposite to the virtual camera VC. That is, the 3D data of the subject OBJ measured by the fourth measuring device 10-4 is often located in a position that cannot be measured from the virtual camera VC. For this reason, it is desirable to set a smaller weight the smaller the angle from the optical axis center direction of the virtual camera VC, and a larger weight the larger the angle. From these considerations, the weight w of the 3D data measured by the measuring device ｉis calculated by, for example, the following equation (1). Note that i is a number for identifying a measuring device (for example, a measuring device ID).

[0039]

[0040] In equation (1), α represents any positive real number, and vector r ｖｃ is a unit vector in the direction of the optical axis center of a virtual camera (this may also be referred to as a unit direction vector), and vector r ｃｉ is a unit direction vector in the direction of the optical axis center of the i-th measuring device, and (vector r ｖｃ ・vector r ｃｉ ) is vector r ｖｃ and vector r ｃｉ indicates the inner product of Specifically, vector r ｖｃ = (θ_x ｖｃ , θ_y ｖｃ , θ_z ｖｃ ) (where |vector r ｖｃ | = 1), and vector r ｃｉ = (θ_x ｃｉ , θ_y ｃｉ , θ_z ｃｉ ) (where |vector r ｃｉ | = 1). Note that the above "_" indicates that the following character is a subscript.

[0041] Finally, using the measured 3D data, a rendering procedure from the viewpoint of a virtual camera that takes the above weights into consideration will be described. First, let the 3D data measured by the i-th measuring device be V ｉ , and let the internal parameters of V ｉ be defined as K ｉ . Furthermore, among the extrinsic parameters for the i-th measuring device, let the rotation matrix be defined as R ｉ , and the translation vector be defined as vector t ｉ .[REND_END]]

[0042] V, which is 3D data, ｉ is, for example, RGBD image data having a vertical resolution of u and a horizontal resolution of v. The internal parameter K ｉ is V ｉThese are the parameters used when unfolding the data into a three-dimensional coordinate system based on each of the multiple measuring devices. Specifically, the internal parameter K ｉ This can be expressed, for example, by the following equation (2).

[0043]

[0044] In equation (2), f_x ｉ indicates the horizontal focal length, and f_y ｉ indicates the focal length in the vertical direction, and c_x ｉ indicates the position of the principal point, which is the optical center in the transverse direction of the image plane, and c_y ｉ This indicates the position of the principal point, which is the optical center in the vertical direction of the image plane.

[0045] Also, the rotation matrix R ｉ and translation vector t ｉ This parameter is used to transform from a three-dimensional coordinate system based on each of the multiple measuring devices to a reference coordinate system common to all of the multiple measuring devices. Rotation matrix R ｉ This is a 3x3 matrix, and the translation vector t ｉ It has three elements. These rotation matrices R ｉ and translation vector t ｉ When using V, the 3D data is ｉ Any point (u, v, r, g, b, d) on the above satisfies, for example, the following equation (3).

[0046]

[0047] Here, using equation (3), the point cloud data on the reference coordinates obtained by transforming all points of the 3D data Vi is Q ｉ = {q_i １ ,q_i ２ ,...,q_i ｍ} is defined as follows. Furthermore, the rotation matrix among the external parameters of the virtual camera is R ｖｃ , and the translation vector is vector t ｖｃ This is defined as: Rotation matrix R ｖｃ This is a 3x3 matrix, and the translation vector t ｖｃ It has three elements. These rotation matrices R ｖｃ and translation vector t ｖｃWhen using this method, the point cloud data Q on the reference coordinates ｉ Point cloud data Q' corresponding to the virtual camera coordinates (virtual camera coordinates) ｉ For example, the following equation (4) can be used to convert to .

[0048]

[0049] In equation (4), the rotation matrix R ｖｃ and translation vector t ｖｃ Q' is a value that changes depending on the position and orientation of the virtual camera. ｉ = {q'_i １ ,q'_i ２ ,...,q'_i ｍ} is a point group composed of m points, where each point is (X, Y, Z, r, g, b).

[0050] Furthermore, Q' ｉ Each point is q'_i ｋ = {q'_i ｋｘ ,q'_i ｋｙ ,...,q'_i ｋｚ The conversion from} to the coordinates (u, v) of the rendered 2D image at the viewpoint of the virtual camera can be done, for example, using equation (5) below.

[0051]

[0052] In equation (5), K ｖｃ Q' is a 3x3 matrix representing the intrinsic parameters of the virtual camera. Here, Q' is the 3D data measured by each of the multiple measuring devices. ｉ When this is converted to coordinates (u, v) on a 2D image observed by a virtual camera, the depth value d of one channel of each coordinate is usually q'_i ｋｚ The following applies: q'_i ｋｚ If the value is < 0, the corresponding point is removed. Furthermore, if there are multiple points with the same coordinate value in the coordinates (u', v') obtained by rounding the transformed coordinates (u, v) to integers, only the single point with a value greater than zero and the smallest value is selected.

[0053] Unlike conventional methods, this invention uses vector r, which is a unit vector in the direction of the optical axis center of the virtual camera, to reduce noise points. ｖｃ And, vector r, which is the unit vector in the direction of the optical axis center of the i-th measuring device. ｃｉ The weight calculated from lol ｉ Correction processing is performed using . Specifically, after selecting points based on depth values, the remaining points are weighted w ｉ Applying, w ｉ q'_i ｋｚ If there are multiple points with the same coordinate values ​​in the integer-converted coordinates (u', v'), then w ｉ q'_i ｋｚ The point with the smallest value will be selected as the point used for rendering.

[0054] Note that the position vector p is the position information of the virtual camera. ｖｃ = (x ｖｃ , y ｖｃ , z ｖｃ ), and the unit direction vector r, which is directional information. ｖｃ Using this, two external parameters in the virtual camera are R ｖｃ and vector t ｖｃ When updating, for example, the following equations (6) and (7) can be used.

[0055]

[0056]

[0057] The above describes the problems and solutions related to the specific noise contained in 3D data. The embodiments of the present invention will now be described.

[0058] <Embodiment> [Configuration] Figure 1 is a block diagram showing an example of a specific configuration of a 3D data processing system including a 3D data processing device according to an embodiment. The 3D data processing system in Figure 1 comprises a 3D data processing device 100, three measuring devices 200 (a first measuring device 200-1, a second measuring device 200-2, and a third measuring device 200-3), an operating device 300, and a display device 400. These devices are connected wirelessly, wired, or both. Multiple devices may also be combined.

[0059] The 3D data processing device 100 is, for example, an information processing device such as a computer. The 3D data processing device 100 is configured to generate display data provided to the user by processing multiple 3D data.

[0060] The measuring device 200 is, for example, a device that combines LiDAR and a camera. Specifically, the measuring device 200 is a ToF camera that acquires RGBD image data (which may be read as "3D data") of any resolution every 33 milliseconds. The measuring device 200 transmits the acquired 3D data to the 3D data processing device 100.

[0061] The operating device 300 is an input device that can change the direction and position of the virtual camera in response to user operations while viewing display data based on 3D data on the display device 400. The operating device 300 transmits user operation information regarding the virtual camera to the 3D data processing device 100 as virtual camera control information.

[0062] The display device 400 is a 2D monitor for providing display data to the user. The display device 400 receives display data from the 3D data processing device 100 and displays the received display data.

[0063] The configuration of the 3D data processing system according to the embodiment has been described above. Next, the hardware configuration of the 3D data processing device included in the 3D data processing system will be described.

[0064] Figure 2 is a block diagram showing an example of the hardware configuration of the 3D data processing device shown in Figure 1. The 3D data processing device 100 in Figure 2 includes a control circuit 101, storage 102, a communication module 103, an interface 104, and a drive 105.

[0065] The control circuit 101 is a circuit that controls all components of the 3D data processing device 100 as a whole. The control circuit 101 includes a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The ROM of the control circuit 101 stores programs used for various processes in the 3D data processing device 100. The CPU of the control circuit 101 controls the entire 3D data processing device 100 according to the programs stored in the ROM of the control circuit 101. The RAM of the control circuit 101 is used as a workspace for the CPU of the control circuit 101.

[0066] The storage 102 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. Information used for various processes in the 3D data processing device 100 is stored in the storage 102.

[0067] The communication module 103 is a circuit used for sending and receiving data between the three measuring devices 200, the operating device 300, and the display device 400.

[0068] Interface 104 is an interface used for communication between the user and the control circuit 101. Interface 104 includes, for example, input devices and output devices. Input devices include a voice microphone, a touch panel, and operation buttons. Output devices include a speaker and a display. If the 3D data processing device 100 is configured integrally with the display device 400, the display included in the output devices and the display device 400 may be the same.

[0069] Drive 105 is a device for reading software stored on the storage medium 106. Drive 105 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0070] The storage medium 106 is a medium for storing software by electrical, magnetic, optical, mechanical, or chemical means. The storage medium 106 may also store programs for executing various processes in the 3D data processing device 100.

[0071] The storage medium 106 may also be a USB (Universal Serial Bus) memory. If the storage medium 106 is a USB memory, it is connected to, for example, the communication module 103. In this case, the 3D data processing device 100 does not need to have a drive 105.

[0072] The hardware configuration of the 3D data processing unit has been described above. Next, the functional configuration of the 3D data processing unit will be described.

[0073] Figure 3 is a block diagram showing an example of the functional configuration of the 3D data processing device shown in Figure 2. The 3D data processing device 100 in Figure 3 includes a 3D data acquisition processing unit 110 (acquisition unit), a virtual camera position control unit 120 (position control unit), a virtual camera information storage unit 130, a 3D data rendering unit 140 (rendering unit), and a measurement device information storage unit 150.

[0074] To realize the functional configuration shown in Figure 3, the CPU of the control circuit 101 in Figure 2 loads the program stored in the ROM or storage medium 106 of the control circuit 101 into the RAM of the control circuit 101. The CPU of the control circuit 101 then interprets and executes the program loaded into the RAM of the control circuit 101. As a result, the 3D data processing device 100 functions as a computer comprising a 3D data acquisition processing unit 110, a virtual camera position control unit 120, a virtual camera information storage unit 130, a 3D data rendering unit 140, and a measurement device information storage unit 150.

[0075] The 3D data acquisition processing unit 110 acquires multiple 3D data from multiple measuring devices 200. The 3D data acquisition processing unit 110 associates the measuring device ID with the 3D data and outputs it to the 3D data rendering unit 140. The measuring device ID is, for example, a number that identifies each of the multiple measuring devices 200.

[0076] Specifically, the 3D data acquisition processing unit 110 receives the 3D data of measurement device ID "1" from the first measurement device 200-1. １ The second measuring device 200-2 obtains the 3D data of measuring device ID "2", V ２ The third measuring device 200-3 obtains the 3D data of measuring device ID "3" V ３ The 3D data acquisition processing unit 110 acquires the 3D data {(1, V １ ), (2, V ２ ), (3, V ３ The output is sent to the 3D data rendering unit 140.

[0077] The virtual camera position control unit 120 acquires virtual camera control information from the operating device 300 and virtual camera information from the virtual camera information storage unit 130. The virtual camera position control unit 120 updates the virtual camera information using the control information and calculates external parameters based on the updated virtual camera information. The virtual camera position control unit 120 outputs the updated virtual camera information to the virtual camera information storage unit 130, associates the updated virtual camera information with the calculated external parameters, and outputs it to the 3D data rendering unit 140.

[0078] The control information includes, for example, a 3x3 rotation matrix R and a translation vector t. The virtual camera information includes, for example, a position vector p. ｖｃ And the vector r, which is the direction information. ｖｃ This includes the following. The virtual camera information is stored in the virtual camera information storage unit 130 as a table, for example, as shown in Figure 4.

[0079] Figure 4 is a table showing an example of virtual camera information in the embodiment. Table 131 in Figure 5 contains the virtual camera position information "vector pｖｃ " and direction information "vector r ｖｃ " of the virtual camera are associated with each other.

[0080] Specifically, the virtual camera position control unit 120 uses the product of a rotation matrix R and vector p ｖｃ as new position information "vector p ｖｃ " for update, and uses the sum of a translation vector t and vector r ｖｃ as new direction information "vector r ｖｃ " for update. In addition, the virtual camera position control unit 120 calculates R, which is an external parameter, ｖｃ and vector t ｖｃ according to the aforementioned formula (6) and formula (7). The virtual camera position control unit 120 outputs update data composed of the updated position information and direction information of the virtual camera and the calculated external parameters (vector p ｖｃ , vector r ｖｃ , R ｖｃ , vector t ｖｃ ) to the 3D data rendering unit 140.

[0081] The virtual camera information storage unit 130 stores virtual camera information, for example, in the form of a table shown in Fig. 4. The virtual camera information storage unit 130 outputs and updates virtual camera information in accordance with instructions from the virtual camera position control unit 120.

[0082] The 3D data rendering unit 140 acquires output data (for example, paired 3D data in which a measurement device ID is paired with 3D data) from the 3D data acquisition processing unit 110, acquires virtual camera information (update data) from the virtual camera position control unit 120, and acquires measurement device information from the measurement device information storage unit 150. The 3D data rendering unit 140 calculates a weight for each of the plurality of measurement devices 200 based on the paired 3D data, the update data, and the measurement device information. The 3D data rendering unit 140 performs rendering processing using the calculated weights to generate display data. The 3D data rendering unit 140 outputs the generated display data to the display device 400.

[0083] The measurement device information includes, for example, internal parameters associated with the measurement device ID, a rotation matrix, a translation vector, position information, and direction information. The measurement device information is stored in the measurement device information storage unit 150 as a table, for example, as shown in Figure 5.

[0084] Figure 5 is a table showing an example of measurement device information in the embodiment. Table 151 in Figure 5 shows the internal parameter "K" for measurement device ID "1". １ ", rotation matrix "R １ ", translation vector "vector t １ ", position information "vector p ｃ１ ", and direction information "r ｃ１ The following is associated with it: Furthermore, for measurement device ID "2", the internal parameter "K ２ ", rotation matrix "R ２ ", translation vector "vector t ２ ", location information "vector p ｃ２ ", and direction information "r ｃ２ The following is associated with the measurement device ID "3": the internal parameter "K ３ ", rotation matrix "R ３ ", translation vector "vector t ３ ", position information "vector p ｃ３ ", and direction information "r ｃ３ The following is associated with it. Furthermore, the position information is a position vector p having three elements. ｃｉ = (x ｃｉ , y ｃｉ , z ｃｉ )

[0085] Specifically, the 3D data rendering unit 140 combines the paired 3D data and the measurement device information, for example, processing data {(1, V １ , K １ , R １ , vector t １ , vector p ｃ１ , vector r ｃ１ ), (2, V ２ , K ２ , R ２ , vector t ２ , vector p ｃ２ , vector r ｃ２), (3, V ３ , K ３ , R ３ , vector t ３ , vector p ｃ３ , vector r ｃ３ The subsequent processing is performed using the}}. The 3D data rendering unit 140 processes the updated data and processing data, using vector r ｃ１ and vector r ｖｃ Using this, the weight related to measurement device ID "1" w １ The 3D data rendering unit 140 calculates the vector r. ｃ２ and vector r ｖｃ Using this, the weight related to measurement device ID "2" w ２ Calculate the vector r ｃ３ and vector r ｖｃ Using this, the weight related to measurement device ID "3" w ３ The following is calculated. Furthermore, to calculate the weight, the aforementioned formula (1) is used, and α = 0.2 is applied.

[0086] After calculating the weights, the 3D data rendering unit 140 renders the 3D data of the measuring device, V, according to the aforementioned equation (3). ｉ point cloud data Q on reference coordinates ｉ Next, the 3D data rendering unit 140 converts the point cloud data Q on the reference coordinates according to the aforementioned equation (4). ｉ point cloud data Q' on virtual camera coordinates ｉ Finally, the 3D data rendering unit 140 converts the point cloud data Q' on the camera coordinates according to the aforementioned equation (5). ｉ This converts the coordinates (u, v) of the rendered 2D image from the viewpoint of the virtual camera.

[0087] After converting the 2D image to coordinates (u, v), the 3D data rendering unit 140 assigns Q' to the depth value of one channel of each coordinate. ｉ Each point is q'_i ｋ q'_i ｋｚ Apply q'_i ｋｚ If the value is <0, the corresponding point is removed. The 3D data rendering unit 140 then assigns a weight w to the remaining points. ｉ Applying, w ｉq'_i ｋｚ If there are multiple points with the same coordinate values ​​in the integer-converted coordinates (u', v'), then w ｉ q'_i ｋｚ The point with the smallest value is selected as the point to be used for rendering. Furthermore, the 3D data rendering unit 140 removes pixels that do not meet the criteria 0 ≤ u' ≤ X and 0 ≤ v' ≤ Y, when the vertical and horizontal resolutions of the display data are X and Y.

[0088] After performing the above series of processes, the 3D data rendering unit 140 converts the pixel data of the finally obtained coordinates (u', v') to the source point cloud data Q' ｉ The same 3-channel color data (r, g, b) as the data for each point (X, Y, Z, r, g, b) is set and output to the display device 400 as display data V'.

[0089] The measurement device information storage unit 150 stores the measurement device information, for example, as a table as shown in Figure 5. The measurement device information storage unit 150 outputs and updates the measurement device information according to the instructions of the 3D data rendering unit 140.

[0090] The configuration of the 3D data processing device and other components according to the embodiment has been described above. Next, the operation of the 3D data processing device according to the embodiment will be described using Figures 6 to 8.

[0091] [Operation] Figure 6 is a flowchart showing an example of 3D data acquisition processing in a 3D data processing device according to the embodiment. The 3D data acquisition processing shown in the flowchart of Figure 6 is performed by the 3D data acquisition processing unit 110.

[0092] (Step ST110) The 3D data acquisition processing unit 110 acquires multiple 3D data from multiple measuring devices.

[0093] (Step ST120) The 3D data acquisition processing unit 110 associates the measurement device ID with the 3D data and outputs it to the 3D data rendering unit 140. After step ST120, the 3D data acquisition process ends.

[0094] The 3D data acquisition process may be performed at any time interval. Furthermore, after step ST120, the 3D data acquisition process may return to step ST110 and repeat the series of processes.

[0095] Figure 7 is a flowchart showing an example of virtual camera position control processing in a 3D data processing device according to an embodiment. The virtual camera position control processing shown in the flowchart of Figure 7 is performed by the virtual camera position control unit 120.

[0096] (Step ST210) The virtual camera position control unit 120 acquires control information from the operating device 300.

[0097] (Step ST220) The virtual camera position control unit 120 acquires virtual camera information from the virtual camera information storage unit 130.

[0098] (Step ST230) The virtual camera position control unit 120 updates the virtual camera information using the control information.

[0099] (Step ST240) The virtual camera position control unit 120 calculates external parameters based on the updated virtual camera information.

[0100] (Step ST250) The virtual camera position control unit 120 stores the updated virtual camera information in the virtual camera information storage unit 130.

[0101] (Step ST260) The virtual camera position control unit 120 outputs the updated virtual camera information and external parameters to the 3D data rendering unit 140. After step ST260, the virtual camera position control process ends.

[0102] The virtual camera position control process may be performed at any time interval. Furthermore, after step ST260, the virtual camera position control process may return to step ST210 and repeat the series of processes.

[0103] Figure 8 is a flowchart showing an example of 3D data rendering processing in a 3D data processing device according to an embodiment. The 3D data rendering processing shown in the flowchart of Figure 8 is performed by the 3D data rendering unit 140.

[0104] (Step ST310) The 3D data rendering unit 140 acquires the output data from the 3D data acquisition processing unit.

[0105] (Step ST320) The 3D data rendering unit 140 acquires measurement device information from the measurement device information storage unit 150.

[0106] (Step ST330) The 3D data rendering unit 140 determines whether or not it has acquired output data from the virtual camera position control unit. If output data has been acquired, the process proceeds to step ST340. If output data has not been acquired, the process waits until output data is received. If previously acquired output data exists, the process may proceed to step ST340.

[0107] (Step ST340) The 3D data rendering unit 140 calculates weights related to the measurement device based on the output data (paired 3D data) from the 3D data acquisition processing unit, the measurement device information, and the output data (updated data) from the virtual camera position control unit.

[0108] (Step ST350) The 3D data rendering unit 140 performs rendering processing using the calculated weights and generates display data.

[0109] (Step ST360) The 3D data rendering unit 140 outputs the generated display data to the display device 400. After step ST360, the 3D data rendering process is completed.

[0110] The 3D data rendering process may be performed at any time interval. Furthermore, after step ST360, the 3D data rendering process may return to step ST310 and repeat the series of processes.

[0111] In summary, the 3D data processing device 100 acquires 3D data (e.g., RGBD image data) from each of the multiple measuring devices (e.g., the first measuring device 200-1, the second measuring device 200-2, and the third measuring device 200-3), and processes the data in a first direction (e.g., vector r) that indicates the direction of the optical axis center of the virtual camera. ｖｃ ) and a second direction indicating the optical axis center direction for each of the multiple measuring devices (for example, vector r ｃｉ Based on this, the weights for each of the multiple measuring devices w ｉ Calculate the 3D data and weights lol ｉ Based on this, rendering processing may be performed from the viewpoint of the virtual camera to generate display data.

[0112] Furthermore, if there are multiple points in the 3D data processing device 100 that can have the same coordinate values ​​on the display data, the device weights the depth value of the 3D data, which is the distance from the virtual camera. ｉ Points in the 3D data used for rendering may be determined based on weighted distances, by assigning a weight to the distance.

[0113] Alternatively, the 3D data processing device 100 may calculate the weights by taking the dot product of the first direction and the second direction.

[0114] [Effect] According to the above embodiment, the 3D data processing device acquires 3D data from each of the multiple measuring devices, calculates a weight for each of the multiple measuring devices based on a first direction indicating the optical axis center direction of the virtual camera and a second direction indicating the optical axis center direction for each of the multiple measuring devices, and performs rendering processing at the viewpoint of the virtual camera based on the 3D data and the weights to generate display data.

[0115] Therefore, since the 3D data processing device can change the rendering process based on weights, it can reduce the impact of noise in the 3D data and improve the user experience in the displayed data.

[0116] Furthermore, if there are multiple points in the 3D data that can have the same coordinate values ​​on the display data, the 3D data processing device may assign weights to the depth value of the 3D data, which is the distance from the virtual camera, and determine which points of the 3D data to be used for rendering based on the weighted distances.

[0117] This allows the 3D data processing device to generate display data using the correct data points that the user should visually perceive.

[0118] Alternatively, the 3D data processing device may calculate the weights by taking the dot product of the first direction and the second direction.

[0119] This allows the 3D data processing device to prioritize generating display data from 3D data of measuring devices that are oriented in the direction of the optical axis center, and that are close to the optical axis center direction of the virtual camera.

[0120] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple components disclosed. For example, if the problem can be solved and effects can be obtained even if some components are deleted from all the components shown in the embodiment, then the configuration with these components deleted can be extracted as an invention.

[0121] 100...3D data processing unit 101...Control circuit 102...Storage 103...Communication module 104...Interface 105...Drive 106...Storage medium 110...3D data acquisition processing unit 120...Virtual camera position control unit 130...Virtual camera information storage unit 131...Table 140...3D data rendering unit 150...Measurement device information storage unit 151...Table 200-1...First measurement device 200-2...Second measurement device 200-3...Third measurement device 300...Operation device 400...Display device 10...Measurement device 10-1...First measurement device 10-2...Second measurement device 10-3...Third measurement device 10-4...Fourth measurement device AR...Abnormal range BO...Background object C...Contact point CP...Measurement center point IMG1, IMG2, IMG3... Displayed image ML... Measurement line N1, N2, N3... Noise data NR... Normal range OBJ... Subject P... Pixel p1, p1', p2, p2'... Measurement point VC... Virtual camera

Claims

1. A 3D data processing device comprising: an acquisition unit that acquires 3D data from each of a plurality of measuring devices; and a rendering unit that calculates a weight for each of the plurality of measuring devices based on a first direction indicating the direction of the optical axis center of a virtual camera and a second direction indicating the direction of the optical axis center for each of the plurality of measuring devices, and performs rendering processing at the viewpoint of the virtual camera based on the 3D data and the weights to generate display data.

2. The 3D data processing apparatus according to claim 1, wherein, if there are multiple points of the 3D data that can have the same coordinate values ​​on the display data, the rendering unit assigns the weight to the depth value of the 3D data, which is the distance from the virtual camera, and determines the point of the 3D data to be used in the rendering process based on the weighted distance.

3. The rendering unit calculates the weight by taking the dot product of the first direction and the second direction, as described in claim 1.

4. A 3D data processing method comprising: acquiring 3D data from each of a plurality of measuring devices; calculating a weight for each of the plurality of measuring devices based on a first direction indicating the optical axis center direction of a virtual camera and a second direction indicating the optical axis center direction for each of the plurality of measuring devices; and performing rendering processing at the viewpoint of the virtual camera and generating display data based on the acquired plurality of 3D data and the calculated plurality of weights.

Images