3D data transmission device and 3D data transmission program

Abstract

A 3D data transmission device according to one embodiment of the present invention comprises a 3D data reception unit, a compression processing unit, and a 3D data transmission unit. The 3D data reception unit receives 3D data from a 3D data measurement device that measures the 3D data. The compression processing unit determines the degree of importance of the 3D data measurement device from the direction of the 3D data measurement device and the direction of a virtual camera, and performs compression processing on the received 3D data by changing the degree of compression according to the determined degree of importance. The 3D data transmission unit transmits the 3D data on which the compression processing has been performed.

Description

3D data transmission device and 3D data transmission program

[0001] This invention relates to a technology for reducing the amount of 3D (three-dimensional) data transmitted.

[0002] In the entertainment fields, such as sports and music concerts, technology that captures the entire space, including competitors and performers, as 3D data and live streams it is attracting attention. This technology will allow viewers to enjoy 3D content in real time through immersive display devices such as 3D displays and head-mounted displays.

[0003] 3D data is generally very large, requiring a high-speed, high-bandwidth network connection between the viewer and the 3D data measurement location. In 3D data transmission, reducing the amount of data transmitted is crucial. However, reducing the amount of data transmitted can degrade the quality of the image displayed to the viewer, potentially significantly impairing their experience.

[0004] Furthermore, in interactive content where viewers can send their reactions to the competitors or performers based on their experience of the live content, low latency is also required for 3D data transmission.

[0005] Non-patent document 1 discloses a Video-Based Point Cloud Compression (V-PCC) based method for compressing point cloud data. V-PCC is a method that achieves efficient compression while utilizing conventional video codecs by projecting the point cloud onto a 2D plane and representing it as a video frame. However, applying this method to real-time transmission presents challenges such as the need for high-speed encoders and decoders, and the need to limit the size and number of points in the point cloud to a certain extent.

[0006] D. Graziosi et al., “An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)”, APSIPA Transactions on Signal and Information Processing, Vol. 9, No. 1, e13, pp. 1-17, April 3, 2020.

[0007] The present invention aims to provide a technology that can reduce the amount of data transmitted without significantly impairing the viewer's experience.

[0008] A 3D data transmission device according to one aspect of the present invention comprises a 3D data receiving unit, a compression processing unit, and a 3D data transmission unit. The 3D data receiving unit receives 3D data from a 3D data measuring device that measures 3D data. The compression processing unit determines the importance of the 3D data measuring device based on the direction of the 3D data measuring device and the direction of a virtual camera, and performs compression processing on the received 3D data, changing the degree of compression according to the determined importance. The 3D data transmission unit transmits the 3D data that has undergone the compression processing.

[0009] According to the present invention, it is possible to provide a technology that can reduce the amount of data transmitted without significantly impairing the viewer's experience.

[0010] Figure 1 is a block diagram showing a 3D data reproduction system according to an embodiment. Figure 2 is a diagram showing a 3D data measurement device arranged around an object to be measured in 3D according to an embodiment. Figure 3 shows how distance image data measured by the 3D data measurement device shown in Figure 2 is reproduced as 3D data according to an embodiment. Figure 4 is a block diagram showing a 3D data reproduction system according to an embodiment. Figure 5 is a flowchart showing the processing flow of the 3D data receiving unit included in the 3D data transmission device shown in Figure 4. Figure 6 is a flowchart showing the processing flow of the virtual camera data receiving unit included in the 3D data transmission device shown in Figure 4. Figure 7 is a diagram showing the data stored in the storage unit included in the 3D data transmission device shown in Figure 4. Figure 8 is a diagram showing the data stored in the storage unit included in the 3D data transmission device shown in Figure 4. Figure 9 is a flowchart showing the processing flow of the 3D data conversion transmission unit included in the 3D data transmission device shown in Figure 4. Figure 10 is a diagram showing the data stored in the storage unit included in the 3D data reproduction device shown in Figure 4. Figure 11 is a flowchart showing the processing flow of the 3D data receiving unit included in the 3D data reproduction device shown in Figure 4. Figure 12 is a flowchart showing the processing flow of the virtual camera control unit included in the 3D data reproduction device shown in Figure 4. Figure 13 is a flowchart showing the processing flow of the 3D data reproduction unit included in the 3D data reproduction device shown in Figure 4. Figure 14 is a block diagram showing a 3D data reproduction system according to another embodiment. Figure 15 is a block diagram showing a 3D data reproduction system according to a further embodiment. Figure 16 is a block diagram showing the hardware configuration of a computer according to an embodiment.

[0011] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative of devices and methods for realizing the technical idea of ​​the invention and are not intended to limit the scope of the invention. The drawings are schematic or conceptual. In the following, components having similar functions or configurations may be given the same reference numerals. Subscripts may be added to reference numerals to distinguish between components having similar functions or configurations. Subscripts are added after the reference numeral, separated by a hyphen "-".

[0012] Figure 1 schematically shows a 3D (three-dimensional) data reproduction system 10 according to one embodiment. As shown in Figure 1, the 3D data reproduction system 10 comprises a plurality of 3D data measuring devices 11, a 3D data transmission device 12, a 3D data reproduction device 13, a virtual camera operation device 14, and a monitoring device 15. In the example shown in Figure 1, four 3D data measuring devices 11-1 to 11-4 are provided as the 3D data measuring devices 11.

[0013] The 3D data measurement device 11 is a device that measures distance image data as 3D data. For example, the 3D data measurement device 11 generates distance image data by measuring the space containing the subject. Distance image data is data in which each pixel has a value indicating depth (distance). Distance image data may be D image data in which each pixel has a depth value, or RGBD image data in which each pixel has both a depth value and an RGB value. The 3D data reproduction system 10 obtains distance image data from multiple viewpoints using the 3D data measurement device 11. Hereinafter, image data may simply be referred to as an image. Examples of the 3D data measurement device 11 include ToF (time-of-flight) cameras, stereo cameras, and combinations of LiDAR (light detection and ranging) and cameras. The 3D data measurement device 11 is sometimes called a distance image sensor.

[0014] The 3D data transmission device 12 receives distance images (D images or RGBD images) from each 3D data measuring device 11. Furthermore, the 3D data transmission device 12 receives virtual camera data from the 3D data reproduction device 13, indicating the position and direction of the virtual camera. The virtual camera represents a virtual viewpoint of the image or video displayed on the monitor device 15. The direction of the virtual camera indicates the direction from the virtual camera toward the area that the virtual camera is observing. In order to reduce the amount of data transmitted, the 3D data transmission device 12 performs compression processing on the distance images based on the virtual camera data and transmits the resulting distance images to the 3D data reproduction device 13.

[0015] The 3D data reproduction device 13 receives operation signals from the virtual camera operation device 14 to control the position and direction of the virtual camera, and changes the position and direction of the virtual camera according to the received operation signals. The 3D data reproduction device 13 receives a depth image from the 3D data transmission device 12, generates an image corresponding to the position and direction of the virtual camera from the depth image, and outputs it to the monitor device 15. Furthermore, the 3D data reproduction device 13 transmits the virtual camera data to the 3D data transmission device 12.

[0016] The virtual camera control device 14 is an input device, such as a mouse, keyboard, or game controller, that allows the user (e.g., a viewer) to control the position and direction of the virtual camera. The monitor device 15 is an image display device, such as a 2D monitor, a 3D monitor, or a head-mounted display.

[0017] Figure 2 shows how 3D data measurement devices 11-1 to 11-4 are arranged around an object to be measured in 3D, and Figure 3 shows how the distance image data measured by the 3D data measurement devices 11-1 to 11-4 shown in Figure 2 is reproduced as 3D data representing the three-dimensional shape of the object in the 3D data reproduction device 13. As shown in Figure 3, in the 3D data reproduction system 10, the positional relationship between the object in the real world and the 3D data measurement devices 11-1 to 11-4 is reconstructed, and a virtual viewpoint is placed at any location within this reconstruction. The image of the object observed from that virtual viewpoint is displayed on the monitor device 15, allowing viewers to experience 3D content.

[0018] In the 3D data reproduction system 10, depending on the position of the virtual viewpoint, among the multiple 3D data measurement devices 11, there are 3D data measurement devices 11 with high importance and 3D data measurement devices 11 with low importance. For example, consider the case where the position and orientation of the virtual viewpoint are set as shown in Figure 3. In the state shown in Figure 3, 3D data measurement device 11-1 is located near the virtual viewpoint and is facing substantially the same direction as the virtual viewpoint. On the other hand, 3D data measurement device 11-3 is located away from the virtual viewpoint and is facing substantially the opposite direction. Most of the distance image data obtained by 3D data measurement device 11-3 does not affect the appearance of the subject from the virtual viewpoint, so the presence or absence of 3D data measurement device 11-3 has less impact on the viewer's perception compared to 3D data measurement device 11-1. Importance represents the degree of influence that the distance images obtained by 3D data measurement devices 11 have on the appearance of the subject from the virtual viewpoint.

[0019] Therefore, in this embodiment, the importance of each 3D data measuring device 11 is determined based on the relationship between the position and orientation of the 3D data measuring device 11 and the position and orientation of the virtual viewpoint, and compression processing is performed according to that importance, thereby reducing the amount of data transmitted without significantly impairing the viewer's experience. Alternatively, the importance of each 3D data measuring device 11 may be determined based on the relationship between the orientation of the 3D data measuring device 11 and the orientation of the virtual viewpoint, regardless of the relationship between the position of the 3D data measuring device 11 and the position of the virtual viewpoint.

[0020] Referring again to Figure 1, the 3D data transmission device 12 comprises a 3D data receiving unit 121, a virtual camera data receiving unit 122, and a 3D data conversion transmission unit 125. The 3D data conversion transmission unit 125 comprises a compression processing unit 123 and a 3D data transmission unit 124.

[0021] The 3D data receiving unit 121 receives distance images from each 3D data measuring device 11 and sends the received distance images to the compression processing unit 123.

[0022] The virtual camera data receiving unit 122 receives virtual camera data indicating the position and direction of the virtual camera from the 3D data reproduction device 13, and sends the received virtual camera data to the compression processing unit 123. The virtual camera data includes position information indicating the position of the virtual camera and direction information indicating the direction of the virtual camera. In one example, the virtual camera data includes the following position vector p as position information. ｃ It includes the following unit direction vector r as directional information. ｃ Includes.

[0023] The compression processing unit 123 calculates the importance of each 3D data measuring device 11 based on the virtual camera data received from the virtual camera data receiving unit 122, and performs compression processing on the distance image obtained by each 3D data measuring device 11 according to the calculated importance of that device. For example, the compression processing unit 123 calculates the importance of 3D data measuring device 11-i and performs compression processing on the distance image obtained by 3D data measuring device 11-i according to the calculated importance of 3D data measuring device 11-i.

[0024] The importance is calculated using virtual camera data and 3D data measurement device information indicating the position and orientation of the 3D data measurement device 11. The 3D data measurement device information includes position information indicating the position of each 3D data measurement device 11 and orientation information indicating the orientation of each 3D data measurement device 11. For example, the position information of 3D data measurement device 11-i is the position vector p shown below. ｃｉ The directional information of the 3D data measuring device 11-i is obtained as shown below, and is represented by the unit direction vector r ｃｉ It is obtained as follows.

[0025] The method for calculating importance includes a first calculation method that uses directional information and a second calculation method that uses both positional information and directional information. When the first calculation method is adopted, the virtual camera data does not need to include the positional information of the virtual camera, and the 3D data measurement device information does not need to include the positional information of each 3D data measurement device 11.

[0026] In the first calculation method, the compression processing unit 123 sets the unit direction vector r of the virtual camera as shown in the following formula (1) ｃ and the unit direction vector r of the 3D data measurement device 11-i ｃｉ to calculate the importance I of the 3D data measurement device 11-i ｉ using the inner product of the vectors. According to formula (1), the closer the direction of the 3D data measurement device 11-i is to the direction of the virtual camera, the higher the importance becomes.

[0027] In the second calculation method, the compression processing unit 123 sets the unit direction vector r of the virtual camera as shown in the following formula (2) ｃ and the unit direction vector r of the 3D data measurement device 11-i ｃｉ to calculate the importance I of the 3D data measurement device 11-i ｉ using the distance between the virtual camera and the 3D data measurement device 11-i in addition to the inner product of the vectors. Here, α and β are parameters that determine whether to attach more importance to direction data or position data, and can take any positive value. According to formula (2), the closer the direction of the 3D data measurement device 11-i is to the direction of the virtual camera, the higher the importance becomes; and the closer the position of the 3D data measurement device 11-i is to the position of the virtual camera, the higher the importance becomes.

[0028] As a method for compressing a depth image using importance, there is a method of downsampling the depth image. For example, when the original data is a depth image with a resolution of 2000×1000, downsampling is performed according to the importance as shown in Table 1 below. In this case, the compression processing unit 123 changes the degree of compression by changing the resolution according to the importance. Reducing the data amount through downsampling generally results in loss of detail and lowers the perceived quality. In the present embodiment, by performing downsampling targeting 3D data measurement devices 11 with low importance, the amount of data transmission can be reduced without significantly impairing the viewing experience of the viewer. In Table 1, T １ and T ２ are numbers satisfying T １ > T ２ and are importance thresholds used to determine resolution levels. I ｉ > T１ In this case, the distance image obtained by the 3D data measurement device 11 is not downsampled. ２ <I ｉ ≦T １ In this case, the distance image obtained by the 3D data measurement device 11-i is downsampled to a distance image with a resolution of 1000 x 500. ｉ ≦T ２ In this case, the distance image obtained by the 3D data measurement device 11-i is downsampled to a distance image with a resolution of 500 x 250.

[0029]

[0030] Table 1 shows an example of downsampling in three stages. Compression may also be performed by downsampling in two or more stages.

[0031] The 3D data transmission unit 124 transmits 3D data, including the compressed depth image, to the 3D data reproduction device 13. The 3D data transmitted from the 3D data transmission device 12 to the 3D data reproduction device 13 includes, in addition to the compressed depth image, internal and external parameters of each 3D data measuring instrument 11 corresponding to the resolution level that specifies the resolution of the compressed depth image. The internal parameter of the 3D data measuring instrument 11-i corresponding to the resolution level j is K. ｉｊ Therefore, the internal parameter K ｉｊ The horizontal and vertical focal lengths f ｘｉｊ , f ｙｉｊ Furthermore, the principal point c, which is the optical center in the image plane. ｘｉｊ , c ｙｉｊ This results in a 3x3 matrix composed of the elements.

[0032] Furthermore, the external parameters of the 3D data measurement device 11-i corresponding to resolution level j are a 3x3 rotation matrix R. ｉｊ and a translation vector t with three elements ｉｊ Therefore, R ｉｊ and t ｉｊThe following equation satisfies the requirement that any point (u, v, d) or (u, v, r, g, b, d) on a D image or RGBD image V with a resolution of u × v can be transformed into reference coordinates (X, Y, Z) by the following equation (4).

[0033] When the D image is converted to point cloud data using equation (4), each point becomes a colorless point data (X, Y, Z) with only position information. When the RGBD image is converted to point cloud data, each point becomes a colored point data (X, Y, Z, r, g, b) with both position information and RGB information. The reference coordinates are the same coordinate system as the position and direction information of the virtual camera and the position and direction information of the 3D data measurement device 11.

[0034] If V is the distance image after compression processing, then the 3D data transmitted from the 3D data transmission device 12 to the 3D data reproduction device 13 is (V, K ｉｊ , R ｉｊ ,t ｉｊ )

[0035] The 3D data reproduction device 13 comprises a 3D data receiving unit 131, a virtual camera control unit 132, and a 3D data reproduction unit 133.

[0036] The 3D data receiving unit 131 receives 3D data (V, K) from the 3D data transmitting device 12. ｉｊ , R ｉｊ ,t ｉｊ The system receives the data and sends the received 3D data to the 3D data reproduction unit 133.

[0037] The 3D data reproduction unit 133 generates display image data from the 3D data and virtual camera data received from the 3D data transmission device 12, and outputs the image data to the monitor device 15. For example, if the monitor device 15 is a 2D monitor, the 3D data reproduction unit 133 uses equation (4) to convert the distance image V into a point cloud Q = {q} on the reference coordinate system. １ ,q ２ ...q ｍ Convert to}. Furthermore, the 3D data reproduction unit 133 converts the 3x3 rotation matrix R, which is an external parameter of the virtual camera, as shown in the following formula. ｃ and a translation vector t with three elements ｃThis allows us to obtain Q', which is the point cloud Q transformed from reference coordinates to virtual camera coordinates.

[0038] Rotation matrix R ｃ and translation vector t ｃ This is updated by the virtual camera control unit 132 based on the operation signals generated in the virtual camera operation device 14, as will be described later. Q' = {q' １ ,q' ２ ...q' ｍ} is a point group consisting of m points, where each point is (X, Y, Z) or (X, Y, Z, r, g, b).

[0039] Furthermore, the 3D data reproduction unit 133 calculates the point cloud Q' = {q'} １ ,q' ２ ...q' ｍ Each point q' of} ｋ = (q' ｋｘ ,q' ｋｙ ,q' ｋｚ Apply the following formula (5) to the given values ​​to convert them into coordinates (u, v) of a two-dimensional image. K ｃ This is a 3x3 matrix, similar to equation (3) above, and represents the intrinsic parameters of the virtual camera. When converted to the coordinates (u, v) of a 2D image, the depth value of one channel of each coordinate has q'. ｋｚ Apply the following, however q' ｋｚ When the value is < 0, the corresponding point is removed, and when (u, v) is rounded to an integer (u', v'), if there are multiple points with the same coordinate values ​​(u', v'), only the point with the smallest value that is greater than 0 is selected. The pixel data of the coordinate values ​​(u', v') obtained through the above process will have a depth value q' when converted from (X, Y, Z) which does not have RGB. ｋｘThis is inserted. On the other hand, when converting from (X, Y, Z, r, g, b) which has RGB, the pixel data of coordinate value (u', v') is set to 3ch (3-channel) (r, g, b). Also, when the vertical and horizontal resolutions of the D image or 3ch RGB image to be converted are X and Y, pixels that do not fall within the values ​​0 ≤ u' ≤ X and 0 ≤ v' ≤ Y are removed. This process is performed on all points of Q', and the D image or 3ch RGB image obtained by this process is called V'. Image V' is output from the 3D data reproduction device 13 to the monitor device 15 and displayed on the monitor device 15.

[0040] The virtual camera control unit 132 receives an operation signal from the virtual camera operating device 14. Based on the received operation signal, the virtual camera control unit 132 receives the virtual camera position information p ｃ , direction information r ｃ , external parameter R ｃ ,t ｃ Update the position information p based on the operation signal. ｃ and direction information r ｃ When you update R ｃ The updated p is calculated using the following formula (6). ｃ and r ｃ It can be calculated from this.

[0041] Furthermore, t ｃ The updated R is calculated using the following formula (7). ｃ It can be calculated from this.

[0042] The virtual camera control unit 132 receives position information p ｃ and direction information r ｃ The data is sent to the 3D data transmission device 12, and the external parameter R ｃ ,t ｃ The data is sent to the 3D data reproduction unit 133.

[0043] It is also possible to individually compress the distance image V included in the 3D data transmitted by the 3D data transmission device 12 and received by the 3D data reproduction device 13 as image data such as jpeg or png, similar to V-PCC. By using the method according to the embodiment in combination with V-PCC, it is possible to reduce the amount of data transmitted compared to using V-PCC alone. Furthermore, although an example in which one 3D data transmission device 12 is provided for multiple 3D data measurement devices 11 has been described, as shown in Figure 14, it is also possible to distribute the processing by providing one 3D data transmission device 12 for each 3D data measurement device 11. In addition, as shown in Figure 15, it is also possible to provide two 3D data transmission devices 12 for four 3D data measurement devices 11, such as assigning one 3D data transmission device 12 to two 3D data measurement devices 11.

[0044] The following describes a specific example of implementing the 3D data reproduction system 10 according to the embodiment.

[0045] Figure 4 schematically shows an embodiment of the 3D data reproduction system 10 according to the embodiment. In Figure 4, the same reference numerals are used for components similar to those shown in Figure 1, and redundant explanations are omitted as appropriate. As shown in Figure 4, the 3D data reproduction system 10 comprises three 3D data measuring devices 11 (3D data measuring devices 11-1, 11-2, and 11-3), a 3D data transmission device 12, a 3D data reproduction device 13, a virtual camera operation device 14, and a monitor device 15.

[0046] The 3D data measurement device 11 is a ToF camera that transmits RGBD image data with a resolution of 2000 x 1000 to the 3D data transmission device 12 every 33 msec. Each 3D data measurement device 11 is assigned a measurement device ID as identification information. In the example described here, the measurement device ID of 3D data measurement device 11-1 is 1, the measurement device ID of 3D data measurement device 11-2 is 2, and the measurement device ID of 3D data measurement device 11-3 is 3. The virtual camera operation device 14 is an input device that can change the direction and position of the virtual camera according to the operation of the user viewing the 3D data. The monitor device 15 is a 2D monitor that displays the RGBD image data output from the 3D data reproduction device 13.

[0047] The 3D data transmission device 12 includes a 3D data receiving unit 121, a virtual camera data receiving unit 122, a 3D data conversion transmission unit 125, a storage unit 126, and a storage unit 127. The 3D data conversion transmission unit 125 includes a compression processing unit 123 and a 3D data transmission unit 124.

[0048] Figure 5 schematically shows the processing flow of the 3D data receiving unit 121. The 3D data receiving unit 121 repeatedly executes the series of processes shown in Figure 5. For example, the series of processes shown in Figure 5 is executed with a period of 33 msec.

[0049] In step S51, the 3D data receiving unit 121 receives RGBD image data V from the 3D data measuring devices 11-1, 11-2, and 11-3. １ , V ２ , V ３ Receives RGBD image data V １ This represents RGBD image data received from 3D data measurement device 11-1 with ID 1, and RGBD image data V ２ This represents RGBD image data received from 3D data measurement device 11-2 with ID 2, and RGBD image data V ３ This represents RGBD image data received from the 3D data measurement device 11-3, whose ID is 3.

[0050] In step S52, the 3D data receiving unit 121 receives the instrument ID and the pair of RGBD image data received in step S51 {(1, V１ ), (2, V ２ ), (3, V ３ The output is sent to the compression processing unit 123. This completes the processing flow.

[0051] Figure 6 schematically shows the processing flow of the virtual camera data receiving unit 122. The virtual camera data receiving unit 122 repeatedly executes the series of processing flows shown in Figure 6.

[0052] In step S61, the virtual camera data receiving unit 122 receives a combination of a position vector representing the position of the virtual camera and a unit direction vector representing the direction of the virtual camera (p ｃ ,r ｃ The virtual camera data is received as virtual camera data. The virtual camera data receiving unit 122 waits until it receives virtual camera data, and once the virtual camera data receiving unit 122 receives virtual camera data, the processing flow proceeds to step S62.

[0053] In step S62, the virtual camera data receiving unit 122 receives the virtual camera data (p ｃ ,r ｃ The output is sent to the compression processing unit 123. This completes the processing flow.

[0054] When the processing flow is completed, the virtual camera data receiving unit 122 resumes the processing flow. The virtual camera data receiving unit 122 waits until the next virtual camera data (update data for virtual camera data) is transmitted from the 3D data reproduction device 13.

[0055] Figure 7 shows an example of data stored in the storage unit 126. As shown in Figure 7, the data stored in the storage unit 126 includes seven elements: instrument ID, resolution level, resolution of the X coordinate after downsampling, resolution of the Y coordinate after downsampling, intrinsic parameters, rotation matrix, and translation vector. The instrument ID takes one integer from the set of {1, 2, 3}. The resolution level takes one integer from the set of {1, 2, 3}. The resolution of the X coordinate after downsampling takes a positive integer value, the resolution of the Y coordinate after downsampling also takes a positive integer value, and twice the resolution of the Y coordinate after downsampling is equal to the resolution of the X coordinate after downsampling. The intrinsic parameters and rotation matrix are 3x3 matrix data. The translation vector is vector data consisting of three elements. The data stored in the storage unit 126 is used for compression processing and is prepared in advance.

[0056] Figure 8 shows an example of data stored in the memory unit 127. As shown in Figure 8, the data stored in the memory unit 127 includes three elements: a measuring instrument ID, position information of the 3D data measuring device 11, and direction information of the 3D data measuring device 11. The measuring instrument ID is one of the integers {1, 2, 3}. The position information of the 3D data measuring device 11 is a position vector p having the following three elements. ｃｉ The directional information of the 3D data measuring device 11 is a unit direction vector r having the following three elements. ｃｉ The data stored in the memory unit 127 is used for calculating importance and is prepared in advance.

[0057] Figure 9 schematically shows the processing flow of the 3D data conversion and transmission unit 125, which includes a compression processing unit 123 and a 3D data transmission unit 124. The compression processing unit 123 repeatedly executes the series of processes shown in Figure 9. For example, the series of processes shown in Figure 9 is executed with a period of 33 msec.

[0058] In step S91, the compression processing unit 123 receives a pair of measuring instrument ID and RGBD image data {(1, V １), (2, V ２ ), (3, V ３ )} is received.

[0059] In step S92, the compression processing unit 123 receives virtual camera data (p ｃ , r ｃ ) from the virtual camera data receiving unit 122. If the virtual camera data receiving unit 122 newly receives virtual camera data from the 3D data reproducing apparatus 13 and transmits the same to the compression processing unit 123, the compression processing unit 123 receives the virtual camera data from the virtual camera data receiving unit 122. If the virtual camera data receiving unit 122 has not received new virtual camera data from the 3D data reproducing apparatus 13 and previously received virtual camera data exists, the compression processing unit 123 skips step S92 and the next step S93, and advances the process to step S94. If the compression processing unit 123 has never received virtual camera data, the compression processing unit 123 waits until it receives the virtual camera data (p ｃ , r ｃ ) from the virtual camera data receiving unit 122.

[0060] In step S93, the compression processing unit 123 acquires position information p of each 3D data measuring device 11 from the storage unit 127 ｃｉ and direction information r ｃｉ read out, and based on, for example, the above formula (2), the position information p of the 3D data measuring device 11 ｃｉ and direction information r ｃｉ and the virtual camera data (p ｃ , r ｃ ) to calculate the importance I of the 3D data measuring devices 11-1, 11-2, and 11-3 １ , I ２ , I ３ are calculated. For example, the importance I of the 3D data measuring device 11-1 whose measuring instrument ID is 1 １ can be obtained by calculating the following. In this embodiment, α=0.9 and β=0.01. Importance I １ , I ２ , I ３ After the calculation of is completed, the processing flow proceeds to step S94.

[0061] In step S94, the compression processing unit 123 reads internal and external parameters corresponding to the 3D data measurement device 11 from the storage unit 127 according to their importance. Specifically, first, the compression processing unit 123 determines the resolution level according to the importance of each 3D data measurement device 11. For example, the compression processing unit 123 determines I ｉ >0.8 means resolution level 1, 0.0 < I ｉ Resolution level 2 when ≤ 0.8, 0.0 ≥ I ｉ At this time, the resolution level is determined to be 3. Next, the compression processing unit 123 obtains from the storage unit 127, for each 3D data measurement device 11, the resolution of the X coordinate after downsampling X', the resolution of the Y coordinate after downsampling Y', the intrinsic parameters, the rotation matrix, and the translation vector associated with the combination of measurement device ID and resolution level. For example, when the resolution level of 3D data measurement device 11-1, whose measurement device ID is 1, is determined to be 2, (1, V １ ) for (X' = 1000, Y' = 500, K １２ , R １２ ,t １２ ) is obtained. When the data corresponding to each 3D data measurement device 11 has been read out, the processing flow proceeds to step S95.

[0062] In step S95, the compression processing unit 123 compresses the RGBD image V based on the resolution X' of the downsampled X coordinate and the resolution Y' of the downsampled Y coordinate, which correspond to each 3D data measurement device 11, obtained in step S94. １ , V ２ , V ３ The RGBD image is downsampled. However, the resolution of the RGBD image of the 3D data measurement device 11, which has been determined to have a resolution level of 1, is maintained. In other words, the RGBD image of the 3D data measurement device 11, which has been determined to have a resolution level of 1, is not downsampled. As an algorithm for downsampling the image, for example, nearest neighbor interpolation, which simply decimates pixels, or bilinear interpolation or bicubic interpolation, which interpolates the compressed pixels from surrounding pixels, can be used. The RGBD image after downsampling is V' １ , V'２ , V' ３ Let's assume that.

[0063] In step S96, the 3D data transmission unit 124 transmits the downsampled RGBD image V' obtained in step S95. １ , V' ２ , V' ３ And the measuring instrument ID obtained in step S94 ｉ , internal parameter K ｉｊ , and also, the external parameter R ｉｊ and t ｉｊ 3D data that includes and relating to each other { (V' １ , K １ｊ , R １ｊ ,t １ｊ ), (V' ２ , K ２ｊ , R ２ｊ ,t ２ｊ ), (V' ３ , K ３ｊ , R ３ｊ ,t ３ｊ The data is sent to the 3D data reproduction device 13. This completes the processing flow shown in Figure 9.

[0064] Referring again to Figure 4, the 3D data reproduction device 13 comprises a 3D data receiving unit 131, a virtual camera control unit 132, a 3D data reproduction unit 133, and a storage unit 134.

[0065] Figure 10 shows an example of data stored in the memory unit 134. As shown in Figure 10, the memory unit 134 stores a position vector p as position information of the virtual camera. ｃ and the unit direction vector r as directional information of the virtual camera ｃ Store the position vector p. ｃ and unit direction vector r ｃ This is updated by the virtual camera control unit 132.

[0066] Figure 11 schematically shows the processing flow of the 3D data receiving unit 131. The 3D data receiving unit 131 repeatedly executes the series of processes shown in Figure 11. For example, the series of processes shown in Figure 11 is executed with a period of 33 msec.

[0067] In step S111, the 3D data receiving unit 131 receives 3D data { (V' １ , K １ｊ , R １ｊ ,t １ｊ ), (V' ２ , K ２ｊ , R ２ｊ ,t ２ｊ ), (V' ３ , K ３ｊ , R ３ｊ ,t ３ｊ} is received.

[0068] In step S112, the 3D data receiving unit 131 sends the 3D data received in step S111 to the 3D data reproduction unit 133.

[0069] Figure 12 schematically shows the processing flow of the virtual camera control unit 132. The virtual camera control unit 132 repeatedly executes the series of processes shown in Figure 12.

[0070] In step S121, the virtual camera control unit 132 receives operation signals from the virtual camera operating device 14 to control the position and direction of the virtual camera. For example, the virtual camera operating device 14 generates a 3x3 rotation matrix R and a translation vector t in response to user operation and transmits the rotation matrix R and translation vector t as operation signals to the virtual camera control unit 132.

[0071] In step S122, the virtual camera control unit 132 receives the position vector p from the storage unit 134. ｃ and direction vector r ｃ Read out the position vector p read from memory unit 134. ｃ and direction vector r ｃ This represents the current position and direction of the virtual camera.

[0072] In step S123, the virtual camera control unit 132 updates the position and direction information of the virtual camera based on the rotation matrix R and translation vector t obtained in step S121. Specifically, the virtual camera control unit 132 updates Rp ｃ The new position vector p ｃ The location information is updated as r ｃ +t is a new unit direction vector rｃ Update the direction information.

[0073] In step S124, the virtual camera control unit 132, for example, according to equations (6) and (7) above, determines the new position vector p obtained in step S123. ｃ and unit direction vector r ｃ External parameter R of the virtual camera ｃ ,t ｃ Calculate.

[0074] In step S125, the virtual camera control unit 132 outputs the position vector p obtained in step S123. ｃ and unit direction vector r ｃ Then, the data in the memory unit 134 is updated.

[0075] In step S126, the virtual camera control unit 132 transmits the virtual camera position information and direction information obtained in step S123 to the 3D data transmission device.

[0076] In step S127, the virtual camera control unit 132 uses the external parameter R obtained in step S124. ｃ ,t ｃ The data is sent to the 3D data reproduction unit 133. This completes the processing flow shown in Figure 12.

[0077] Figure 13 schematically shows the processing flow of the 3D data reproduction unit 133. The 3D data reproduction unit 133 repeatedly executes the series of processes shown in Figure 13. For example, the series of processes shown in Figure 13 is executed with a period of 33 msec.

[0078] In step S131, the 3D data reproduction unit 133 receives 3D data { (V' １ , K １ｊ , R １ｊ ,t １ｊ ), (V' ２ , K ２ｊ , R ２ｊ ,t ２ｊ ), (V' ３ , K ３ｊ , R ３ｊ ,t ３ｊ} is received.

[0079] In step S132, the 3D data reproduction unit 133 receives the updated external parameter R from the virtual camera control unit 132. ｃ ,t ｃ The 3D data reproduction unit 133 receives the updated external parameter R. ｃ ,t ｃ If no external parameters have been sent and there are external parameters previously received from the 3D data reproduction unit 133, the 3D data reproduction unit 133 skips step S132 and the next step S133 and proceeds to step S134. If the 3D data reproduction unit 133 has never received external parameters from the virtual camera control unit 132, the 3D data reproduction unit 133 waits until it receives external parameters from the virtual camera control unit 132.

[0080] In step S133, the 3D data reproduction unit 133 processes the data (V' obtained in step S131) １ , K １ｊ , R １ｊ ,t １ｊ ), (V' ２ , K ２ｊ , R ２ｊ ,t ２ｊ ), (V' ３ , K ３ｊ , R ３ｊ ,t ３ｊ ) 3D point cloud data Q = {q １ ,q ２ ...q ｍ Convert to}. Use the above formula (4) for the conversion. For example, RGBD image V １ The position of any pixel (u, v, r, g, b, d) on the 3D point cloud can be determined as follows:

[0081] RGBD image V' １ , V' ２ , V' ３ Each of these is transformed into a three-dimensional point on the reference coordinate system, and the resulting point cloud Q is represented as a set of points. Next, the 3D data reproduction unit 133 transforms the point cloud Q from the reference coordinate system to virtual camera coordinates using the following formula to obtain the point cloud Q'.

[0082] Here, Q' = {q'}１ ,q' ２ ...q' ｍ} is a point cloud composed of m points, and each point has information about (X, Y, Z, r, g, b). Furthermore, the 3D data reproduction unit 133 calculates the point cloud Q' = {q' １ ,q' ２ ...q' ｍ Each point q' of} ｍ = (q' ｋｘ ,q' ｋｙ ,q' ｋｚ Apply the above equation (5) to ) to convert it into coordinates (u, v) of a two-dimensional image. In equation (5), K ｃ This is a 3x3 matrix that represents the intrinsic parameters of the virtual camera. In this example, K ｃ This is assumed to be data held by the 3D data reproduction unit 133 as an invariant fixed value. When converted to the coordinates of a 2D image (u, v), the depth value of one channel of each coordinate is q'. ｋｚ Apply the following, however q' ｋｚ If the value is < 0, the point in question is removed, and when (u, v) is rounded to an integer (u', v'), if there are multiple points with the same coordinate value (u', v'), only the point with the smallest value that is greater than 0 is selected. Also, when the vertical and horizontal resolutions of the image to be converted are X and Y, pixels that do not satisfy 0 ≤ u' ≤ X and 0 ≤ v' ≤ Y are removed. The pixel data of the coordinate value (u', v') obtained through this process is set to the same 3 channels (r, g, b) as the source (X, Y, Z, r, g, b). The resolutions X and Y are determined by the resolution of the monitor device 15, and in this embodiment, X = 2000 and Y = 1000. The RGB image obtained through this process is denoted as V'.

[0083] In step S134, the 3D data reproduction unit 133 outputs the image V' obtained in step S133 to the monitoring device 15. The monitoring device 15 displays the image V'. This completes the processing flow shown in Figure 13.

[0084] Figure 16 schematically shows the hardware configuration of a computer 20 according to an embodiment. The computer 20 shown in Figure 16 is an example of a computer capable of realizing a 3D data transmission device 12.

[0085] As shown in Figure 16, the computer 20 includes a CPU (central processing unit) 21 as a processing circuit, RAM (random access memory) 22, a storage device 23, an input / output interface 24, and a communication interface 25. The CPU 21 is communicated to the RAM 22, the storage device 23, the input / output interface 24, and the communication interface 25.

[0086] The CPU 21 is an example of a general-purpose processor capable of executing programs. A GPU (graphics processing unit) may also be used as the general-purpose processor. The RAM 22 is volatile memory and is used by the CPU 21 as a workspace. The storage device 23 is non-volatile memory such as an HDD (hard disk drive) or SSD (solid state drive). The storage device 23 stores programs executed by the CPU 21, including a 3D data transmission program, and various data. The storage device 23 is configured to function as the storage units 126 and 127 shown in Figure 4.

[0087] The CPU 21 loads the program stored in the storage device 23 into the RAM 22, interprets the program, and executes it. When the 3D data transmission program is executed by the CPU 21, it causes the CPU 21 to perform a series of processes as described with respect to the 3D data transmission device 12 shown in Figure 1. In other words, the CPU 21 is configured to function as a 3D data receiving unit 121, a virtual camera data receiving unit 122, and a 3D data conversion transmission unit 125.

[0088] The input / output interface 24 is an interface for direct communication with external devices. For example, the input / output interface 24 can be used to communicate with each 3D data measurement device 11. The communication interface 25 is an interface for communication with external devices via a communication network, which may include the Internet. For example, the communication interface 25 can be used to communicate with the 3D data reproduction device 13.

[0089] The hardware configuration shown in Figure 16 is illustrative, and a different hardware configuration may be adopted. For example, a dedicated processor such as an FPGA (field programmable gate array) may be used instead of a general-purpose processor. The processing circuit may include a general-purpose processor, a dedicated processor, or a combination of a general-purpose processor and a dedicated processor.

[0090] Programs such as 3D data transmission programs may be provided to computer 20 in a state where they are stored on a computer-readable recording medium. In this case, computer 20 is equipped with a drive to read data from the recording medium and retrieves the program from the recording medium. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memory. Programs may also be distributed via a communication network. Specifically, programs may be stored on a server on the communication network, and computer 20 may download the programs from the server.

[0091] The computer 20 shown in Figure 16 can also be used as a 3D data reproduction device 13. In this case, the CPU 21 is configured to function as a 3D data receiving unit 131, a virtual camera control unit 132, and a 3D data reproduction unit 133. The storage device 23 functions as a memory unit 134. The input / output interface 24 is used to communicate with the virtual camera operating device 14 and the monitoring device 15. The communication interface 25 is used to communicate with the 3D data transmission device 12.

[0092] As described above, in the 3D data reproduction system 10, the 3D data transmission device 12 receives distance image data from the 3D data measuring device 11 which measures distance image data as 3D data, determines the importance of the 3D data measuring device 11 based on the direction of the 3D data measuring device 11 and the direction of the virtual camera, performs compression processing on the distance image data by changing the degree of compression according to the determined importance, and transmits the compressed distance image data to the 3D data reproduction device 13.

[0093] The configuration, which changes the degree of compression according to the importance determined from the direction of the 3D data measurement device 11 and the direction of the virtual camera, makes it possible to increase the degree of compression for distance image data that has little contribution to the display image, and decrease the degree of compression for distance image data that has a large contribution to the display image. This makes it possible to reduce the amount of data transmitted while ensuring the quality of the display image. In other words, it makes it possible to reduce the amount of data transmitted without significantly impairing the viewer's experience.

[0094] The 3D data transmission device 12 may determine the importance of the 3D data measuring device 11 based on the direction of the 3D data measuring device 11, the direction of the virtual camera, the position of the 3D data measuring device 11, and the position of the virtual camera. In this case, the contribution of the distance image data obtained by each 3D data measuring device 11 to the display image can be evaluated more accurately. This makes it possible to ensure higher quality for the display image.

[0095] The degree of compression can be changed by altering the resolution. This reduces the time required for compression processing. As a result, it becomes possible to reduce the amount of data transmitted with low latency without significantly impairing the viewer's experience.

[0096] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple components disclosed. For example, if the problem can be solved and effects can be obtained even if some components are deleted from all the components shown in the embodiment, then the configuration with these components deleted can be extracted as an invention.

[0097] 10...3D data reproduction system 11...3D data measurement equipment 12...3D data transmission device 13...3D data reproduction device 14...Virtual camera operation device 15...Monitor device 20...Computer 21...CPU 22...RAM 23...Storage device 24...Input / output interface 25...Communication interface 121...3D data receiving unit 122...Virtual camera data receiving unit 123...Compression processing unit 124...3D data transmission unit 125...3D data conversion transmission unit 126...Storage unit 127...Storage unit 131...3D data receiving unit 132...Virtual camera control unit 133...3D data reproduction unit 134...Storage unit

Claims

1. A 3D data transmission device comprising: a 3D data receiving unit that receives 3D data from a 3D data measuring device that measures 3D data; a compression processing unit that determines the importance of the 3D data measuring device based on the direction of the 3D data measuring device and the direction of a virtual camera, and performs compression processing on the received 3D data by changing the degree of compression according to the determined importance; and a 3D data transmission unit that transmits the 3D data that has undergone the compression processing.

2. The 3D data transmission device according to claim 1, wherein the compression processing unit determines the importance of the 3D data measuring device from the direction of the 3D data measuring device, the direction of the virtual camera, the position of the 3D data measuring device, and the position of the virtual camera.

3. The 3D data transmission device according to claim 1, wherein the compression processing unit changes the degree of compression by changing the resolution of the 3D data.

4. A 3D data transmission program for causing a computer to function as one of the components of the data transmission device according to any one of claims 1 to 3.

Images