Image processing device and program

Abstract

An image processing device according to one aspect of the present invention includes: an acquisition unit that acquires, from the outside, dynamic point cloud data in which each point has three-dimensional position information and color information and which has a temporal dimension; a first storage unit that stores static point cloud data in which each point has three-dimensional position information and color information and which does not have a temporal dimension; a generation unit that calculates an overlapping portion, in the dynamic point cloud data, spatially overlapping the static point cloud data and generates a first average and a first standard deviation in the color information of the overlapping portion; a conversion unit that performs color conversion on the static point cloud data on the basis of the first average and the first standard deviation; and an integration unit that integrates the color-converted static point cloud data and the dynamic point cloud data to generate an image.

Description

Image processing device and program

[0001] One aspect of the present invention relates to an image processing device and a program.

[0002] A technology that captures distant areas as dynamic 3D data and live streams that 3D data is attracting attention. This technology will allow viewers to enjoy 3D content in real time through immersive display devices such as 3D displays and head-mounted displays.

[0003] On the other hand, devices capable of dynamically measuring space, such as ToF cameras, multiple cameras, or devices combining cameras and LiDAR, have limitations in the area they can measure with a single unit. Measuring a wide area dynamically in 3D requires a vast amount of equipment, making it difficult.

[0004] Conversely, there are technologies that measure wide areas as static 3D data by moving devices such as ToF cameras, multiple cameras, or devices combining cameras and LiDAR through space. While this method can capture information over a wide area with a small number of devices, it cannot capture and transmit dynamic changes in space, such as the movement of people or objects, because it is a static 3D space.

[0005] By combining the two methods, rapidly changing areas can be captured as dynamic point cloud data, which measures the 3D space in real time, albeit in a narrow area. Areas that cannot be captured by dynamic point cloud data can be covered as static point cloud data. This allows us to provide viewers with 3D content that is wide-ranging but can also represent dynamic changes, thus compensating for the shortcomings of each method.

[0006] However, the above method has its drawbacks. Because static point cloud data has fixed color information, for example, if you try to capture the movement of people outdoors throughout the day using the above method, a problem arises where the color information of the moving point cloud, which can track changes in lighting conditions in the space being measured due to time of day (morning and evening) or weather changes (when the sun is directly shining or when the sun is obscured by clouds), does not match the color information of the static point cloud, which retains the color it was measured in at the time.

[0007] Shoji Otsuki, "Research and Development of Automatic Coloring Technology for Point Cloud Data Using GANs," Transactions of the Japan Society of Civil Engineers, Series F3 (Civil Engineering Informatics), Vol. 78, No. 2, I_150-I_157, 2022.

[0008] As shown in Non-Patent Document 1, a method has been proposed that predicts the color of a point cloud and performs coloring based on machine learning. However, in order to properly color point cloud data with the above method, a large amount of training data of objects present in the point cloud space to be colored, such as roads, block walls, and vegetation, is required. Furthermore, the above method cannot color the point cloud in response to changes in the lighting environment, and therefore cannot change the coloring in response to changes in the color of the input moving point cloud data.

[0009] This invention was made in view of the above circumstances, and its objective is to provide a technology for converting the color information of static point cloud data in accordance with changes in the color information of input dynamic point cloud data.

[0010] An image processing apparatus according to one aspect of the present invention comprises: an acquisition unit that acquires moving point cloud data from an external source, in which each point has three-dimensional positional information and color information and has a temporal dimension; a first storage unit that stores static point cloud data in which each point has three-dimensional positional information and color information and does not have a temporal dimension; a generation unit that calculates the overlapping portion of the moving point cloud data that spatially overlaps with the static point cloud data and generates a first mean and a first standard deviation in the color information of the overlapping portion; a conversion unit that converts the static point cloud data to color based on the first mean and the first standard deviation; and an integration unit that integrates the color-converted static point cloud data and the moving point cloud data to generate an image.

[0011] According to one aspect of the present invention, a technique can be provided for converting the color information of static point cloud data in accordance with changes in the color information of input dynamic point cloud data.

[0012] Figure 1 is a block diagram of an image processing system according to an embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device. Figure 3 is a flowchart illustrating the operation of the 3D data measurement device. Figure 4 is a flowchart illustrating the operation of the 3D data acquisition unit. Figure 5 is a flowchart illustrating the operation of the mean / standard deviation generation unit. Figure 6 is a schematic diagram illustrating the process of calculating the overlap portion between moving point cloud data S and static point cloud data T. Figure 7 is a flowchart illustrating the operation of the color information conversion unit. Figure 8 is a flowchart illustrating the operation of the color update unit. Figure 9 is a flowchart illustrating the operation of the point cloud data integration unit.

[0013] Embodiments will be described below with reference to the drawings. The embodiments shown below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not defined by the shape, structure, arrangement, etc. of the components. Each functional block can be realized as a combination of hardware, software, or both. It is not essential that each functional block is distinguished as in the following example. For example, some functions may be performed by functional blocks other than the illustrative functional blocks. Furthermore, the illustrative functional blocks may be further divided into finer functional subblocks. In the following description, elements having the same function and configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] [1] Configuration of Image Processing System 1 The image processing system 1 processes point cloud data. Point cloud data is a collection of points that have three-dimensional position information (three-dimensional coordinate information) and color information. Point cloud data that has color information is also called colored point cloud data. Point cloud data that has three-dimensional position information is also called three-dimensional point cloud data.

[0015] Furthermore, the image processing system 1 can handle both dynamic point cloud data and static point cloud data. Dynamic point cloud data is point cloud data in which each point has three-dimensional positional information and color information, and also has a temporal dimension. In other words, dynamic point cloud data is point cloud data in which the position of an object that changes over time can be identified. Dynamic point cloud data is also called dynamic point cloud data. Static point cloud data is point cloud data in which each point has three-dimensional positional information and color information, and also does not have a temporal dimension. In other words, static point cloud data is point cloud data in which the positional information of an object does not change for at least a certain period of time. Static point cloud data is also called static point cloud data.

[0016] Figure 1 is a block diagram of an image processing system 1 according to an embodiment of the present invention. The image processing system 1 comprises a 3D data measurement device 10, an image processing device 20, and a display device 30.

[0017] The 3D data measurement device 10 is a device for measuring 3D data (three-dimensional data). 3D data is synonymous with point cloud data. The 3D data measurement device 10 comprises a 3D data measurement unit 11 and a 3D data transmission unit 12.

[0018] The 3D data measurement unit 11 measures the real space where sunlight conditions and / or lighting environments change as moving point cloud data. The 3D data measurement unit 11 is composed of a 3D camera. The 3D camera includes a ToF (Time of Flight) camera, a stereo system with multiple cameras, or a device that combines a camera with LiDAR.

[0019] The 3D data transmission unit 12 receives dynamic point cloud data from the 3D data measurement unit 11. After receiving the data, the 3D data transmission unit 12 transmits the dynamic point cloud data to the image processing device 20.

[0020] The image processing apparatus 20 is an apparatus that receives moving point cloud data as input and generates video data in which a two-dimensional image and a three-dimensional image are integrated. The image processing apparatus 20 includes a 3D data acquisition unit 21, an average / standard deviation generation unit 22, a color information conversion unit 23, a color update unit 24, a point cloud data integration unit 25, a static point cloud data storage unit 26, a color-converted static point cloud data storage unit 27, and a parameter storage unit 28.

[0021] The 3D data acquisition unit 21 receives and acquires moving point cloud data from the 3D data measurement apparatus 10 (specifically, the 3D data transmission unit 12). The 3D data acquisition unit 21 transmits the moving point cloud data to the average / standard deviation generation unit 22, the color information conversion unit 23, and the point cloud data integration unit 25.

[0022] The average / standard deviation generation unit 22 uses the moving point cloud data as reference data and the static point cloud data as a conversion target to calculate a spatially overlapping portion between the moving point cloud data and the static point cloud data. The average / standard deviation generation unit 22 calculates the average and standard deviation of color information of the reference data for point cloud data corresponding to the portion overlapping with the static point cloud data in the moving point cloud data.

[0023] The color information conversion unit 23 uses the static point cloud data as a conversion target and the moving point cloud data as reference data to calculate a spatially overlapping portion between the static point cloud data and the moving point cloud data. The color information conversion unit 23 calculates the average and standard deviation of color information of the conversion target for point cloud data corresponding to the portion overlapping with the moving point cloud data in the static point cloud data. The color information conversion unit 23 converts the color information of all points of the static point cloud data T based on the average and standard deviation of the color information of the reference data and the average and standard deviation of the color information of the conversion target.

[0024] The color update unit 24 updates the data stored in the color-converted static point cloud data storage unit 27 to the color-converted static point cloud data. The color update unit 24 also generates update information indicating whether or not color conversion has been performed on the static point cloud data.

[0025] The point cloud data integration unit 25 integrates the color-converted static point cloud data and the moving point cloud data based on the update information. Then, the point cloud data integration unit 25 generates an image of the integrated point cloud data based on parameters of a virtual camera.

[0026] A stationary point cloud data storage unit 26 stores stationary point cloud data. The stationary point cloud data stored in the stationary point cloud data storage unit 26 can be rewritten from the outside when it is desired to change the image of the stationary point cloud data.

[0027] A color-converted stationary point cloud data storage unit 27 stores the color-converted stationary point cloud data transmitted from a color updating unit 24.

[0028] A parameter storage unit 28 stores update information (boot values) transmitted from the color updating unit 24. The parameter storage unit 28 also stores parameters of a virtual camera.

[0029] A display device 30 displays video using the video data transmitted from a point cloud data integration unit 25. Video data has the same meaning as image data. The display device 30 is configured of a 2D display, a 3D display, an HMD (Head Mounted Display), or the like. The image data displayed by the display device 30 is processed into an image captured by a virtual camera. The virtual camera is set to virtually capture an image at a specific camera position. For the virtual camera, there are a pinhole camera or the like in the case of a 2D display, a stereo camera or the like in the case of a 3D display, and a stereo 360-degree panoramic camera or the like in the case of an HMD.

[0030] (Hardware Configuration of Image Processing Apparatus 20) FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus 20.

[0031] The image processing apparatus 20 can be configured by a computer. The image processing apparatus 20 includes a processor 40, a program storage unit 41, a data storage unit 42, a communication interface unit (communication I / F unit) 43, and an input / output interface unit (input / output I / F unit) 44. The data storage unit 42, the program storage unit 41, the communication interface unit 43, and the input / output interface unit 44 are connected to the processor 40 via a bus 45.

[0032] The processor 40 consists of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and the like. The processor 40 executes the operations of each of the processing units described above.

[0033] The program storage unit 41 includes, for example, a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and a non-volatile memory such as ROM (Read Only Memory). The program storage unit 41 stores the programs necessary for the processor 40 to perform various processes according to this embodiment. In other words, the operation of each processing unit described above is realized by having the processor 40 execute the programs stored in the program storage unit 41.

[0034] The data storage unit 42 includes, for example, non-volatile memory such as an HDD or SSD, and volatile memory such as RAM (Random Access Memory). The data storage unit 42 temporarily stores various data acquired and generated by the processor 40 during the process of executing various processes. The data storage unit 42 is also used as a workspace for the processor 40.

[0035] The communication interface unit 43 includes a wired communication module and / or a wireless communication module. The wireless communication module includes a wireless LAN (Local Area Network). The communication interface unit 43 performs interface processing with an external device according to a predetermined communication standard. The communication interface unit 43 is capable of receiving information from an external device and transmitting information to an external device.

[0036] An input unit 46 and an output unit 47 are connected to the input / output interface unit 44. The input / output interface unit 44 performs interface processing with the input unit 46 and the output unit 47 according to a predetermined communication standard.

[0037] The input unit 46 receives information entered by the user. The input unit 46 includes, for example, a keyboard and a mouse.

[0038] The output unit 47 outputs data generated by the processor 40. The output unit 47 includes, for example, a liquid crystal display device or an organic EL (Electro-Luminescence) display device.

[0039] [2] Operation Next, the operation of the image processing system 1 configured as described above will be explained.

[0040] First, let's explain the operation of the 3D data measurement device 10. Figure 3 is a flowchart illustrating the operation of the 3D data measurement device 10.

[0041] The 3D data measurement unit 11 measures moving point cloud data (step S100). For example, the 3D data measurement unit 11 continuously (dynamically) measures a narrow space as a three-dimensional point cloud at short time intervals.

[0042] The moving point cloud data is colored moving point cloud data that contains color information. The color information includes the RGB color system and the Lab color system. The RGB color system is a color system that represents color by the mixing ratio of the three primary colors of light: red (R), green (G), and blue (B). The Lab color system is a color system that represents lightness with L and chromaticity, which indicates hue and saturation, with a and b. The color information may also be RGBD data. RGBD data is data that includes color information (RGB) and depth information (D). In the case of the RGB color system, the moving point cloud data is point cloud data in which each point is represented by (X, Y, Z, R, G, B). In the case of the Lab color system, the moving point cloud data is point cloud data in which each point is represented by (X, Y, Z, L, a, b).

[0043] As an example, the 3D data measurement unit 11 is composed of a device that combines a camera and LiDAR, and is capable of measuring colored moving point cloud data in real time at 10 Hz. The colored moving point cloud data output by the 3D data measurement unit 11 is point cloud data in which each point is represented by (X, Y, Z, L, a, b), and the color information is 3-channel (ch) information represented in the Lab color system.

[0044] Next, the 3D data transmission unit 12 receives dynamic point cloud data from the 3D data measurement unit 11 (step S101). Specifically, the 3D data transmission unit 12 receives dynamic point cloud data S = {s} from the 3D data measurement unit 11, where each point is represented by (X, Y, Z, L, a, b). １ ,s ２ , , s ｍ The 3D data transmission unit 12 then receives the moving point cloud data S and transmits it to the image processing device 20 after the reception is complete (step S102). After the transmission is complete, the 3D data measurement device 10 returns to step S100.

[0045] Next, the operation of the image processing device 20 will be explained. Figure 4 is a flowchart illustrating the operation of the 3D data acquisition unit 21.

[0046] The 3D data acquisition unit 21 receives moving point cloud data S from the 3D data transmission unit 12 (step S200). Next, the 3D data acquisition unit 21 transmits the moving point cloud data S received in step S200 to the mean / standard deviation generation unit 22 (step S201). Next, the 3D data acquisition unit 21 transmits the moving point cloud data S received in step S200 to the color information conversion unit 23 (step S202). Next, the 3D data acquisition unit 21 transmits the moving point cloud data S received in step S200 to the point cloud data integration unit 25 (step S203). After the transmission is complete, the 3D data acquisition unit 21 returns to step S200.

[0047] Figure 5 is a flowchart illustrating the operation of the mean / standard deviation generation unit 22. The mean / standard deviation generation unit 22 receives moving point cloud data S from the 3D data acquisition unit 21 (step S300).

[0048] The static point group data storage unit 26 stores the static point group data T = {t} of n points. １ ,t ２ , , t ｎ} is stored. The static point cloud data T is colored static point cloud data having color information. The static point cloud data T is, for example, point cloud data in which one point is represented by (X, Y, Z, L, a, b). The static point cloud data T is, for example, point cloud data obtained by measuring a space wider than the space of the moving point cloud data S as a three-dimensional point cloud. The average / standard deviation generating unit 22 reads the static point cloud data T from the static point cloud data storage unit 26 (step S301).

[0049] Subsequently, the average / standard deviation generating unit 22 converts the color information of the static point cloud data T to be converted in accordance with the color system of the color information of the moving point cloud data S serving as reference data. For example, when the color information of the moving point cloud data S is in the Lab color system and the color information of the static point cloud data T to be converted is in the RGB color system, the average / standard deviation generating unit 22 converts the RGB values of the static point cloud data T to the Lab color system. Color system conversion methods are generally well-known knowledge. This process is not required if the color system of the color information of the static point cloud data T to be converted is originally the same as the color system of the color information of the moving point cloud data S serving as reference data.

[0050] Subsequently, the average / standard deviation generating unit 22 calculates a spatially overlapping portion of the moving point cloud data S with the static point cloud data T (step S302).

[0051] FIG. 6 is a schematic diagram illustrating a process of calculating an overlapping portion between the moving point cloud data S and the static point cloud data T. In FIG. 6, the reference colored point cloud corresponds to the moving point cloud data S, and the colored point cloud to be converted corresponds to the static point cloud data T.

[0052] In the present embodiment, when a moving point cloud and a static point cloud are superimposed, color conversion of the static point cloud is performed so that the static point cloud side matches the color tone of the moving point cloud side and looks natural. That is, as shown in FIG. 6, a case is assumed where there is an overlap between a measurement area of a reference colored point cloud and a measurement area of a colored point cloud to be converted.

[0053] Moving point cloud data S = {s １ , s ２ , ..., s ｍ} among which static point cloud data T = {t １ , t ２ , ..., tｎ The overlapping portion is denoted as point cloud data S'. Point cloud data S' can be obtained as follows.

[0054] First, point s ｉ The nearest neighbor t for ∈S ｊ We search for ∈T. And the distance s is less than or equal to the threshold ε ｉ This is added as point cloud data S'. In addition to the brute-force method for finding nearest neighbors, there is a method to efficiently search for nearest neighbors by organizing the data structures of the moving point cloud data S and the static point cloud data T using a spatial partitioning method for point cloud data such as KD-Tree.

[0055] Next, the mean and standard deviation generation unit 22 calculates the mean and standard deviation of the color information of the reference data based on the point cloud data S' (step S303).

[0056] As the mean and standard deviation of the color information of the reference data, the mean and standard deviation of the color information of points included in the moving point cloud data, or the mean and standard deviation of the color information of pixels included in the two-dimensional image data (RGB data) or RGBD data created when measuring the moving point cloud data can be used. The color information includes three RGB channels, or three Lab channels obtained by converting RGB data to the Lab color system. In this embodiment, it is desirable to use the Lab color system in which brightness and complementary color space are separated as separate axes.

[0057] The mean and standard deviation generation unit 22 uses the color information of all points in the point cloud data S' to calculate the mean and standard deviation of each channel of the color information of all points. The mean calculated from the point cloud data S' , standard deviation Let's assume that channel c is, for example, a Lab color system, then c ∈ {L, a, b}.

[0058] Next, the mean / standard deviation generation unit 22 calculates the mean , and standard deviation This is transmitted to the color information conversion unit 23 (step S304). After transmission is complete, the mean / standard deviation generation unit 22 returns to step S300.

[0059] Figure 7 is a flowchart illustrating the operation of the color information conversion unit 23. The color information conversion unit 23 receives the mean and standard deviation of the color information of the reference data (moving point cloud data S) from the mean / standard deviation generation unit 22 (step S400). Subsequently, the color information conversion unit 23 receives the moving point cloud data S from the 3D data acquisition unit 21 (step S401). Subsequently, the color information conversion unit 23 reads the static point cloud data T from the static point cloud data storage unit 26 (step S402).

[0060] Next, the color information conversion unit 23 calculates the portion of the static point cloud data T that spatially overlaps with the moving point cloud data S (step S403). Static point cloud data T = {t １ ,t ２ , , t ｎ Among the moving point data S = {s １ ,s ２ , , s ｍ The overlapping portion of} is denoted as point cloud data T'. Point cloud data T' can be obtained as follows.

[0061] First point t ｉ The nearest neighbor s for ∈T ｊ Search for ∈S. And distance t is below the threshold ε ｉ This is added as point cloud data T'. In addition to the brute-force method for finding nearest neighbors, there is a method to efficiently search for nearest neighbors by organizing the data structures of the moving point cloud data S and the static point cloud data T using a spatial partitioning method for point cloud data such as KD-Tree.

[0062] Next, the color information conversion unit 23 calculates the mean and standard deviation of the color information to be converted based on the point cloud data T' (step S404). Specifically, the color information conversion unit 23 uses the color information of all points in the point cloud data T' to calculate the mean and standard deviation of each channel of the color information of all points. The calculated mean is , standard deviation Let's assume that channel c is, for example, a Lab color system, then c ∈ {L, a, b}.

[0063] Next, the color information conversion unit 23 converts the color of each point in the static point group data T using equation (1) (step S405). The color information of each channel of point i in the static point group data T Let's assume that the static point group data T to be converted consists of n points, so i takes the form i ∈ {1, 2, ..., n}. Also, if channel c is, for example, the Lab color system, then c ∈ {L, a, b}. The color information of each channel after color conversion of point i is... Let's assume that equation (1) can be expressed as follows:

[0064] Then, the color information conversion unit 23 converts the color information of all points in the static point group data T using equation (1) to obtain static point group data U = {u １ , u ２ , ..., u ｎ Calculate}.

[0065] Next, the color information conversion unit 23 performs a process to convert the color-converted static point cloud data U back to the required color system. For example, it converts color information expressed in the Lab color system to the RGB color system. The method for converting color systems is generally well known. This process is unnecessary if the color system of the color information of the color-converted static point cloud data U is originally the same as the color system of the color information of the moving point cloud data S used as reference data.

[0066] Next, the color information conversion unit 23 transmits the color-converted static point cloud data U to the color update unit 24 (step S406). After the transmission is complete, the color information conversion unit 23 returns to step S400.

[0067] Figure 8 is a flowchart illustrating the operation of the color update unit 24. The color update unit 24 receives the color-converted static point cloud data U from the color information conversion unit 23 (step S500). Subsequently, the color update unit 24 updates the data in the color-converted static point cloud data storage unit 27 with the color-converted static point cloud data U (step S501).

[0068] Next, the color update unit 24 updates the boot value (step S502). The boot value is update information indicating that the data in the color-converted static point cloud data storage unit 27 has been updated. The boot value has a True / False value. True is also called activation, and False is also called deactivation. The boot value is set to True when the data in the color-converted static point cloud data storage unit 27 has been updated, and to False when it has not been updated. The color update unit 24 stores the boot value in the parameter storage unit 28. As an initial value, the static point cloud data T stored in the static point cloud data storage unit 26 is stored in the color-converted static point cloud data storage unit 27, and the boot value is set to True.

[0069] Figure 9 is a flowchart illustrating the operation of the point cloud data integration unit 25. The point cloud data integration unit 25 receives moving point cloud data S from the 3D data acquisition unit 21 (step S600).

[0070] Next, the point cloud data integration unit 25 determines whether the data in the color-converted static point cloud data storage unit 27 has been updated (step S601). Specifically, the point cloud data integration unit 25 determines the contents of the Boot value stored in the parameter storage unit 28.

[0071] If the static point cloud data has been updated, that is, if the Boot value is True (step S601 = Yes), the point cloud data integration unit 25 reads the static point cloud data U from the color-converted static point cloud data storage unit 27 (step S602). Since the boot value is initially set to True, the static point cloud data T that is initially stored in the color-converted static point cloud data storage unit 27 is read.

[0072] Next, the point cloud data integration unit 25 integrates the static point cloud data U read in step S602 and the dynamic point cloud data S received in step S600 to obtain point cloud data V = {s} with (n + m) points. １ ,s ２ , , s ｍ , u １ , u ２ , ..., u ｎ Generate} (step S604).

[0073] On the other hand, if the static point cloud data has not been updated, that is, if the Boot value is False (step S601 = No), the point cloud data integration unit 25 proceeds to step S603. The point cloud data integration unit 25 then integrates the static point cloud data and the dynamic point cloud data using the static point cloud data currently in use.

[0074] Next, the point cloud data integration unit 25 generates an image of the point cloud data V integrated in step S604 (step S605). The point cloud data integration unit 25 performs a three-dimensional coordinate transformation from the actually captured camera to the virtual camera. The rotation and translation parameters representing the difference between the two coordinate systems before and after the transformation are a 3x3 rotation matrix R ｃ and the three-dimensional translation vector t ｃ It is expressed as follows. The parameters of the virtual camera are stored in the parameter storage unit 28. Rotation matrix R ｃ and translation vector t ｃ This value is determined by the position and orientation of the virtual camera.

[0075] The point cloud data integration unit 25 converts the point cloud data V from reference coordinates to virtual camera coordinates, and sets V' to "V' = R ｃ V + t ｃ It is calculated as follows: V' is a point group composed of (n+m) points, and each point has information about (X, Y, Z, L, a, b).

[0076] Each point v' in the point cloud data V' ｉ of This is how it is written.

[0077] The point cloud data integration unit 25 integrates each point v' of the point cloud data V'. ｉ Applying equation (2) to this, we convert it to the coordinates (u, v) of the 2D image rendered by the virtual camera.

[0078]

[0079] Using equation (2), v' is determined by a virtual camera. ｉ When converting to coordinates (u, v) on a two-dimensional image, the point cloud data integration unit 25 performs the following: In that case, the corresponding point is removed. The point cloud data integration unit 25 rounds the coordinates (u, v) to an integer and removes v' which has the same coordinate values ​​(u, v). ｉ If there are multiple values, the point with the smallest value greater than 0 is selected. The transformed coordinates (u, v) are v' ｉ It contains color information from the Lab color system.

[0080] Furthermore, the point cloud data integration unit 25 generates image A with vertical and horizontal resolutions of X and Y from the 2D image rendered by the virtual camera. When generating image A, the point cloud data integration unit 25 rounds the coordinates (u, v) transformed by equation (2) to integers (u', v') and removes points from the 2D image that do not fall under the values ​​"0 ≤ u' ≤ X, 0 ≤ v' ≤ Y". Image A is an image with vertical and horizontal pixel counts of X and Y, and whose color information is represented in a 3-channel Lab color system.

[0081] Next, the point cloud data integration unit 25 transmits the image A generated in step S605 to the display device 30 (step S606).

[0082] The display device 30 has resolutions X and Y and is capable of displaying images represented in the Lab color system. The display device 30 displays image A transmitted from the point cloud data integration unit 25 on its screen.

[0083] [3] Effects of the Embodiment According to this embodiment, by taking only moving point cloud data as input, the color information of static point cloud data can be converted in accordance with the changes in the color information of the moving point cloud data. As a result, when representing a real space with a changing lighting environment as 3D content by combining moving point cloud data and static point cloud data, the effect of the color tone of the static point cloud data being converted to match that of the moving point cloud data is reduced, making it possible to provide users with a more natural 3D content experience.

[0084] Furthermore, by using a ToF camera, multiple cameras, or a device combining cameras and LiDAR, it is possible to continuously measure a narrow area as a point cloud at short time intervals, and when there is both dynamic point cloud data that can be experienced as three-dimensional and dynamic data, and wider-area static point cloud data that encompasses the area measured by the dynamic point cloud, it is possible to take the dynamic point cloud data as input and automatically adjust the color information of the static point cloud data to match the dynamic point cloud data.

[0085] Furthermore, it is possible to display more natural 3D content while keeping the cost of the image processing system 1 down and preventing an increase in the number of devices.

[0086] Each process according to the above-described embodiment can be stored as a program (software means) that can be executed by a computer, for example, on a storage medium such as a magnetic disk, optical disk, or semiconductor memory, or transmitted and distributed via a communication medium. The storage medium includes a storage medium provided in a computer or a storage medium provided in a device connected via a network. The computer can then read the program stored in the storage medium and execute the above-described process by having its operation controlled by the read program.

[0087] The present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0088] 1…Image processing system 10…3D data measurement device 11…3D data measurement unit 12…3D data transmission unit 20…Image processing device 21…3D data acquisition unit 22…Mean / standard deviation generation unit 23…Color information conversion unit 24…Color update unit 25…Point cloud data integration unit 26…Static point cloud data storage unit 27…Color converted static point cloud data storage unit 28…Parameter storage unit 30…Display device 40…Processor 41…Program storage unit 42…Data storage unit 43…Communication interface unit 44…Input / output interface unit 45…Bus 46…Input unit 47…Output unit

Claims

1. An image processing device comprising: an acquisition unit that acquires moving point cloud data from an external source, in which each point has three-dimensional positional information and color information and has a temporal dimension; a first storage unit that stores static point cloud data in which each point has three-dimensional positional information and color information and does not have a temporal dimension; a generation unit that calculates the overlapping portion of the moving point cloud data that spatially overlaps with the static point cloud data and generates a first mean and a first standard deviation in the color information of the overlapping portion; a conversion unit that converts the color of the static point cloud data based on the first mean and the first standard deviation; and an integration unit that integrates the color-converted static point cloud data and the moving point cloud data to generate an image.

2. The image processing apparatus according to claim 1, wherein the conversion unit calculates the overlapping portion of the static point cloud data that spatially overlaps with the moving point cloud data, generates a second mean and a second standard deviation in the color information of the overlapping portion, and converts the static point cloud data to color based on the first mean, the first standard deviation, the second mean, and the second standard deviation.

3. The image processing apparatus according to claim 1, further comprising: storing the color-converted static point cloud data in a second storage unit; and generating update information indicating whether or not the static point cloud data has been color-converted; and the integration unit, when the update information is activated, reads the static point cloud data stored in the second storage unit and integrates the read static point cloud data with the moving point cloud data.

4. A program for causing a computer to function as each part of the image processing apparatus according to any one of claims 1 to 3.

Images