Three-dimensional model generation device, three-dimensional model generation method, and three-dimensional model generation program
The 3D model generation device and method address the challenge of reproducing complex surfaces and missing components by segmenting and modeling external components from point cloud data, ensuring accurate 3D model generation.
Patent Information
- Application Number
- PCT/JP2026/005345
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-02-13
- Publication Date
- 2026-11-12
AI Technical Summary
Conventional 3D model generation techniques struggle to accurately reproduce complex surfaces and missing components in a 3D model due to occlusions or defects in the point cloud data of indoor spaces, such as beams, columns, and steps within buildings.
A 3D model generation device and method that performs segmentation, automatic segment determination, and modeling processes to extract flat surfaces from a point cloud, distinguishing between external and internal components, and generates a 3D model based on these segments, with optional user interaction for manual correction.
Enables the accurate reproduction of complex surfaces and missing components in a 3D model, even with defects in the point cloud data, by appropriately segmenting and modeling external components.
Smart Images

Figure JP2026005345_12112026_PF_FP_ABST
Abstract
Description
3D model generation device, 3D model generation method, and 3D model generation program
[0001] This disclosure relates to a 3D model generation apparatus, a 3D model generation method, and a 3D model generation program that generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building.
[0002] Conventionally, a 3D reconstruction technique is known that generates point cloud data (reconstructed point cloud) as 3D spatial information about a target space based on images taken of the target space within a building. Furthermore, a technique is known that generates a 3D model of the target space from the point cloud data of the target space.
[0003] In particular, a technique has been proposed for hybrid 3D reconstruction of indoor scenes that integrates object recognition. According to this technique, the point cloud is divided into internal and external components, and each is reconstructed using different methods. Specifically, the external components are reconstructed using plane estimation to recreate the room shape, while the internal components are reconstructed by recognizing objects using a random forest and fitting a CAD model. A technique is known in which the reconstructed internal and external components are merged to generate a lightweight 3D reconstruction model (see Non-Patent Literature 1).
[0004] Furthermore, a technique is known in which a 3D point cloud acquired from the interior of a building is read, divided into clusters according to the distance between the points, the floor, ceiling, and wall surfaces are sequentially polygonized based on each cluster, the 3D point clouds of structures other than the floor, ceiling, and wall surfaces are separated based on their positional relationship with the generated polygons, and a 3D model of structures other than the building is generated using primitives prepared in advance for structures from the separated 3D point clouds (see Patent Document 1).
[0005] Mingfan Li, 3 others, "Hybrid 3D Reconstruction of Indoor Scenes Integrating Object Recognition," [online], February 8, 2024, MDPI, [searched on March 4, 2020], Internet <https: / / www.mdpi.com / 2072-4292 / 16 / 4 / 638>
[0006] Japanese Patent Publication No. 2019-168976
[0007] According to conventional technology, a lightweight or more accurate 3D model of the building can be generated by separating internal and external components from point cloud data within the building and performing a reconstruction process.
[0008] However, rooms may contain beams, columns, and steps in addition to floors, ceilings, and walls. In such cases, conventional technology has the problem of not being able to accurately reproduce the complex surfaces of the room's components in a 3D model. Furthermore, problems have arisen where missing points in the reconstructed point cloud of the target space due to occlusion or other reasons result in missing components of the room being reproduced in the 3D model.
[0009] Therefore, the main objective of this disclosure is to provide a 3D model generation device, a 3D model generation method, and a 3D model generation program that can generate a 3D model in which the components of the target space are appropriately reproduced, even when the components of the target space are composed of complex surfaces and when there are defects in the restored point cloud of the target space.
[0010] The 3D model generation apparatus of the present disclosure is a 3D model generation apparatus that uses a processor to perform a process of generating a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to a target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0011] Furthermore, the 3D model generation method of this disclosure is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to a target space within a building, wherein the method is configured to perform a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, perform an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and perform a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0012] Further, the three-dimensional model generation program of the present disclosure is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model related to a component of a target space based on a restored point cloud obtained by a three-dimensional restoration process related to the target space inside a building, wherein the processor executes a segmentation process for extracting a plurality of segments representing flat surfaces from the restored point cloud, executes an automatic segment determination process for determining whether each segment extracted in the segmentation process is a modeling target representing an external component or a non-modeling target representing an internal component, and is configured to execute a modeling process for generating the three-dimensional model representing the external component based on the segments determined to be the modeling target in the automatic segment determination process.
[0013] According to the present disclosure, segments representing flat surfaces are extracted from the restored point cloud, it is determined whether each of the segments is a modeling target representing an external component, and a three-dimensional model representing the external component is generated using the segments representing the external component. Accordingly, even when the components of the target space are configured with complex surfaces and a defect occurs in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0014] Diagram of the overall configuration of the 3D model generation system according to this embodiment Diagram showing the restored point cloud of the room (target space) and the restored point cloud of the segment division state Diagram showing segments representing external components of the room (external segments) and segments representing internal components of the room (internal segments) Diagram showing a 3D model representing the external components of the room Diagram showing the schematic configuration of the user terminal Diagram showing the schematic configuration of the server Diagram showing the procedure of the 3D model generation process performed on the server Diagram showing the procedure of the preprocessing performed on the server Diagram showing the procedure of the segmentation process performed on the server Diagram showing the detection status of line segment Diagram showing the procedure of the automatic inside / outside determination process performed on the server Diagram showing the status of the floating surface determination process Diagram showing the maximum inter-face distance Diagram showing an overview of the bounding box determination process A flowchart showing the modeling process performed on the server. An explanatory diagram showing the generation status of the initial and candidate surfaces. An explanatory diagram showing an overview of the coverage score and distance penalty. An explanatory diagram showing how to calculate the same-area division cost. An explanatory diagram showing how to calculate the same-area division cost. An explanatory diagram showing how to calculate the same-area division cost. An explanatory diagram showing the basic screen displayed on the user terminal. An explanatory diagram showing the restored point cloud display screen displayed on the user terminal. An explanatory diagram showing the segment display screen displayed on the user terminal. An explanatory diagram showing the state in which the results of the automatic inside / outside determination process are reflected in the segment display screen. An explanatory diagram showing the 3D model display screen with the interpolation region displayed on the user terminal. An explanatory diagram showing the 3D model display screen with the error region displayed on the user terminal. An explanatory diagram showing the settings screen displayed on the user terminal.
[0015] A first invention made to solve the above problem is a three-dimensional model generation device that causes a processor to execute processing for generating a three-dimensional model related to a component of a target space based on a restored point cloud obtained by three-dimensional restoration processing related to the target space in a building, wherein the processor executes segmentation processing to extract a plurality of segments representing flat surfaces from the restored point cloud, executes automatic segment determination processing to determine whether each segment extracted in the segmentation processing is a modeling target representing an external component or a non-modeling target representing an internal component, and is configured to execute modeling processing for generating the three-dimensional model representing the external component based on the segments determined to be the modeling target in the automatic segment determination processing.
[0016] According to this, segments representing flat surfaces are extracted from the restored point cloud, it is determined whether each segment is a modeling target representing an external component, and a three-dimensional model representing the external component is generated using the segments representing the external component. Thereby, even when the components of the target space are configured with complex surfaces and a defect occurs in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0017] In a second aspect of the invention, in the modeling processing, the processor generates candidate surfaces that are candidates for constituent surfaces of the three-dimensional model based on the segments to be the modeling target, calculates an index value related to suitability as a constituent surface of the three-dimensional model for each of the candidate surfaces based on the distribution state of the point cloud on the candidate surfaces, selects the candidate surfaces suitable as the constituent surfaces of the three-dimensional model based on the index value, and is configured to generate the three-dimensional model related to the external component to be the modeling target based on the selected candidate surfaces.
[0018] According to this, it is possible to generate a three-dimensional model with high reproducibility regarding the external components of the target space.
[0019] Furthermore, the third invention is configured such that, before executing the segmentation process, the processor performs noise reduction, downsampling, and coordinate axis setting processes on the restored point cloud as preprocessing.
[0020] This allows for improved accuracy in subsequent processing. In addition, normal estimation, flatness estimation, and floor surface estimation may also be performed.
[0021] Furthermore, the fourth invention is configured such that the processor, in the segmentation process, performs a point cloud division process in which the restored point cloud is divided into multiple segments by a region expansion method that expands the region by sequentially adding neighboring points, and also performs a linear segment detection process to detect linear segments that are not to be modeled from among the segments obtained in the point cloud division process.
[0022] According to this method, segments representing flat surfaces can be appropriately extracted from the restored point cloud.
[0023] Furthermore, the fifth invention is configured such that, in the automatic segment determination process, the processor groups adjacent segments into one group, determines that the segment included in the group with the maximum number of points is a modeling target representing an external component, and determines that the segment not included in the group with the maximum number of points is an external component that forms a floating surface not in contact with an external component and is not a modeling target.
[0024] According to this, floating surfaces that are not in contact with external components can be considered as forming internal components, allowing for the appropriate determination of segments representing internal components of a building.
[0025] Furthermore, the sixth invention provides that, in the automatic segment determination process, the processor groups segments whose normal direction is the direction of each coordinate axis of the three-dimensional orthogonal coordinate system for each coordinate axis, determines the maximum distance between segments in each coordinate axis group as the maximum inter-face distance for each coordinate axis, sets a three-dimensional bounding box for each segment with sides parallel to each coordinate axis, and determines that a segment is a modeling target representing an external structure if the ratio of the length of the longest side to the maximum inter-face distance with respect to the coordinate axis parallel to the longest side of the three-dimensional bounding box exceeds a predetermined threshold.
[0026] According to this, segments that are sufficiently long relative to the overall dimensions of the target space can be appropriately identified as segments representing external components of the room.
[0027] Furthermore, the seventh invention is configured such that, in the modeling process, the processor calculates a point cloud distribution index value as the index value, which represents the proportion of the range in which point clouds are distributed on the candidate surface, and performs an optimization process to select the candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the point cloud distribution index value.
[0028] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0029] Furthermore, the eighth invention is configured such that, in the modeling process, the processor calculates a distance index value as the index value, which represents the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the distance index value.
[0030] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0031] Furthermore, the ninth invention is configured such that the processor calculates a region division index value as the index value, which represents the degree of inappropriateness regarding the combination of two candidate faces that cause the division of the segment that will become the candidate face, and performs an optimization process to select a candidate face that is appropriate as a constituent face of the three-dimensional model based on the region division index value.
[0032] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0033] Furthermore, the tenth invention is configured such that the processor presents the determination result of the automatic segment determination process to the user and executes a manual segment setting process that modifies the determination result of the automatic segment determination process in response to the user's operation of selecting a segment and specifying whether or not it is a modeling target.
[0034] According to this, if there is an error in the result of the automatic segment determination process, the user can easily correct that error.
[0035] Furthermore, the eleventh invention is a three-dimensional model generation method in which a processor performs a process to generate a three-dimensional model relating to the components of a target space based on a restored point cloud obtained by a three-dimensional restoration process relating to the target space within a building, wherein the method is configured to perform a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, perform an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and perform a modeling process to generate the three-dimensional model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0036] According to this, similar to the first invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0037] Furthermore, the twelfth invention is a 3D model generation program that uses a processor to perform a process to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the program performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0038] According to this, similar to the first invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0039] Furthermore, the 13th invention is a 3D model generation device that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, displays a screen on a display device that includes a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them, and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively, and performs a manual segment setting process to set the segments to be modeled according to the user's operation, and performs a modeling process to generate the 3D model of the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
[0040] According to this, by visually identifying and selecting segments displayed in the viewer, the user can appropriately set the segments necessary for generating a 3D model of the components in the target space as the modeling target. This makes it possible to generate a 3D model that appropriately reproduces the components in the target space, even if the components in the target space are composed of complex faces or if there are gaps in the restored point cloud of the target space. In this case, if the user operates to set the segments representing external components as the modeling target, a 3D model representing the external components can be generated.
[0041] Furthermore, the fourteenth invention provides a configuration in which the processor, in the modeling process, generates candidate faces that are candidates to be constituent faces of the three-dimensional model based on the segment to be modeled, calculates an index value for the suitability of each candidate face as a constituent face of the three-dimensional model based on the distribution of points on that candidate face, selects a candidate face that is suitable as a constituent face of the three-dimensional model based on that index value, and generates the three-dimensional model based on the selected candidate face.
[0042] According to this method, it is possible to generate a highly reproducible 3D model of the target space.
[0043] Furthermore, the fifteenth invention is configured such that the processor, in the viewer section, highlights the segments extracted in the segmentation process and suppresses the point cloud that was not extracted as segments.
[0044] This allows users to easily identify the extracted segments. In this case, highlighting and suppressing may be performed based on differences in color attributes (hue, lightness, saturation, etc.). For example, segments may be highlighted using chromatic colors, while points other than segments may be suppressed using achromatic colors.
[0045] Furthermore, the sixteenth invention is configured such that the processor, in the viewer section, highlights the segments set as modeling targets in the manual segment setting process and suppresses the display of segments set as not to be modeled targets.
[0046] This allows users to easily identify the segments to be modeled. In this case, highlighting and suppressing may be performed based on differences in color attributes (hue, lightness, saturation, etc.). For example, segments to be modeled may be highlighted using high-lightness chromatic colors, while segments not to be modeled may be suppressed using low-lightness chromatic colors.
[0047] Furthermore, the seventeenth invention is configured such that the processor highlights the selected segment in the viewer unit.
[0048] This allows the user to easily confirm the range of the selected segment. In this case, the selected segment may be highlighted by blinking (a change in brightness).
[0049] Furthermore, the 18th invention is configured such that, in the manual segment setting process, the processor collectively sets whether or not the selected segments are to be modeled, in response to user operations performed when multiple segments are selected.
[0050] According to this, the settings for multiple segments can be changed all at once.
[0051] Furthermore, the 19th invention is configured such that the processor displays a screen that includes an operation unit that instructs the user to cancel an add and delete operation and to redo the canceled operation.
[0052] According to this, users can instruct others to undo or redo addition and deletion operations.
[0053] Furthermore, the 20th invention is configured such that, before the manual segment setting process, the processor performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets or not, and in the manual segment setting process, the determination result of the automatic segment determination process is displayed on the viewer unit, and the determination result of the automatic segment determination process is changed according to the user's operation.
[0054] This allows for automatic segment determination, reducing the user's effort in selecting segments and specifying whether or not they should be modeled. Furthermore, if there are errors in the target determination results, the user can easily correct them.
[0055] Furthermore, the 21st invention is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to a target space within a building, the method comprising: performing a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud; displaying a screen on a display device that includes a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them; and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively; performing a manual segment setting process to set the segments to be modeled according to the user's operation; and performing a modeling process to generate the 3D model relating to the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
[0056] According to this, similar to the 13th invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a 3D model in which the components of the target space are appropriately reproduced.
[0057] Furthermore, the 22nd invention is a 3D model generation program that uses a processor to perform a process to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, and displays a screen on a display device that includes a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them, and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively, and performs a manual segment setting process to set the segments to be modeled according to the user's operation, and performs a modeling process to generate the 3D model of the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
[0058] According to this, similar to the 13th invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a 3D model in which the components of the target space are appropriately reproduced.
[0059] Furthermore, the 23rd invention is a 3D model generation device that uses a processor to perform a process of generating a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, presents the segments extracted by the segmentation process to the user, and performs a manual segment setting process to set whether the selected segments are to be modeled or not in response to the user's operation of selecting a segment and specifying whether or not it is to be modeled, and performs a modeling process to generate the 3D model based on the segments set as modeling targets in the manual segment setting process.
[0060] According to this, the user's operation of selecting segments allows for the appropriate setting of the segments necessary to generate a 3D model of the components in the target space as the modeling target. This makes it possible to generate a 3D model that appropriately reproduces the components in the target space, even if the components in the target space are composed of complex faces or if there are gaps in the reconstructed point cloud of the target space. In this case, if the user operates to set the segments representing external components as the modeling target, a 3D model representing the external components can be generated.
[0061] Furthermore, the 24th invention provides that the processor, in the modeling process, generates candidate faces that are candidates to be constituent faces of the three-dimensional model based on the segment to be modeled, calculates an index value for the suitability of each candidate face as a constituent face of the three-dimensional model based on the distribution of points on that candidate face, selects a candidate face that is suitable as a constituent face of the three-dimensional model based on that index value, and generates the three-dimensional model based on the selected candidate face.
[0062] According to this method, it is possible to generate a highly reproducible 3D model of the target space.
[0063] Furthermore, the 25th invention is configured such that, before executing the segmentation process, the processor performs noise reduction, downsampling, and coordinate axis setting processes on the restored point cloud as preprocessing.
[0064] This allows for improved accuracy in subsequent processing. In addition, normal estimation, flatness estimation, and floor surface estimation may also be performed.
[0065] Furthermore, the 26th invention is configured such that the processor, in the segmentation process, performs a point cloud division process in which the restored point cloud is divided into a plurality of segments by a region expansion method that expands the region by sequentially adding neighboring points, and also performs a linear segment detection process to detect linear segments that are not to be modeled from among the segments obtained in the point cloud division process.
[0066] According to this method, segments representing flat surfaces can be appropriately extracted from the restored point cloud.
[0067] Furthermore, the 27th invention is configured such that, in the modeling process, the processor calculates a point cloud distribution index value as the index value, which represents the proportion of the area in which point clouds are distributed on the candidate surface, and performs an optimization process to select the candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the point cloud distribution index value.
[0068] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0069] Furthermore, the 28th invention is configured such that, in the modeling process, the processor calculates a distance index value as the index value, which represents the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the distance index value.
[0070] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0071] Furthermore, the 29th invention is configured such that the processor calculates a region division index value as the index value, which represents the degree of inappropriateness regarding the combination of two candidate faces that cause the division of the segment that will become the candidate face, and performs an optimization process to select a candidate face that is appropriate as a constituent face of the three-dimensional model based on the region division index value.
[0072] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0073] Furthermore, the 30th invention is configured such that the processor performs an automatic segment determination process before the manual segment setting process to determine whether the segment extracted in the segmentation process is a modeling target or not, and in the manual segment setting process, presents the determination result of the automatic segment determination process to the user and changes the determination result of the automatic segment determination process according to the user's operation.
[0074] According to this system, an automatic segment determination process is executed, reducing the effort required of the user to select segments and specify whether or not they should be modeled. Furthermore, if there are errors in the results of the automatic segment determination process, the user can easily correct those errors.
[0075] Furthermore, the 31st invention is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to the target space within a building, wherein a segmentation process is performed to extract a plurality of segments representing flat surfaces from the restored point cloud, the segments extracted in the segmentation process are presented to the user, and in response to the user's operation of selecting a segment and specifying whether or not it is a modeling target, a manual segment setting process is performed to set whether or not the selected segment is a modeling target, and a modeling process is performed to generate the 3D model based on the segment set as a modeling target in the manual segment setting process.
[0076] According to this, similar to the 23rd invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0077] Furthermore, the 32nd invention is a 3D model generation program that uses a processor to perform a process to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the program performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, presents the segments extracted by the segmentation process to the user, and performs a manual segment setting process to set whether the selected segments are to be modeled or not in response to the user's operation of selecting a segment and specifying whether or not it is to be modeled, and performs a modeling process to generate the 3D model based on the segments set as modeling targets in the manual segment setting process.
[0078] According to this, similar to the 23rd invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0079] Furthermore, the 33rd invention is a 3D model generation device that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, generates candidate surfaces that are candidates for constituent surfaces of the 3D model based on the segments extracted by the segmentation process, calculates an index value for the appropriateness of each candidate surface as a constituent surface of the 3D model based on the distribution of points on the candidate surface, selects an appropriate candidate surface as a constituent surface of the 3D model based on the index value, and performs a modeling process to generate the 3D model of the components to be modeled based on the selected candidate surface.
[0080] According to this method, appropriate candidate faces are selected from among the candidate faces generated based on the extracted segments, and the modeling process is performed. This makes it possible to generate a 3D model that appropriately reproduces the structures of the target space, even if the structures in the target space are composed of complex faces, or if there are gaps in the reconstructed point cloud of the target space.
[0081] Furthermore, the 34th invention is configured such that, before executing the segmentation process, the processor performs noise reduction, downsampling, and coordinate axis setting processes on the restored point cloud as preprocessing.
[0082] This allows for improved accuracy in subsequent processing. In addition, normal estimation, flatness estimation, and floor surface estimation may also be performed.
[0083] Furthermore, the 35th invention is configured such that the processor, in the segmentation process, performs a point cloud division process in which the restored point cloud is divided into a plurality of segments by a region expansion method that expands the region by sequentially adding neighboring points, and also performs a linear segment detection process to detect linear segments that are not to be modeled from among the segments obtained in the point cloud division process.
[0084] According to this method, segments representing flat surfaces can be appropriately extracted from the restored point cloud.
[0085] Furthermore, the 36th invention is configured such that, in the modeling process, the processor calculates a point cloud distribution index value as the index value, which represents the proportion of the range in which point clouds are distributed on the candidate surface, and performs an optimization process to select the candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the point cloud distribution index value.
[0086] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0087] Furthermore, the 37th invention is configured such that, in the modeling process, the processor calculates a distance index value as the index value, which represents the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the distance index value.
[0088] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0089] Furthermore, the 38th invention is configured such that the processor calculates a region division index value as the index value, which represents the degree of inappropriateness regarding the combination of two candidate faces that cause the division of the segment that will become the candidate face, and performs an optimization process to select a candidate face that is appropriate as a constituent face of the three-dimensional model based on the region division index value.
[0090] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0091] Furthermore, the 39th invention is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to the target space within a building, wherein the method performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, generates candidate surfaces that are candidates to be constituent surfaces of the 3D model based on the segments extracted by the segmentation process, calculates an index value for the appropriateness of each candidate surface as a constituent surface of the 3D model based on the distribution of points on the candidate surface, selects an appropriate candidate surface as a constituent surface of the 3D model based on the index value, and performs a modeling process to generate the 3D model relating to the components to be modeled based on the selected candidate surface.
[0092] According to this, similar to the 33rd invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0093] Furthermore, the 40th invention is a 3D model generation program that uses a processor to perform a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to the target space within a building, wherein the program performs a segmentation process to extract multiple segments representing flat surfaces from the restored point cloud, generates candidate surfaces that are candidates for constituent surfaces of the 3D model based on the segments extracted by the segmentation process, calculates an index value for the appropriateness of each candidate surface as a constituent surface of the 3D model based on the distribution of points on the candidate surface, selects an appropriate candidate surface as a constituent surface of the 3D model based on the index value, and performs a modeling process to generate the 3D model relating to the components to be modeled based on the selected candidate surface.
[0094] According to this, similar to the 33rd invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the components of the target space are appropriately reproduced.
[0095] The 41st invention, made to solve the aforementioned problems, is a 3D model generation device that uses a processor to perform a process of generating a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs a modeling process to generate the 3D model based on the segments extracted in the segmentation process, performs a focus region detection process to detect a focus region where there is a noteworthy discrepancy between the 3D model generated in the modeling process and the restored point cloud, and displays the 3D model as a 2D image and displays the focus region on the 3D model in an identifiable manner, and displays a screen including a viewer section on a display device.
[0096] According to this, the user can easily confirm situations where there is a discrepancy between the generated 3D model and the reconstructed point cloud because the components of the target space are composed of complex surfaces and there are gaps in the reconstructed point cloud of the target space. In this case, the area of interest may be displayed in a way that distinguishes it from other areas based on differences in color attributes (hue, lightness, saturation, etc.).
[0097] Furthermore, the 42nd invention provides a configuration in which, in the modeling process, the processor generates candidate faces that are candidates for constituent faces of the 3D model based on the segment to be modeled, calculates an index value for the suitability of each candidate face as a constituent face of the 3D model based on the distribution of points on that candidate face, selects a candidate face that is suitable as a constituent face of the 3D model based on that index value, and generates the 3D model relating to the constituent object to be modeled based on the selected candidate face.
[0098] According to this method, it is possible to generate a highly reproducible 3D model of the target space.
[0099] Furthermore, the 43rd invention provides that the processor, when the restored point cloud generated by the 3D restoration process is input to the viewer unit, displays the restored point cloud as a 2D image in which the viewpoint can be changed by the user's operation; when segments are extracted by the segmentation process, displays the restored point cloud in the segmented state as a 2D image in which the viewpoint can be changed by the user's operation; and when the 3D model is generated by the modeling process, displays the 3D model as a 2D image in which the viewpoint can be changed by the user's operation.
[0100] According to this, the input reconstructed point cloud, the reconstructed point cloud in its segmented state, and the 3D model can be observed in detail by changing the viewpoint.
[0101] Furthermore, the 44th invention is configured such that the processor displays a screen on the display device that includes an operation unit that instructs the execution of an automatic segment determination process that determines whether or not the segment extracted in the segmentation process is a modeling target.
[0102] According to this, users can have the automated segment determination process run as needed.
[0103] Furthermore, the 45th invention is configured such that the processor displays a screen including an operation unit for specifying the processing conditions for the automatic segment determination process.
[0104] According to this, users can change the processing conditions for the automatic segment determination process as needed. In this case, the processing conditions may be a threshold (segment size) used to determine whether a segment is an external segment representing an external component or an internal segment representing an internal component, based on the size of the segment relative to the size of the entire target space.
[0105] Furthermore, the 46th invention provides a configuration in which the processor displays a screen on the display device that includes a viewer unit that displays the segments extracted in the segmentation process in a way that allows the user to select them, and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively, and executes a manual segment setting process to set the segments to be modeled in response to the user's operation, and executes the modeling process based on the segments set as the modeling target in the manual segment setting process.
[0106] According to this, users can visually identify the segments displayed in the viewer and set the segments to be modeled.
[0107] Furthermore, the 47th invention is configured such that the processor performs a process to detect an interpolation region in which a surface of the 3D model is formed in a manner that interpolates the missing parts of the restored point cloud, and displays the interpolation region on the screen.
[0108] This allows users to easily see where interpolation was performed correctly and where it was performed incorrectly.
[0109] Furthermore, the 48th invention is configured such that the processor performs a process to determine the interpolation reason for the interpolation region and displays the interpolation reason on the screen.
[0110] According to this, users can easily see the reason for interpolation.
[0111] Furthermore, the 49th invention is configured such that the processor performs a process to detect an error region in which the error of the 3D model with respect to the restored point cloud is greater than a predetermined threshold, and displays the error region on the screen.
[0112] According to this, users can easily identify areas in the 3D model where large errors have occurred.
[0113] Furthermore, the 50th invention is configured such that the processor displays a screen on the display device that includes an operation unit for specifying whether or not to display the area of interest for each item, an operation unit for specifying the display mode of the area of interest, and an operation unit for specifying the processing conditions for the area of interest detection process.
[0114] According to this, the user can specify whether or not to display the area of interest for each item, for example, whether or not to display the interpolation area for each item of interpolation reason. Furthermore, the user can specify the display method of the area of interest, for example, the color of the interpolation area and error area. Additionally, the user can specify the processing conditions for the area of interest detection process, for example, the threshold used in the area of interest detection process.
[0115] Furthermore, the 51st invention is configured such that the processor displays a screen on the display device that includes an operation unit that instructs the three-dimensional model generated in the modeling process to be output in a specified format.
[0116] According to this, the 3D model can be saved to an appropriate memory location.
[0117] Furthermore, the 52nd invention is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to a target space within a building, the method comprising: performing a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud; performing a modeling process to generate the 3D model based on the segments extracted by the segmentation process; performing a focus region detection process to detect a focus region where there is a noteworthy discrepancy between the 3D model generated by the modeling process and the restored point cloud; and displaying the 3D model as a 2D image and displaying the focus region on the 3D model, with the display screen including a viewer section.
[0118] According to this, similar to the 41st invention, the user can easily confirm a situation where the components of the target space are composed of complex surfaces, and where there is a discrepancy between the generated 3D model and the restored point cloud due to missing data in the restored point cloud of the target space.
[0119] Furthermore, the 53rd invention is a 3D model generation program that uses a processor to perform a process to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, the program performing a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, a modeling process to generate the 3D model based on the segments extracted by the segmentation process, a process to detect areas of interest where there is a noteworthy discrepancy between the 3D model generated by the modeling process and the restored point cloud, and a display device that displays a screen including a viewer that displays the areas of interest in an identifiable manner on the 3D model, while also displaying the 3D model as a 2D image.
[0120] According to this, similar to the 41st invention, the user can easily confirm a situation where the components of the target space are composed of complex surfaces, and where there is a discrepancy between the generated 3D model and the restored point cloud due to missing data in the restored point cloud of the target space.
[0121] The embodiments of this disclosure will be described below with reference to the drawings.
[0122] Figure 1 is an overall diagram of the 3D model generation system according to this embodiment.
[0123] This system performs a 3D reconstruction process on a target space within a building to obtain a reconstructed point cloud of the target space, and then generates a 3D model of the components of the target space based on that reconstructed point cloud. In this embodiment, a 3D model is generated for a room (living room) where furniture is placed, but the target space is not limited to a room and may be a warehouse, workshop, etc.
[0124] This system comprises a user terminal 1 (display device) and a server 2 (3D model generation device). User terminal 1 and server 2 are connected via a network such as a LAN or the internet.
[0125] User terminal 1 comprises a main unit 11 and a camera 12. User terminal 1 can be a tablet or a notebook PC.
[0126] Camera 12 is a visible light camera, that is, a monocular camera that detects visible light and photographs the subject, and outputs a captured image, for example, an RGB color image. In addition to camera 12 (visible light camera), the user terminal 1 may also be equipped with sensors such as a depth camera for measuring the distance to the subject and an IMU (Inertial Measurement Unit) for detecting three-dimensional angular velocity and acceleration, for the 3D reconstruction performed on server 2.
[0127] The user (worker) moves through the target space (room) while holding the main unit 11 of the user terminal 1. At this time, the target space is photographed by the camera 12 on the user terminal 1, and images of each point in the target space are acquired sequentially. Here, an example is shown in which images are acquired by a user walking through the target space while holding the main unit 11, but images of the target space may also be acquired using a self-propelled robot equipped with the user terminal 1 or the like.
[0128] Server 2 is composed of a PC. Based on the captured image received from user terminal 1, Server 2 performs a 3D reconstruction process on the target space and generates a reconstructed point cloud of the target space. Server 2 also generates a 3D model of the target space based on the reconstructed point cloud of the target space.
[0129] Next, we will explain the overview of the 3D model generation process performed on Server 2. Figure 2 is an explanatory diagram showing the restored point cloud of the room (target space) and the restored point cloud of the segment division state. Figure 3 is an explanatory diagram showing the segment representing the external components of the room (external segment) and the segment representing the internal components of the room (internal segment). Figure 4 is an explanatory diagram showing the 3D model (room model) representing the external components of the room. In the examples shown in Figures 3 and 4, the viewpoint is set to the outside of the room, and the interior of the room is made visible by making the wall and ceiling surfaces on the side closer to the viewpoint transparent.
[0130] On server 2, as shown in Figure 2(A), a reconstructed point cloud of the room is generated by a 3D reconstruction process based on captured images of the room (target space). In the example shown in Figure 2(A), the external components of the room include a floor, ceiling, and walls. Furthermore, there is a beam-like step at the boundary between the ceiling and the wall, and a columnar protrusion at the boundary between the two walls, resulting in the external components of the room being composed of complex surfaces. The internal components of the room include a sofa, desk, chair, and shelves.
[0131] Here, a 3D model of only the external components of a room, that is, a 3D model of a room without any furniture or other objects inside, can be used, for example, to conduct on-site inspections of buildings without visiting the site, or to simulate room layout changes or renovations. For this reason, in this embodiment, segments (sub-point clouds) representing the flat surfaces of the external components are extracted from the reconstructed point cloud of the room, and a 3D model representing the external components is generated based on these segments.
[0132] Specifically, as shown in Figure 2(B), when segmentation processing is performed on server 2, multiple segments (sub-point clouds) representing flat surfaces are extracted from the reconstructed point cloud of the room. Each extracted segment has a different position and orientation. In this embodiment, the segmentation processing is performed using the region expansion method to extract flat surfaces as segments, but small flat areas, such as the legs of a desk or chair, are not extracted as segments representing flat surfaces. Figure 2(B) shows an example where the tabletop of a desk and the seat and back of a chair are extracted as segments.
[0133] Next, as shown in Figures 3(A) and (B), Server 2 performs a process (internal / external sorting) to divide the extracted segments into external segments representing external components of the room and internal segments representing internal components of the room. As shown in Figure 3(A), the external segment group includes segments representing external components of the room such as floors, ceilings, walls, beams, columns, and steps. As shown in Figure 3(B), the internal segment group includes segments representing furniture such as sofas, desks (tabletops), chairs (seats and backs), and shelves, which are internal components of the room. The internal segment group also includes linear segments that appear at the boundary between two surfaces of external components.
[0134] In this embodiment, it is determined whether the segments extracted from the restored point cloud correspond to external or internal components of a room, and based on the determination result, the segments are divided into external segments representing external components and internal segments representing internal components (automatic internal / external determination process).
[0135] Furthermore, in this embodiment, segments extracted from the restored point cloud are presented to the user, and in response to the user's operation (internal / external sorting operation) specifying whether they belong to internal or external components, the selected segments are sorted into external segments representing external components and internal segments representing internal components (manual internal / external setting process).
[0136] Next, as shown in Figure 4, Server 2 generates a 3D model representing the external components based on the external segments representing the external components. In this 3D model, the internal components of the room, i.e., furniture such as sofas, desks, chairs, and shelves placed in the room, are removed. In addition, as external components of the room, complex surfaces such as beams, columns, and steps are appropriately reproduced, as well as the floor, ceiling, and walls.
[0137] In this embodiment, an internal / external sorting process is performed to generate a 3D model representing the external components of the room, i.e., a 3D model in which the internal components of the room (such as furniture) have been removed. However, an internal / external sorting process may not be performed, and a 3D model in which the internal components of the room remain in place may be generated. Such a 3D model in which the internal components of the room remain in place can be used for simulations to analyze the physical characteristics of the room, such as simulations to analyze acoustic characteristics or air conditioning characteristics.
[0138] Furthermore, in this embodiment, segments representing the external components of a room are set as the modeling target, and a 3D model representing the external components of a room is generated. However, segments representing components required by the user may be set as the modeling target, and a 3D model representing the components required by the user may be generated.
[0139] Furthermore, in this embodiment, segments representing flat surfaces are extracted from the reconstructed point cloud of the room as a partial point cloud that forms the basis of the constituent surfaces of the 3D model. However, in addition to segments representing flat surfaces, segments representing basic shapes such as cylindrical surfaces may also be extracted, and the constituent surfaces of the 3D model may be formed based on the segments representing these basic shapes.
[0140] Furthermore, when the goal is to generate a 3D model representing the external components, which are room models, it is desirable to restrict the segmentation process of the restored point cloud by using area conditions of the constituent surfaces to prevent the extraction of fine segments.
[0141] Next, we will describe the general configuration of user terminal 1. Figure 5 is a block diagram showing the general configuration of user terminal 1.
[0142] In addition to the camera 12, the user terminal 1 includes a display 13 (display unit), an input device 14, a storage unit 15, a processor 16 (CPU), and a communication unit 17.
[0143] The display 13 displays various operation screens related to shooting and editing operations, setting screens for setting various processing conditions, and viewing screens for checking the generated restored point cloud and 3D model. The input device 14 is used by the user for input operations. If the user terminal 1 is a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.
[0144] The memory unit 15 stores programs executed by the processor 16, etc. The memory unit 15 also stores images captured by the camera 12. The captured images are transmitted to the server 2 at appropriate intervals.
[0145] The processor 16 executes programs stored in the memory unit 15 to perform various processes related to taking pictures with the camera 12, displaying images on the display 13, and acquiring operation information from the input device 14.
[0146] The communication unit 17 communicates with the server 2. Specifically, the communication unit 17 transmits images captured by the camera 12 and operation information acquired by the user through operation of the input device 14 to the server 2. The communication unit 17 also receives various screen display information transmitted from the server 2.
[0147] Next, we will describe the general configuration of Server 2. Figure 6 is a block diagram showing the general configuration of Server 2.
[0148] Server 2 comprises a communication unit 21, a storage unit 22, and a processor 23.
[0149] The communication unit 21 communicates with the user terminal 1. Specifically, the communication unit 21 receives captured images and operation information transmitted from the user terminal 1. The communication unit 21 also transmits display information for various screens to be displayed on the user terminal 1 to the user terminal 1.
[0150] The memory unit 22 stores programs executed by the processor 23, etc. The memory unit 22 also stores data such as the reconstructed point cloud, segments, and 3D model of the target space generated by the processor 23.
[0151] The processor 23 performs various processes by executing programs stored in the memory unit 22. In this embodiment, the processor 23 performs 3D reconstruction processing, 3D model generation processing, focus region detection processing, and display processing.
[0152] In the 3D reconstruction process, the processor 23 generates a reconstructed point cloud of the target space based on the captured image received from the user terminal 1. The 3D reconstruction process can be realized, for example, by estimating the position and orientation of the camera 12 at the time of capture in the world coordinate system using the SLAM (Simultaneous Localization and Mapping) method, and then superimposing depth information obtained from a depth camera or the like onto the world coordinate system based on this position and orientation estimation information.
[0153] In the 3D model generation process, the processor 23 generates a 3D model of the room based on the reconstructed point cloud of the room generated in the 3D reconstruction process. The 3D model generation process includes preprocessing, segmentation, internal / external sorting, and modeling.
[0154] In the preprocessing stage, the processor 23 performs processes such as noise reduction, downsampling, and setting of coordinate axes on the reconstructed point cloud generated by the 3D reconstruction process.
[0155] In the segmentation process, the processor 23 extracts multiple segments (sub-point clouds) representing flat surfaces from the pre-processed restored point cloud.
[0156] In the internal / external sorting process, the processor 23 sorts each segment extracted in the segmentation process into an external segment representing the external components of the room and an internal segment representing the internal components of the room. The internal / external sorting process includes an automatic internal / external determination process (automatic segment determination process) and a manual internal / external setting process (manual segment setting process).
[0157] In the automatic internal / external determination process, the processor 23 determines whether the extracted segment corresponds to an external or internal component of the room, and then divides the segment into an internal segment representing an internal component and an external segment representing an external component.
[0158] In the manual internal / external configuration process, the processor 23 distributes the selected segments into internal segments representing internal components and external segments representing external components, in response to a user operation (internal / external distribution operation) specifying whether the extracted segments belong to internal or external components.
[0159] In the modeling process, the processor 23 generates a three-dimensional model (room model) representing the external components of the room based on the external segments obtained in the internal / external sorting process (automatic internal / external determination process, manual internal / external setting process). Alternatively, in the modeling process, a three-dimensional model (room model) with furniture and other items placed inside the room may be generated based on all the segments extracted in the segmentation process.
[0160] In the region of interest detection process, the processor 23 detects regions where there is a significant discrepancy between the 3D model generated in the 3D model generation process and the restored point cloud, and identifies these regions of interest. The region of interest detection process includes interpolation region detection, interpolation reason determination, and error region detection.
[0161] In the interpolation region detection process, the processor 23 compares the restored point cloud with the 3D model and detects as an interpolation region a region in which a surface of the 3D model is formed in a manner that interpolates the missing points in the restored point cloud.
[0162] In this embodiment, when determining the constituent surfaces of a 3D model from segments during the modeling process, the surfaces represented by the segments are first extended to the entire range of the reconstructed point cloud, and then surfaces are selected based on the distribution of the point cloud. As a result, in the 3D model, surfaces are formed with interpolation applied to areas in the reconstructed point cloud where there are no points.
[0163] In the interpolation reason determination process, the processor 23 determines the interpolation reason for the interpolation region detected in the interpolation region detection process, based on the restored point cloud generated in the 3D reconstruction process and the position and orientation estimation information obtained by the SLAM method in the 3D reconstruction process.
[0164] Specifically, it is determined whether the interpolation region is included in the field of view (shooting range) of camera 12. It is also determined whether occlusion occurs in the interpolation region due to other objects. If the interpolation region is included in the field of view of camera 12 and occlusion does not occur due to other objects, it is determined to be interpolation of a point cloud area with no points due to an opening. If the interpolation region is included in the field of view of camera 12 and occlusion occurs due to other objects, it is determined to be interpolation of a point cloud gap due to occlusion. If the interpolation region is not included in the field of view of camera 12, it is determined to be interpolation of a point cloud gap due to a missed shot. Note that point clouds are not formed in openings such as doorways and windows. Also, point clouds are not formed when the floor or walls are obscured by furniture such as sofas or shelves.
[0165] In the error region detection process, the processor 23 compares the restored point cloud with the 3D model and detects as an error region any region where a large error occurs in the 3D model relative to the restored point cloud. Specifically, the error of the 3D model relative to the restored point cloud (the difference between the restored point cloud and the 3D model) is calculated (error calculation process), and any region where this error is greater than a predetermined threshold is detected as an error region. At this time, for example, corresponding points are found in the restored point cloud and the 3D model (mesh model), and the error is calculated as a percentage based on the distance between these corresponding points.
[0166] Furthermore, the interpolation region may be detected overlapping with the error region. That is, since the interpolation region is a region where a surface of the 3D model is formed in a manner that interpolates the missing points in the restored point cloud, the error of the 3D model relative to the restored point cloud also becomes large in the interpolation region. For this reason, a region in which a large error occurs in the 3D model relative to the restored point cloud, and which does not correspond to the interpolation region, may be detected as the error region.
[0167] In the display process, the processor 23 generates a screen for instructions, confirmations, and settings related to the 3D reconstruction process and the 3D model generation process, and displays this screen on the display 13 of the user terminal 1.
[0168] The display process includes the restoration point cloud display process, the segment display process, the 3D model display process, and the focus region display process.
[0169] In the restored point cloud display process, the restored point cloud generated in the 3D restoration process is converted into a 2D image based on a specified viewpoint and displayed on the screen of the user terminal 1. At this time, the display mode of the restored point cloud changes according to the user's 3D operations, that is, operations that specify the position and angle of the viewpoint in 3D space.
[0170] In the segment display process, the restored point cloud, which reflects the extraction results from the segmentation process, is converted into a two-dimensional image based on a specified viewpoint and displayed on the screen of the user terminal 1. At this time, multiple segments included in the restored point cloud are displayed in different colors according to a predetermined color coding rule. Furthermore, the display mode of the restored point cloud in the segmented state changes in response to the user's three-dimensional operation, that is, the operation of specifying the position and angle of the viewpoint in three-dimensional space.
[0171] In the 3D model display process, the 3D model generated in the 3D model generation process is converted into a 2D image based on a specified viewpoint and displayed on the screen of the user terminal 1. At this time, the display mode of the 3D model changes according to the user's 3D operation, that is, the operation of specifying the position and angle of the viewpoint in 3D space.
[0172] The process for displaying the area of interest includes the interpolation area display process, the interpolation reason display process, and the error area display process.
[0173] In the interpolation region display process, the processor 23 displays the interpolation region detected in the interpolation region detection process on the 3D model (room model) displayed on the screen of the user terminal 1 in a way that allows it to be distinguished from other regions. Specifically, the interpolation region is highlighted in a different color from other regions. Furthermore, the interpolation region can be displayed in a different color depending on the reason for interpolation. In addition, the interpolation region can be displayed in a color specified by the user.
[0174] In the interpolation reason display process, the processor 23 displays the interpolation reasons determined in the interpolation reason determination process on the 3D model (room model) displayed on the screen of the user terminal 1. Specifically, the interpolation reasons for the selected interpolation area are displayed in response to the user's operation (mouse over) to select an interpolation area.
[0175] In the error region display process, the processor 23 displays the error region detected in the error region detection process on the 3D model (room model) displayed on the screen of the user terminal 1. Specifically, the error region is highlighted in a different color from other regions. Furthermore, the error region can be displayed in a color specified by the user.
[0176] Next, we will explain the 3D model generation process performed on server 2. Figure 7 is a flowchart showing the procedure for the 3D model generation process.
[0177] In the 3D model generation process, the processor 23 first performs preprocessing (ST101). In the preprocessing, the reconstructed point cloud generated in the 3D reconstruction process is subjected to processes such as noise reduction, downsampling, and setting of coordinate axes.
[0178] Next, the processor 23 performs segmentation processing (ST102). In segmentation processing, multiple segments (sub-point clouds) representing flat surfaces are extracted from the restored point cloud after preprocessing.
[0179] Next, when the user performs an operation to instruct automatic interior / exterior determination (Yes in ST103), the processor 23 executes the automatic interior / exterior determination process (ST104). In the automatic interior / exterior determination process, it is determined whether the segments extracted in the segmentation process correspond to external or internal components of the room, and the segments are divided into internal segments representing internal components and external segments representing external components.
[0180] Next, when the user performs an internal / external sorting operation (Yes in ST105), the processor 23 executes a manual internal / external setting process (ST106). In the manual internal / external setting process, the selected segment is sorted into an internal segment representing an internal component and an external segment representing an external component, according to the user's operation (internal / external sorting operation) specifying whether the segment is an internal component or an external component.
[0181] Next, the processor 23 performs modeling processing (ST107). In the modeling processing, a three-dimensional model (room model) representing the external components of the room is generated based on the external segments obtained in the internal / external sorting processing (automatic internal / external determination processing, manual internal / external setting processing).
[0182] Here, depending on whether or not an operation to instruct automatic inside / outside determination is performed (Yes, No in ST103) and whether or not an inside / outside sorting operation is performed (Yes, No in ST105), there are four patterns: one in which the automatic inside / outside determination process and the manual inside / outside setting process are performed in sequence; one in which only the automatic inside / outside determination process is performed; one in which only the manual inside / outside setting process is performed; and one in which neither the automatic inside / outside determination process nor the manual inside / outside setting process is performed. Furthermore, in the pattern in which the automatic inside / outside determination process and the manual inside / outside setting process are performed in sequence, the manual inside / outside setting process is performed to correct the result of the automatic inside / outside determination process.
[0183] Next, we will explain the preprocessing performed on server 2 (ST101 in Figure 7). Figure 8 is a flowchart showing the preprocessing procedure.
[0184] In the preprocessing stage, noise reduction (ST201), downsampling (ST202), normal estimation (ST203), flatness estimation (ST204), floor surface estimation (ST205), and coordinate axis setting (ST206) are performed sequentially.
[0185] In the noise reduction process (ST201), the processor 23 removes noise (isolated points) included in the restored point cloud.
[0186] In the downsampling process (ST202), the processor 23 performs downsampling of the point cloud using a voxel grid filter. The size of the voxel grid filter (voxel size) is set to, for example, 0.01m.
[0187] In the normal estimation process (ST203), the processor 23 estimates the normal vector (point normal) of each point included in the restored point cloud.
[0188] In the flatness estimation process (ST204), the processor 23 estimates the flatness of each point included in the restored point cloud. At this time, when the eigenvalues of the covariance matrix of neighboring points are set in descending order e0 > e1 > e2, the flatness for each point is calculated as (e1 - e2) / e0.
[0189] In the floor surface estimation process (ST205), the processor 23 estimates the horizontal plane that will be the floor of the room from the restored point cloud. Specifically, first, points where the dot product of the point normal estimated in the normal estimation process and the unit vector [0,0,1] in the z-axis direction exceeds a predetermined threshold (e.g., 0.9) are extracted as the horizontal plane (sub-point cloud). Next, a histogram of the z-coordinates of each point in the point cloud is obtained, and the first and second most important peaks are detected as corresponding to the floor and ceiling of the room, and the one with the smaller z-coordinate value is determined as the floor height. Note that the vertical direction of the point cloud can be set based on the detection data at the time of shooting by the IMU installed in the user terminal 1, so the z-axis is known.
[0190] In the coordinate axis setting process (ST206), the processor 23 sets the x and y axes of the three-dimensional Cartesian coordinate system (x, y, and z axes) for the restored point cloud. The z axis is perpendicular to the floor surface obtained in the floor surface estimation process, and the x and y axes are aligned with the walls (vertical planes) of the room. At this time, with respect to the direction of the reference maximum vertical plane, estimation considering errors is performed using the RANSAC (Random Sample Consensus) method, which eliminates the influence of outliers. Furthermore, translation and rotation processing is performed on the entire restored point cloud so that the floor surface becomes z=0, the center of the restored point cloud becomes the origin, and the maximum vertical plane becomes the x-axis direction.
[0191] Next, we will explain the segmentation process performed on server 2 (ST102 in Figure 7). Figure 9 is a flowchart showing the procedure for the segmentation process. Figure 10 is an explanatory diagram showing the detection status of line segment.
[0192] In the segmentation process, multiple segments (sub-point clouds) representing flat surfaces are extracted from the reconstructed point cloud.
[0193] As shown in Figure 9, the segmentation process sequentially performs point cloud division using the region expansion method (ST301) and linear segment detection (ST302).
[0194] In the point cloud division process using the domain expansion method (ST301), the reconstructed point cloud is divided into multiple segments (sub-point clouds) representing flat surfaces using the domain expansion method. In the domain expansion method, the process of searching for neighboring points starts from the seed point, and the found neighboring points are added to the region of the segment. In this embodiment, when setting the initial point and the seed point for domain expansion, flatness is used as the selection criterion instead of curvature. This makes it easier to select flat and highly isotropic points, and prevents corners such as the boundary between the ceiling and the wall from being selected as the starting point. The search range for neighboring points may also be set to, for example, 0.05 m. Furthermore, as a condition for adding neighboring points to the region of the segment, the angle between the point normals of the initial point and the neighboring points may be set to be less than a threshold (for example, 30°).
[0195] Specifically, each surface of the external components of a room (floor, ceiling, wall, beam, pillar, step, etc.) and each surface of the internal components of the room (furniture such as sofas, desks, chairs, and shelves, etc.) are divided into separate segments (see FIG. 2(B)). At this time, a plurality of segments with different positions and orientations are extracted. On the other hand, point groups that do not form a flat surface, that is, point groups representing surfaces with high curvature or edges (for example, the legs of desks and chairs) are not extracted as segments.
[0196] In the linear segment detection process (ST302), the processor 23 detects linear segments that appear at the boundaries between two surfaces and the like (see FIG. 10). The detected linear segments are excluded from the external segments that are modeling targets.
[0197] Specifically, a rectangular parallelepiped (three-dimensional bounding box) with the minimum volume that encloses the segment is set, and the lengths L of the three sides of the rectangular parallelepiped 0 , L 1 , L 2 (L 0 ≧L 1 ≧L 2 ), when the length L of the side with the intermediate length is less than a threshold value (for example, 0.05 m), the segment is determined to be a linear segment. In addition, the length L of the side with the intermediate length 1 1 The ratio L of the length of the shortest side to 2 2 / L 1 is less than a threshold value (for example, 0.5), the segment is determined to be a linear segment.
[0198] Next, the automatic inside / outside determination process (ST104 in FIG. 7) performed by the server 2 will be described. FIG. 11 is a flowchart showing the procedure of the automatic inside / outside determination process. FIG. 12 is an explanatory diagram showing the state of the floating surface determination process. FIG. 13 is an explanatory diagram showing the maximum inter-surface distance. FIG. 14 is an explanatory diagram showing an overview of the bounding box determination process.
[0199] In the automatic interior / exterior determination process, each segment extracted in the segmentation process is determined to be either an exterior or interior component of the room, and each segment is then classified into an exterior segment representing the exterior components of the room and an interior segment representing the interior components of the room.
[0200] As shown in Figure 11, the automatic inside / outside determination process sequentially performs the following steps: floating surface determination process (ST401), segment integration process (ST402), maximum inter-surface distance acquisition process (ST403), and bounding box determination process (ST404).
[0201] In the floating surface determination process (ST401), the processor 23 determines whether each segment extracted in the segmentation process corresponds to a floating surface that is not in contact with the ceiling, walls, or floor. Segments that correspond to floating surfaces are excluded from the external segments to be modeled and are treated as internal segments.
[0202] Here, since the ceiling, walls, and floor, as external components, are in contact with each other, if we group adjacent segments together, the external segments representing external components can be grouped into one group. On the other hand, segments not included in this group become floating surfaces that are not in contact with the ceiling, walls, and floor, and these floating surfaces are determined to be internal segments. Furthermore, since the ceiling, walls, and floor have relatively large areas, the total number of points included in the group of segments representing external components will be large. Conversely, since the segments corresponding to floating surfaces have relatively small areas, the total number of points included in the group of segments corresponding to floating surfaces will be small.
[0203] Therefore, in this embodiment, it is assumed that the group with the highest score is composed of segments representing external components (ceiling, walls, and floor), and segments not included in the group with the highest score are determined to be segments corresponding to floating surfaces.
[0204] Specifically, first, the minimum distance between points in each segment is taken as the distance between segments, and the distance between segments is calculated for all combinations of segments. Next, combinations of segments whose distance between segments is within a threshold (e.g., 0.05 m) are determined to be in the same group. Then, the scores of each segment in each group are added together to calculate the group score. Finally, segments that are not included in the group with the highest score are determined to be the floating surfaces.
[0205] In the example shown in Figure 12, during the floating surface determination process, segments representing external components such as floors, ceilings, walls, beam-like steps, columnar protrusions, sofas, and shelves are grouped together. On the other hand, segments representing some internal components such as desks (tabletops) and chairs (seats and backs) form a separate group from the segments representing external components. Here, the group of segments representing external components is identified as such based on the number of points in the point cloud, as it contains a large number of points.
[0206] Segments corresponding to floating surfaces are excluded from subsequent segment integration processing, maximum inter-surface distance acquisition processing, and bounding box determination processing.
[0207] In the segment integration process (ST402), the processor 23 integrates segments that are assumed to constitute the same plane.
[0208] Specifically, for each combination of segment A and segment B which has more points than segment A, the plane equation for the plane represented by segment B is determined, and the distance from each point in segment A to the plane represented by segment B is calculated. Next, points where the calculated distance is less than or equal to a predetermined threshold (e.g., 0.075 m) are counted. Then, if such points constitute a predetermined percentage (e.g., 75%) or more, the two segments A and B are determined to constitute the same plane, and the two segments A and B are merged. This process is carried out for all combinations of segments.
[0209] In the maximum inter-face distance acquisition process (ST403), the processor 23 determines the maximum distance between segments in each of the x, y, and z axes as the maximum inter-face distance. The maximum inter-face distance is used as an indicator of the size of the room.
[0210] In the example shown in Figure 13, the entrance door is open and the outside of the room is visible, causing some segments to protrude from the room. Similarly, even when the outside of the room is visible through a transparent windowpane, some segments still protrude from the room. In such cases, if a 3D bounding box is set to enclose all segments, the size of that 3D bounding box will differ significantly from the actual size of the room, making it unsuitable as an indicator of room size.
[0211] On the other hand, the maximum inter-surface distance represents the distance between opposing surfaces of the room's external components (floor, ceiling, walls, etc.), and serves as an indicator of the overall size of the room, even if some segments extend beyond the room's boundaries.
[0212] Specifically, first, the segments are grouped according to which axis direction (x, y, or z) the normal direction of each segment is closest to. Specifically, first, the plane equation for the plane represented by each segment is determined. Next, the dot product of the normal direction of the plane represented by each segment and the x, y, and z axes is calculated, and the axis to which the absolute value of the dot product is maximized becomes the group to which that segment belongs. Next, for each group of x, y, and z axes, the centroid of all segments included in that group is calculated.
[0213] Next, the distance between the furthest centroids of the segments included in each group of the x, y, and z axes is determined as the maximum inter-plane distance for each of the x, y, and z axes. For example, in the case of the z-axis group, the segment with the maximum z-coordinate value and the segment with the minimum z-coordinate value are found, and the difference in the z-coordinate values of these two segments is the maximum inter-plane distance for the z axis. The maximum inter-plane distance can be determined similarly for the x-axis group and the y-axis group.
[0214] The x-axis group includes segments close to the vertical plane, where the segment with the maximum x-coordinate value and the segment with the minimum x-coordinate value represent two opposing walls, and the maximum distance between these walls in the x-axis direction represents the depth of the room. Similarly, the y-axis group includes segments close to the vertical plane, where the segment with the maximum y-coordinate value and the segment with the minimum y-coordinate value represent two opposing walls, and the maximum distance between these walls in the y-axis direction represents the width of the room. The z-axis group includes segments close to the horizontal plane, where the segment with the maximum z-coordinate value represents the ceiling, and the segment with the minimum z-coordinate value represents the floor, and the maximum distance between these walls in the z-axis direction represents the ceiling height of the room.
[0215] In the bounding box determination process (ST404), the processor 23 determines whether a segment is an external segment representing an external component of the room or an internal segment representing an internal component of the room, based on the size of the segment relative to the overall size of the room. External components (floors, ceilings, walls, beams, columns, etc.) are assumed to have sufficient length in one of the x, y, or z axes relative to the overall size of the room. Therefore, segments that are sufficiently long relative to the dimensions of the room are determined to be external segments representing external components. On the other hand, segments that are not sufficiently long relative to the dimensions of the room, such as sofas and shelves, are not considered external segments and are determined to be internal segments representing internal components.
[0216] Specifically, as shown in Figure 14, a three-dimensional bounding box is defined for each segment, with edges parallel to the x, y, and z axes. Next, the size of the three-dimensional bounding box for each segment is compared with the maximum inter-face distances in the x, y, and z axes to determine whether the segment is an external or internal segment. At this time, the length L of the longest side of the three-dimensional bounding box is determined. max The ratio (L) of the maximum inter-plane distance D along the axis parallel to its longest side max If the ratio (L) exceeds a predetermined threshold (e.g., 0.5), the segment is determined to be an external segment. maxIf / D) is below the threshold, the segment is determined to be an internal segment. In the example shown in Figure 14, the target segment is long in the y-axis direction, and the longest side of the 3D bounding box is parallel to the y-axis.
[0217] Next, we will explain the modeling process performed on server 2 (ST107 in Figure 7). Figure 15 is a flowchart showing the steps of the modeling process. Figure 16 is an explanatory diagram showing the generation status of the initial surface and candidate surfaces. Figure 17 is an explanatory diagram showing an overview of the coverage score and distance penalty. Figures 18, 19, and 20 are explanatory diagrams showing the procedure for calculating the cost of dividing the same region.
[0218] In the modeling process, a 3D model (polygon mesh model) of the room is generated based on the segments to be modeled (external segments) that represent external components obtained through the internal / external sorting process (automatic internal / external determination process, manual internal / external setting process).
[0219] As shown in Figure 15(A), the modeling process sequentially performs candidate surface generation (ST501), candidate surface index calculation (ST502), and optimization (ST503).
[0220] In the candidate surface generation process (ST501), the processor 23 generates candidate surfaces that will be candidates for constituent surfaces of the 3D model based on the segments to be modeled (external segments) obtained in the internal / external sorting process.
[0221] In the candidate surface index calculation process (ST502), the processor 23 calculates index values (coverage score, distance penalty, same-domain partitioning cost) for each candidate surface, which serve as indicators for deciding whether or not to adopt a candidate surface obtained in the candidate surface generation process, that is, when to adopt a candidate surface as a constituent surface of the 3D model. The index values represent the degree of appropriateness of the candidate surface as a constituent surface of the 3D model, according to the degree of point cloud allocation (distribution of point clouds), etc.
[0222] In the optimization process (ST503), the processor 23 selects appropriate candidate faces from among the candidate faces obtained in the candidate face generation process, based on the index values obtained in the candidate face index calculation process, to be used as constituent faces of the 3D model. In the optimization process, an optimization solver is used to solve the question of whether or not to adopt a candidate face, that is, whether or not to adopt each candidate face as a constituent face of the 3D model, as an optimization problem, particularly a 0-1 integer programming problem, and the adoption status of a candidate face is determined by the solution vectors 0 and 1.
[0223] As shown in Figure 15(B), in the candidate surface generation process (ST501 in Figure 15(A)), the processor 23 first sets a three-dimensional bounding box that surrounds the entire restored point cloud (ST601). The three-dimensional bounding box is circumscribing the entire restored point cloud, and its three sides are parallel to the x, y, and z axes, respectively.
[0224] Next, the processor 23 estimates the initial faces that will become candidate faces based on the external segments to be modeled, which represent the external components (ST602). Specifically, first, the plane equations for the planes represented by each segment are obtained using the RANSAC method. Next, the intersection points (intersection lines) where the bounding box surrounding the entire restored point cloud intersects with the planes represented by each segment are obtained. Then, the initial faces of polygons are created by connecting these intersection points.
[0225] Next, the processor 23 divides the initial faces at the points where the initial faces intersect (intersection lines) for all combinations of initial faces to find candidate faces (face division process) (ST603). In the example shown in Figure 16, initial faces are generated that include the faces of external components such as the floor, ceiling, walls, beam-like steps, and columnar protrusions, and a large number of candidate faces are generated by dividing these initial faces at the points where they intersect. Note that some candidate faces include faces of external components, while others do not.
[0226] In this candidate surface generation process, initial surfaces are generated by extending the planes represented by the segments to the extent that they intersect with the 3D bounding box that surrounds the entire restored point cloud. Candidate surfaces are then generated by dividing these initial surfaces at the points where they intersect.
[0227] As shown in Figure 15(C), in the candidate surface index calculation process (ST502 in Figure 15(A)), the processor 23 first calculates a coverage score (point cloud distribution index value) for each candidate surface (ST701).
[0228] Specifically, as shown below, the coverage score Score(f) is calculated as the ratio of the area of the alpha shape formed by the point cloud on the candidate surface f to the area of the candidate surface f. The alpha shape is a polygonal demarcation of the area where points exist that are included in the segment (sub-point cloud) that forms the basis of the initial surface before the candidate surface is subdivided and generated (see Figure 17(A)).
[0229]
[0230] Here, the coverage score is an index that represents the degree of appropriateness of a candidate surface as a constituent surface of a 3D model, depending on whether or not a sufficient number of points included in the segment are present on the candidate surface. If a sufficient number of points included in the segment are present on the candidate surface and the candidate surface is appropriate as a constituent surface of the 3D model, the coverage score will be positive, and the value of the coverage score will increase according to the degree of appropriateness. On the other hand, if a sufficient number of points included in the segment are not present on the candidate surface and the candidate surface is inappropriate as a constituent surface of the 3D model, the coverage score will be negative, and the value of the coverage score will decrease according to the degree of inappropriateness.
[0231] Next, the processor 23 calculates a distance penalty (distance index value) for each candidate face (ST702).
[0232] Specifically, as shown below, for each candidate face f, the point closest to each vertex of the candidate face is determined from among the points included in the segment that forms the basis of the initial face before the candidate face is subdivided and generated. The average distance between that vertex and the points included in the segment is then calculated as the distance penalty Penalty(f).
[0233]
[0234] In the example shown in Figure 17(B), the point closest to each vertex v1, v2, v3, v4 of the candidate face is determined from among the points included in the segment that forms the basis of the initial face before the candidate face is subdivided and generated. The average of the distances d1, d2, d3, d4 between these vertices v1, v2, v3, v4 and the points included in the segment is calculated as the distance penalty Penalty(f).
[0235] Here, the distance penalty is an indicator of the degree of unsuitability of a candidate face as a constituent surface of a 3D model. If a candidate face is far from the segment that forms the basis of the initial face before it is subdivided and generated, the candidate face is assumed to be unsuitable as a constituent surface of the 3D model, and therefore the distance penalty value will be large.
[0236] Next, the processor 23 calculates the same-domain partitioning cost (domain partitioning index value) for a combination of two candidate faces that are adjacent to each other (ST703). The same-domain partitioning cost is an index that represents the degree of inappropriateness in the state in which the region of one face is inappropriately partitioned when both candidate faces are adopted as constituent faces of the 3D model.
[0237] Specifically, as shown below, there are two candidate surfaces f 1 , f 2 Regarding the combination, the candidate surface f 1 , f 2 The scores RScore(f1) and RScore(f2) for each region are calculated, and based on these scores RScore(f1) and RScore(f2), the cost of partitioning the same region, PairCost(f1,f2), is calculated. Also, the candidate surface f(f 1 , or f 2 Regarding the combination of a candidate surface f' that is coplanar with the candidate surface f', the point cloud ratio r represents the proportion of points located on the candidate surfaces f' and f' among the points included in the segment (subpoint cloud) that forms the basis of the initial surface before the candidate surfaces f' and f' are divided and generated. f ,r f ' is calculated, and its point cloud ratio r f ,r f Based on this, the score RScore(f) is calculated.
[0238]
[0239] In the candidate surface generation process (ST501 in Figure 15(A)), as shown in Figure 18, the initial surfaces are divided by a single intersection line where the two initial surfaces intersect, generating four candidate surfaces that are tangent to the intersection line. Then, in the optimization process (ST503 in Figure 15(A)), candidate surfaces to be adopted as constituent surfaces of the 3D model are selected. At this time, there are cases where two candidate surfaces are adopted from the four candidate surfaces, and cases where none of the four candidate surfaces are adopted.
[0240] As shown in Figure 19(A), there are two candidate surfaces f tangent to the intersection line. 1 , f 2 When considering combinations of these, the two candidate surfaces f 1 , f 2 The segments that form the basis of the initial faces before each of them is divided and generated are the two candidate faces f 1 , f 2 If it exists spanning across, there are two candidate surfaces f 1 , f 2 It is desirable that these be selected together as the constituent surfaces of the 3D model. Therefore, in this embodiment, in such a case, two candidate surfaces f 1 , f 2 The cost of partitioning the same domain is set so that a reward is given for each combination.
[0241] On the other hand, unlike the example shown in Figure 19(A), even though the segment that forms the basis of the initial surface exists so as to span two candidate surfaces, there are two other candidate surfaces f 1 , f 2 It is undesirable for a set of these to be selected as constituent surfaces of a 3D model. Therefore, in this embodiment, in such cases, two candidate surfaces f 1 , f 2 The cost of partitioning the same domain is set to impose a penalty on combinations of the same domain.
[0242] Specifically, as shown in Figure 19(B), the point cloud ratio r represents the proportion of points located on the candidate surface f among the points included in the segment (sub-point cloud) that forms the basis of the initial surface before the candidate surface f is divided and generated. f This is calculated. This point cloud ratio r fAs shown in Figure 19(C), this is determined for each candidate surface f and its coplanar candidate surface f'. Next, the smaller of the point cloud ratios rf and rf' for each candidate surface f and f' is compared with a predetermined threshold, and a reward or penalty is given to the combination of the two candidate surfaces f and f'.
[0243] Also, two candidate surfaces f 1 , f 2 When calculating the cost of partitioning the same region for a given combination of two candidate surfaces f 1 , f 2 Case distinctions are made depending on whether or not the two points are coplanar.
[0244] The example shown in Figure 20(A) involves two candidate surfaces f 1 , f 2 This is the case when the two surfaces are coplanar. In this case, the candidate surface f (= f 1 From the point cloud ratio rf (=0.4), the candidate surface f' (=f 2 ) Point cloud ratio r f ' (= 0.2) is small, and this r f Since the value exceeds the threshold (0.05), a reward is given. Note that the threshold is not limited to 0.05.
[0245] On the other hand, the example shown in Figure 20 (B-1) has two candidate surfaces f 1 , f 2 This is the case when the two planes are not coplanar. In this case, as shown in Figures 20 (B-2) and (B-3), the two candidate planes f 1 , f 2 Let's assume two candidate surfaces f' that are coplanar to each of the two, and then consider these two candidate surfaces f(f 1 , f 2 Consider whether there is a penalty for the combination of ), f'.
[0246] In the example shown in Figure 20 (B-2), the candidate surface f (= f 2 ) Point cloud ratio r of candidate surface f' which is coplanar with ) f From (=0.4), the point cloud ratio r of candidate surface f is f (=0.2) is small, and this r f Since (=0.2) is greater than or equal to the threshold (0.05), a penalty is imposed. On the other hand, in the example shown in Figure 20 (B-3), candidate surface f (=f1 ) Point cloud ratio r f (=0.2) The point cloud ratio r of candidate surface f' that is coplanar with candidate surface f is f ' (= 0.01) is small, and this r f Since '(=0.01) is below the threshold (0.05), no penalty is imposed.
[0247] As shown in Figure 15(D), in the optimization process (ST503 in Figure 15(A)), the processor 23 first sets the decision variable (ST801). Specifically, as shown below, the decision variable x is set to 1 when each candidate face is adopted and 0 when it is not adopted. f This will be set.
[0248]
[0249] Next, the processor 23 sets the objective function (ST802). Specifically, as shown below, the objective function E is set using the index values for each candidate surface obtained in the candidate surface index calculation process, namely the coverage score Score(f), distance penalty Penalty(f), and identical region partitioning cost PairCost(f1,f2).
[0250]
[0251] Next, the processor 23 sets the constraints for the integer programming problem (ST803). The constraints are set so that the 3D model is watertight and generated as a manifold. Specifically, the constraints are set so that the candidate faces f adjacent to each edge are either 0 or 2, as shown below.
[0252]
[0253] Next, the processor 23 executes a general-purpose optimization solver to solve 0-1 integer programming problems based on the objective function and constraints, and selects appropriate candidate faces as constituent faces of the 3D model (ST804). At this time, a solution vector is obtained that satisfies the constraints while minimizing the objective function based on the index values for each candidate face (coverage score, distance penalty, same-domain partitioning cost).
[0254] Next, the processor 23 creates a three-dimensional model of the room (polygon mesh model) (see Figure 4) based on the candidate faces selected as constituent faces of the three-dimensional model (ST805).
[0255] Next, we will explain the basic screen 31 (initial screen) displayed on the user terminal 1. Figure 21 is an explanatory diagram showing the basic screen 31.
[0256] The basic screen 31 is provided with a 3D viewer section 32 and an operation panel section 33.
[0257] The 3D viewer unit 32 is provided with a mode display selection unit 34 and an image display unit 35. The mode display selection unit 34 is provided with a "point cloud" tab 37, a "segment" tab 38, and a "model" tab 39. Tabs 37, 38, and 39 are switched according to the progress of the 3D model generation process (point cloud input process, segmentation process, and modeling process), and the image displayed in the image display unit 35 (reconstructed point cloud, segment, and 3D model) changes accordingly. The user can also switch to the respective mode screen by operating tabs 37, 38, and 39.
[0258] The control panel 33 is equipped with a "Modeling" button 41 and a "Settings" button 42. Figure 21 shows the state when the "Modeling" button 41 is selected. When the user operates the "Settings" button 42, the system transitions to the settings screen 91 (see Figure 27).
[0259] Furthermore, the control panel 33 is equipped with a "File Import" button 43, a "Segmentation Execution" button 44, an automatic internal / external determination instruction unit 45, an internal / external sorting operation unit 46, a "Modeling Execution" button 47, and a "File Export" button 48.
[0260] When the user operates the "File Import" button 43, the restored point cloud file stored in the storage unit 22 is input, and the system transitions to the restored point cloud display screen 61 (see Figure 22). The restored point cloud file is generated by the 3D restoration process and stored in the storage unit 22. Alternatively, when the "File Import" button 43 is operated, a screen (not shown) may be displayed in which the user can select one of several restored point cloud files stored in the storage unit 22.
[0261] Next, we will explain the restored point cloud display screen 61 that is displayed on the user terminal 1. Figure 22 is an explanatory diagram showing the restored point cloud display screen 61.
[0262] On the restored point cloud display screen 61, the restored point cloud stored in the input file is displayed on the image display unit 35. The restored point cloud is converted into a two-dimensional image and displayed on the image display unit 35 based on the specified viewpoint. In the example shown in Figure 22, the viewpoint is set to the outside of the room, and the interior of the room is visible by making the wall and ceiling surfaces on the side closer to the viewpoint transparent. However, the wall and ceiling surfaces on the side closer to the viewpoint can also be made opaque.
[0263] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the restored point cloud. That is, the user can observe the entire restored point cloud in detail by changing the position and angle of the viewpoint relative to the restored point cloud (for example, by dragging and dropping).
[0264] On the restored point cloud display screen 61, when the user operates the "Execute Segmentation" button 44, preprocessing and segmentation processing begin. Once preprocessing and segmentation processing are complete, the screen transitions to the segment display screen 71 (see Figure 23).
[0265] Next, the segment display screen 71 displayed on the user terminal 1 will be described. Figure 23 is an explanatory diagram showing the segment display screen 71. Figure 24 is an explanatory diagram showing the state in which the results of the automatic inside / outside determination process are reflected on the segment display screen 71.
[0266] As shown in Figure 23, the segment display screen 71 displays the restored point cloud, that is, the restored point cloud in the segment division state, on the image display unit 35, which reflects the extraction results from the segmentation process. The image display unit 35 displays the restored point cloud in the segment division state as a two-dimensional image based on the specified viewpoint. In the example shown in Figure 23, the viewpoint is set to the outside of the room, and the interior of the room is visible by making the wall and ceiling surfaces on the side closer to the viewpoint transparent. However, the wall and ceiling surfaces on the side closer to the viewpoint can also be made opaque.
[0267] Furthermore, in the reconstructed point cloud in segmented state, each segment is displayed in a different color. That is, each point in the reconstructed point cloud is colored according to the color of the segment. Specifically, each segment is displayed in a different chromatic color (a color other than gray). On the other hand, point clouds that were not extracted as segments, such as the legs of a desk or chair, are displayed in achromatic colors (gray). This allows the user to easily check the generation status of segments that form the basis of the constituent surfaces of the 3D model.
[0268] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the reconstructed point cloud in the segmented state. That is, the user can observe the entire reconstructed point cloud in the segmented state in detail by changing the position and angle of the viewpoint relative to the reconstructed point cloud in the segmented state (for example, by dragging and dropping).
[0269] The automatic inside / outside determination instruction unit 45 is equipped with a toggle switch 51 and a slider 52.
[0270] When the user turns on the toggle switch 51, the automatic inside / outside determination process is executed. Once the automatic inside / outside determination process is complete, the restored point cloud of the segment division state, reflecting the results of the automatic inside / outside determination process, is displayed on the image display unit 35, as shown in Figure 24.
[0271] In the restored point cloud reflecting the results of the automatic internal / external determination process, external segments representing external components and internal segments representing internal components are displayed with different brightness levels. Specifically, external segments, i.e., segments targeted for modeling, are highlighted using high-brightness chromatic colors. Internal segments, i.e., segments not targeted for modeling, are suppressed using low-brightness chromatic colors. This allows users to easily check the generation status of segments representing external components that form the basis of the 3D model's surfaces.
[0272] The user can change the segment size ratio used in the automatic inside / outside determination process by operating the slider 52. In the bounding box determination process of the automatic inside / outside determination process, it is determined whether a segment is an external segment or not based on the ratio of the length of the longest side of the bounding box set for the segment to the maximum inter-face distance, and the segment size ratio is used as the threshold for this determination. When the user operates the slider 52 and changes the segment size ratio, the automatic inside / outside determination process is re-executed, and the color coding state of the restored point cloud of the segment division state displayed on the image display unit 35 changes according to the determination result.
[0273] The internal / external sorting operation unit 46 can perform internal / external sorting operations to specify whether the segments displayed on the image display unit 35 belong to internal or external components. When a user selects a segment on the restored point cloud of the segment division state displayed on the image display unit 35 and then operates the internal / external sorting operation unit 46, a manual internal / external setting process is executed for the selected segment. In the manual internal / external setting process, the selected segment is sorted into segments that represent external components of a room and are subject to modeling, and segments that represent internal components of a room and are not subject to modeling, according to the user's internal / external sorting operation.
[0274] In the image display unit 35, the user can visually determine whether there are errors in the settings of segments to be modeled representing external components of a room and segments not to be modeled representing internal components of a room, in the restored point cloud of segment division state reflecting the results of the automatic interior / exterior determination process, and select the segments with errors by operating the input device 14 (for example, by clicking the mouse). At this time, the selected segments are highlighted in the image display unit 35. Specifically, the selected segments are highlighted by blinking (change in brightness).
[0275] The internal / external distribution operation unit 46 is equipped with an "Undo" button 55, a "Redo" button 56, an "Add" button 57, and a "Delete" button 58. When the user operates any of the "Undo" button 55, "Redo" button 56, "Add" button 57, or "Delete" button 58, the manual internal / external setting process corresponding to that operation is executed.
[0276] Specifically, if a user selects a segment that is not set to be modeled (an internal segment) and then operates the "Add" button 57, the selected segment will be added to the modeling target and changed to an external segment. Also, if a user selects a segment that is set to be modeled (an external segment) and then operates the "Delete" button 58, the selected segment will be removed from the modeling target and changed to an internal segment. Furthermore, if a user operates the "Undo" button 55, they can revert to the settings before the operation of the "Add" button 57 or the "Delete" button 58. Also, if a user operates the "Redo" button 56, they can revert to the settings before the operation of the "Undo" button 55.
[0277] Here, when multiple segments are selected by the user through user operation on the image display unit 35, if the user operates the internal / external distribution operation unit 46, the settings of the selected multiple segments are changed all at once in the manual internal / external setting process. At this time, for example, multiple segments can be selected sequentially by operating the input device 14 (e.g., keyboard and mouse shift-click operations). Also, by inputting a bounding box through operation of the input device 14 (e.g., drag operation), multiple segments contained in that bounding box can be selected simultaneously.
[0278] Furthermore, by turning on the toggle switch 51 of the automatic indoor / outdoor determination instruction unit 45, the user can operate the indoor / outdoor distribution operation unit 46 after the automatic indoor / outdoor determination process has been executed. In this case, manual indoor / outdoor setting processing is performed on the segment in which the results of the automatic indoor / outdoor determination process have been reflected, in accordance with the user's indoor / outdoor distribution operation. This allows the user to manually correct any errors in the results of the automatic indoor / outdoor determination process.
[0279] On the other hand, the user can also operate the interior / exterior sorting operation unit 46 while the toggle switch 51 of the automatic interior / exterior determination instruction unit 45 remains in the off state. In this case, manual interior / exterior setting processing is performed on the initial segments according to the user's interior / exterior sorting operation. In the initial state, all segments are set as modeling targets, i.e., external segments representing external components of a room. The user can exclude segments that correspond to internal components from the modeling targets by selecting segments that correspond to internal components but are set as external segments and then operating the "delete" button 58.
[0280] In this embodiment, the user can visually identify segments to be modeled and segments to be excluded from modeling and manually change the settings. This improves the accuracy of the generated 3D model (room model). Furthermore, in this embodiment, the user can set segments to be modeled and segments to exclude on a segment-by-segment basis. Therefore, compared to, for example, a case where the user specifies a range using a bounding box for the restored point cloud and sets segments to be modeled and segments to exclude, the working time can be significantly reduced.
[0281] On the segment display screen 71 (see Figures 23 and 24), when the user operates the "Execute Modeling" button 47, the 3D modeling process begins. Once the 3D modeling process is complete, the screen transitions to the 3D model display screen 81 (see Figures 25 and 26).
[0282] Furthermore, if the user operates the "Execute Modeling" button 47 without operating the toggle switch 51 of the automatic interior / exterior determination instruction unit 45 or the interior / exterior sorting operation unit 46, the interior / exterior sorting process (automatic interior / exterior determination process and manual interior / exterior setting process) is not executed, and the 3D modeling process starts. In this case, since the interior / exterior sorting process is omitted, a 3D model is generated that includes both the exterior and interior components of the room.
[0283] Next, we will explain the 3D model display screen 81 displayed on the user terminal 1. Figure 25 is an explanatory diagram showing the 3D model display screen 81 with the interpolation region displayed. Figure 26 is an explanatory diagram showing the 3D model display screen 81 with the error region displayed.
[0284] On the 3D model display screen 81, the 3D model (room model) generated by the 3D modeling process is displayed on the image display unit 35. The image display unit 35 displays the 3D model as a 2D image based on the specified viewpoint. In the example shown in Figures 25 and 26, the viewpoint is set to the outside of the room, and the interior of the room is visible by making the walls and ceiling on the side closer to the viewpoint transparent. However, the walls and ceiling on the side closer to the viewpoint can also be made opaque.
[0285] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the three-dimensional model. That is, the user can observe the entire three-dimensional model in detail by changing the position and angle of the viewpoint relative to the three-dimensional model (for example, by dragging and dropping).
[0286] As shown in Figure 25, on the 3D model display screen 81, images 82, 83, and 84 of the interpolated regions are displayed on the 3D model in the image display unit 35. Specifically, the images 82, 83, and 84 of the interpolated regions are highlighted with a color that is more conspicuous than other parts (a highly eye-catching color). In addition, the images 82, 83, and 84 of the interpolated regions may be drawn in different drawing patterns (color, pattern, etc.) depending on the reason for interpolation. In the example shown in Figure 25, image 82 of the interpolated region relating to point cloud loss due to concealment by furniture (sofas and shelves), image 83 of the interpolated region relating to point cloud-less areas due to openings (doorways), and image 84 of the interpolated region relating to point cloud loss due to missed shooting are displayed. This allows the user to easily check the interpolation status.
[0287] Furthermore, when the user performs an operation to select images 82, 83, and 84 of the interpolation region (for example, a mouseover operation) in the image display unit 35, the interpolation reason for the selected interpolation region is displayed. Specifically, an interpolation reason display unit 85 (tooltip) is displayed with text indicating the interpolation reason, such as "Interpolation of openings such as doors," "Interpolation of defects due to occlusion," or "Interpolation of defects due to missed image capture." This allows the user to easily confirm the interpolation reason. For example, the user can confirm that point cloud defects due to missed image capture have been appropriately interpolated. Also, the user can confirm that openings such as doorways have been incorrectly interpolated even though they are not point cloud defects.
[0288] As shown in Figure 26, on the 3D model display screen 81, the image display unit 35 displays an image 86 of the error region on the 3D model. Specifically, the image 86 of the error region is highlighted with a color that is more conspicuous (a highly visible color), similar to the images 82, 83, and 84 of the interpolation region. In the example shown in Figure 26, the columnar protrusion at the boundary between the two walls is not reproduced in the 3D model, and this area is highlighted as an error region. This allows the user to easily check the error status of the 3D model relative to the restored point cloud.
[0289] When the user operates the "Settings" button 42 on the 3D model display screen 81, the screen transitions to the settings screen 91 (see Figure 27).
[0290] Furthermore, when the user operates the "File Export" button 48 on the 3D model display screen 81, the generated 3D model (room model) is output in a predetermined file format. The user may choose the file format as appropriate.
[0291] In this embodiment, a three-dimensional model (room model) representing the external components of the room is generated. However, the output file may also contain, in addition to the three-dimensional model, segments representing the external components of the room and segments representing the internal components of the room, in an identifiable manner. This allows the segments representing the internal components to be used later for processes such as object recognition or three-dimensional model replacement.
[0292] Next, we will explain the settings screen 91 displayed on the user terminal 1. Figure 27 is an explanatory diagram showing the settings screen 91.
[0293] The settings screen 91 allows the user to configure how the 3D model (room model) is displayed on the 3D model display screen 81 (see Figures 25 and 26). The settings screen 91 includes an interpolation region display setting unit 92 and an error region display setting unit 93.
[0294] The interpolation area display setting unit 92 is provided with checkboxes 95 and display color boxes 96 for each interpolation reason item (opening, missed shot, concealment). By operating the checkboxes 95, the user can switch the display of the interpolation area for each interpolation reason item. In addition, by operating the display color box 96, a color selection box (not shown) will be displayed, and by selecting a color in that color selection box, the user can specify the display color of the interpolation area for each interpolation reason item.
[0295] The error area display setting unit 93 is provided with an input box 97 and a display color box 98. In the input box 97, the user can input a threshold value as a detection condition in the error area detection process. Areas where the error is greater than the input threshold value are displayed as an error area image 86 (see Figure 26). In addition, by operating the display color box 98, a color selection box (not shown) is displayed, and the user can specify the display color of the error area by selecting a color in that color selection box.
[0296] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments.
[0297] The 3D model generation apparatus, 3D model generation method, and 3D model generation program according to this disclosure have the effect of generating a 3D model in which the components of the target space are appropriately reproduced, even when the components of the target space are composed of complex surfaces and when there are defects in the restored point cloud of the target space. They are useful as a 3D model generation apparatus, 3D model generation method, and 3D model generation program, etc., which use a processor to execute a process to generate a 3D model of the components of the target space based on the restored point cloud obtained by a 3D restoration process of the target space in a building.
[0298] 1: User terminal (display device) 2: Server (3D model generation device) 23: Processor 31: Basic screen 32: 3D viewer section 33: Operation panel section 34: Mode display selection section 35: Image display section 43, 44: Buttons 45: Automatic inside / outside determination instruction section 46: Inside / outside sorting operation section 47, 48: Buttons 51: Toggle switch (operation section) 52: Slider (operation section) 55, 56, 57, 58: Buttons 61: Restored point cloud display screen 71: Segment display screen 81: 3D model display screen 82, 83, 84: Interpolation area 85: Interpolation reason display section 86: Error area 91: Settings screen 92: Interpolation area display setting section 93: Error area display setting section 95: Checkbox 96: Display color box 97: Input box 98: Display color box
Claims
1. A 3D model generation apparatus that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
2. The three-dimensional model generation apparatus according to claim 1, characterized in that, in the modeling process, the processor generates candidate faces that are candidates to be constituent faces of the three-dimensional model based on the segment to be modeled, calculates an index value for the appropriateness of each candidate face as a constituent face of the three-dimensional model based on the distribution of point clouds on the candidate face, selects an appropriate candidate face as a constituent face of the three-dimensional model based on the index value, and generates the three-dimensional model relating to the external components to be modeled based on the selected candidate face.
3. The 3D model generation apparatus according to claim 1, characterized in that the processor performs noise reduction processing, downsampling processing, and coordinate axis setting processing on the restored point cloud as preprocessing before performing the segmentation processing.
4. The 3D model generation apparatus according to claim 1, wherein the processor performs a point cloud division process in which the restored point cloud is divided into a plurality of segments by a region expansion method that expands the region by sequentially adding neighboring points in the segmentation process, and performs a linear segment detection process to detect linear segments that are not to be modeled from among the segments obtained in the point cloud division process.
5. The 3D model generation apparatus according to claim 1, characterized in that, in the automatic segment determination process, the processor groups adjacent segments into one group, determines that the segment included in the group with the maximum number of points is a modeling target representing an external component, and determines that the segment not included in the group with the maximum number of points is an external component that forms a floating surface not in contact with an external component and is not a modeling target.
6. The 3D model generation apparatus according to claim 1, wherein the processor, in the automatic segment determination process, groups segments whose normal direction is the direction of each coordinate axis of the 3D Cartesian coordinate system for each coordinate axis, determines the maximum distance between segments in each coordinate axis group as the maximum inter-face distance for each coordinate axis, sets a 3D bounding box for each segment having sides parallel to each coordinate axis, and determines that a segment is a modeling target representing an external structure if the ratio of the length of the longest side to the maximum inter-face distance with respect to the coordinate axis parallel to the longest side of the 3D bounding box exceeds a predetermined threshold.
7. The three-dimensional model generation apparatus according to claim 2, characterized in that the processor, in the modeling process, calculates a point cloud distribution index value as the index value, which represents the proportion of the range in which point clouds are distributed on the candidate surface, and performs an optimization process to select the candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the point cloud distribution index value.
8. The three-dimensional model generation apparatus according to claim 2, characterized in that the processor, in the modeling process, calculates a distance index value as the index value, which represents the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the distance index value.
9. The three-dimensional model generation apparatus according to claim 2, characterized in that the processor calculates a region division index value as the index value, which represents the degree of inappropriateness regarding a combination of two candidate faces that cause division of the segment that will become the candidate face, and performs an optimization process to select a candidate face that is appropriate as a constituent face of the three-dimensional model based on the region division index value.
10. The 3D model generation apparatus according to claim 1, characterized in that the processor presents the determination result of the automatic segment determination process to the user and performs a manual segment setting process to change the determination result of the automatic segment determination process in response to the user's operation of selecting a segment and specifying whether or not it is a target for modeling.
11. A method for generating a three-dimensional model, comprising: a processor executing a process to generate a three-dimensional model of the components of a target space based on a restored point cloud obtained by a three-dimensional restoration process of the target space within a building; a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud; an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components; and a modeling process to generate the three-dimensional model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
12. A 3D model generation program that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, the program comprising: performing a segmentation process to extract multiple segments representing flat surfaces from the restored point cloud; performing an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components; and performing a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
13. A three-dimensional model generation apparatus that uses a processor to generate a three-dimensional model of the components of a target space based on a restored point cloud obtained by a three-dimensional restoration process of the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, displays a screen on a display device that includes a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them, and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively, performs a manual segment setting process to set the segments to be modeled according to the user's operation, and performs a modeling process to generate the three-dimensional model of the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
14. The three-dimensional model generation apparatus according to 13, characterized in that the processor generates candidate faces that are candidates for constituent faces of the three-dimensional model based on the segment to be modeled in the modeling process, calculates an index value for the appropriateness of each candidate face as a constituent face of the three-dimensional model based on the distribution of point clouds on the candidate faces, selects a candidate face that is appropriate as a constituent face of the three-dimensional model based on the index value, and generates the three-dimensional model based on the selected candidate face.
15. The 3D model generation apparatus according to claim 13, characterized in that the processor highlights the segments extracted in the segmentation process and suppresses the point cloud that was not extracted as segments in the viewer section.
16. The 3D model generation apparatus according to claim 13, characterized in that the processor, in the viewer section, highlights the segments set as modeling targets in the manual segment setting process and suppresses the display of segments set as not to be modeled targets.
17. The three-dimensional model generation apparatus according to claim 13, characterized in that the processor highlights the selected segment in the viewer unit.
18. The 3D model generation apparatus according to claim 13, characterized in that, in the manual segment setting process, the processor collectively sets whether or not the selected segments are to be modeled, in response to user operations performed when multiple segments are selected.
19. The 3D model generation apparatus according to claim 13, characterized in that the processor displays a screen including an operation unit that instructs the user to cancel an add and delete operation and to redo the canceled operation.
20. The 3D model generation apparatus according to 13, wherein the processor performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets or not, before the manual segment setting process, and in the manual segment setting process, displays the determination result of the automatic segment determination process on the viewer unit and changes the determination result of the automatic segment determination process according to the user's operation.
21. A method for generating a three-dimensional model, comprising: a processor executing a process to generate a three-dimensional model of the components of a target space based on a restored point cloud obtained by a three-dimensional restoration process of the target space within a building; a method for generating a three-dimensional model, comprising: executing a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud; displaying a screen on a display device that includes a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them; and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively; executing a manual segment setting process to set the segments to be modeled according to the user's operation; and executing a modeling process to generate the three-dimensional model of the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
22. A 3D model generation program that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, the program comprising: performing a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud; displaying a screen on a display device that includes a viewer unit that displays the segments extracted by the segmentation process in a way that allows the user to select them; and an operation unit that instructs the user to add the selected segments to the modeling target and to delete them from the modeling target, respectively; performing a manual segment setting process to set the segments to be modeled according to the user's operation; and performing a modeling process to generate the 3D model of the components to be modeled based on the segments set as the modeling target in the manual segment setting process.
