Information processing apparatus, information processing method, carrier means, and information processing system

Abstract

An information processing apparatus includes a processing unit configured to create first two-dimensional data based on a first three-dimensional point cloud and a projection plane and create second two-dimensional data based on a second three-dimensional point cloud and the projection plane, and a registration unit configured to register the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.

Description

FN202502341[DESCRIPTION][Title of Invention]INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, CARRIER MEANS, AND INFORMATION PROCESSING SYSTEM[Technical Field]

[0001] The present disclosure relates to an information processing apparatus, an information processing method, carrier means, and an information processing system.[Background Art]

[0002] In the related art, there is a technique for automatically generating a three-dimensional model from point cloud data acquired by measurement of a target. In such a technique, a frame structure including multiple portions such as a ceiling, a floor, and walls is detected from point cloud data of the inside of a building, so that a three-dimensional model representing each room in the building is generated.Generation of the three-dimensional model may involve registration of multiple pieces of point cloud data that have been separately acquired.PTL 1 describes a method for performing high-speed sequential matching (registration) of two two-dimensional figures by converting local features of the two-dimensional figures into hash values using a locality sensitive hashing (LSH) function.[Citation List][Patent Literature]

[0003] [PTL 1]lapanese Patent No. 5182762[Summary of Invention][Technical Problem]

[0004] In the related art, however, when point cloud data includes multiple similar shapes, registration may be performed at incorrect positions (e.g., local optimum positions). For example, buildings such as office buildings, apartment buildings, or hotels have a structure in which rooms, windows, or doors having similar shapes are arranged monotonously.Therefore, it is difficult to accurately perform registration of point cloud data acquired from such a structure.[Solution to Problem]

[0005] The present disclosure described herein provides an information processing apparatus including a processing unit that creates first two-dimensional data based on a first three-dimensional point cloud and a projection plane and creates second two-dimensional data based on a second three-dimensional point cloud and the projection plane; and a registrationFN202502341unit that registers the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.The present disclosure described herein provides an information processing method performed by an information processing apparatus, including creating first two-dimensional data based on a first three-dimensional point cloud and a projection plane; creating second two-dimensional data based on a second three-dimensional point cloud and the projection plane; and registering the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.The present disclosure described herein provides carrier means carrying computer readable code for controlling a computer system to carry out an information processing method, the information processing method including creating first two-dimensional data based on a first three-dimensional point cloud and a projection plane; creating second two-dimensional data based on a second three-dimensional point cloud and the projection plane; and registering the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data. The present disclosure described herein provides an information processing system including a processing unit that creates first two-dimensional data based on a first three-dimensional point cloud and a projection plane and creates second two-dimensional data based on a second three-dimensional point cloud and the projection plane; a registration unit that registers the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data; a display control unit that causes a display to display the first two-dimensional data and the second two-dimensional data; and a reception unit that receives a registration instruction. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud based on the registration instruction received by the reception unit.[Advantageous Effects of Invention]

[0006] According to one aspect of the present disclosure, even when point cloud data includes multiple similar shapes, registration can be accurately performed on the point cloud data. [Brief Description of Drawings]

[0007] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.FIG. l is a diagram illustrating an example of a general arrangement of an information processing system according to a first embodiment.FIG. 2 is a diagram illustrating an example of a hardware configuration of a terminal apparatus and a management server according to the first embodiment.FN202502341FIG. 3 is a diagram illustrating an example of functional blocks of the information processing system according to the first embodiment.FIG. 4A is a diagram illustrating an example of a three-dimensional point cloud.FIG. 4B is a diagram illustrating an example of a three-dimensional point cloud.FIG. 5 is a diagram illustrating an example of functional blocks of a processing unit according to the first embodiment.FIG. 6A is a diagram illustrating an example of a tilt-corrected three-dimensional point cloud. FIG. 6B is a diagram illustrating an example of a tilt-corrected three-dimensional point cloud. FIG. 7Ais a diagram illustrating an example of two-dimensional data.FIG. 7B is a diagram illustrating an example of two-dimensional data.FIG. 8 is a flowchart illustrating an example of a procedure of a registration process according to the first embodiment.FIG. 9 is an illustration of an example of a display screen including an operation screen for the registration process.FIG. 10 is an illustration of an example of the operation screen that displays two pieces of two-dimensional data.FIG. 11 is an illustration of an example of the operation screen in which the orientation of a piece of two-dimensional data is changed.FIG. 12 is an illustration of an example of the operation screen that displays a two-dimensional registration candidate.FIG. 13 is a diagram illustrating an example of functional blocks of a management server according to a second embodiment.The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.[Description of Embodiments]

[0008] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0009] An information processing apparatus, an information processing method, a program, and an information processing system according to embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.FN202502341

[0010] In industries such as civil engineering and architecture, building information modeling (BIM) and construction information modeling (CIM) are increasingly used to improve labor productivity, for example.

[0011] BIM is a solution for utilizing information of a database of buildings in all steps from architectural design and construction to maintenance and management. The database includes three-dimensional digital models of buildings created on a computer and attribute data such as cost, finishing details, and management information of the respective buildings. The three-dimensional digital models are hereinafter referred to as three-dimensional models or 3D models.

[0012] CIM is a solution proposed for the field of civil engineering (covering general infrastructure such as roads, electricity, gas, and water supply) following BIM that has been increasingly used in the field of architecture. Similarly to BIM, CIM is implemented as a solution to increase the efficiency and sophistication of a series of construction production systems by allowing information such as 3D models to be shared among persons concerned.

[0013] A challenge in promoting BIM and CIM is how to easily acquire 3D information of a target such as a space in a building or a public facility. The 3D information refers to a three-dimensional point cloud (hereinafter also simply referred to as a point cloud) that retains distance information to a target and is acquired by measuring the target with a laser scanner (hereinafter referred to as LS). The 3D information also refers to a mesh object or 3D model generated based on point cloud data representing the three-dimensional point cloud.

[0014] It is easy to introduce BIM and CIM in the case where a structure is constructed from scratch because the completed structure can be designed from scratch by using BIM and CIM software or the like. On the other hand, in the case of an existing building, design drawings at the time of construction may be lost or the current state may differ from drawings at the time of design due to renovations over time. This raises the hurdle for introducing BIM and CIM. Such BIM applied to existing buildings is called as-built BIM. As-built BIM is a significant challenge in promoting BIM and CIM in the future.

[0015] One method for implementing as-built BIM is a workflow including measurement of a space using the LS described above and creation of a 3D model from point cloud data obtained by measurement. This work has hitherto been performed using methods such as measuring a space using a photograph and a tape measure or hand-sketching a space. However, depending on the size of the space, the presence or absence of installed objects, and the complexity of installed objects (such as arrangement of pipes), the work may incur enormous cost.FN202502341Therefore, introduction of the LS that enables acquisition of the 3D information of a space is attracting attention as an effective method for addressing this challenge.

[0016] As-built BIM using an LS makes acquisition of 3D information easier but produces a new and previously nonexistent work of performing point cloud processing on point cloud data. In general point cloud processing, processing such as "multi-point measurement using an LS", "generation of an integrated point cloud from individual point clouds", "removal of unnecessary point clouds such as noise", "mesh conversion of point clouds", "texture mapping to meshes", or "generation of a 3D model" is performed.

[0017] In generation of an integrated point cloud from individual point clouds, the integrated point cloud is generated by registering and superimposing the multiple point clouds. For example, point clouds acquired more finely from respective rooms on the same floor are registered with a point cloud acquired coarsely from the entire floor of a building, so that an integrated point cloud is generated. In this case, the use of the point cloud of the entire floor enables registration of the point clouds of the respective rooms.

[0018] In this example, different devices are usable for different purposes such as using an easy-to-handle device for a coarse point cloud and using an accurate but time-consuming device to acquire data of fine point clouds. Examples of the former device include a smartphone or 360-degree camera. Examples of the latter device include a costly laser scanner.

[0019] In the example described above, a new point cloud is acquired for an updated portion on the floor and is matched with the integrated point cloud representing the entire floor, so that updated data can be easily reflected.

[0020] In another example of generating the integrated point cloud, point clouds are acquired for respective portions on the same floor, and the point clouds are registered with each other to generate a point cloud of the entire floor. In this case, for example, multiple people may collaboratively acquire the point clouds of a large floor. In this case, point clouds of respective portions are acquired to include regions common to each other. The point clouds are registered so that the common regions overlap, so that a point cloud of the entire floor can be generated.

[0021] First EmbodimentFIG. l is a diagram illustrating an example of a general arrangement of an information processing system 1 according to a first embodiment. The information processing system 1 according to the present embodiment includes a terminal apparatus 3 and a management server 5.

[0022] FN202502341The management server 5 is an example of an information processing apparatus that performs one or more information processing processes on point cloud data representing a three-dimensional point cloud.

[0023] A three-dimensional point cloud is a collection of X-axis, Y-axis, and Z-axis coordinate points corresponding to measurement points on the surface of an obj ect obtained when a space having the object is measured using the LS or the like. A coordinate point is represented as (1, 3, 5), for example. In the present embodiment, the X-axis and the Y-axis are two-dimensional coordinate axes on a horizontal plane and the Z-axis is a coordinate axis in the vertical direction perpendicular to the horizontal plane (i.e., the XY plane).

[0024] Each coordinate point may additionally have color information. The color information added may be red, green, and blue (RGB) values of each coordinate point. The three-dimensional point cloud may also be referred to as a point cloud. The point cloud data is data that allows the three-dimensional point cloud to be handled by a computer or the like as a collection of coordinate points in a virtual 3D space.

[0025] An example of measuring a three-dimensional point cloud using the LS is presented above. However, the three-dimensional point cloud may be measured using another optical measuring means or a mechanical measuring means. Examples of another optical measuring means include a method using a stereo camera and a method using visual simultaneous localization and mapping (SLAM).

[0026] The terminal apparatus 3 and the management server 5 communicate with each other via a communication network 100. The communication network 100 includes the Internet, a mobile communication network, or a local area network (LAN), for example. The communication network 100 may include, in addition to a wired communication network, a wireless communication network that is compliant with 3rd Generation (3G), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE), 5th Generation (5G), or the like. The terminal apparatus 3 may perform communication using short-range communication technology such as Near Field Communication (NFC®).

[0027] Hardware ConfigurationFIG. 2 is a diagram illustrating an example of a hardware configuration of the terminal apparatus 3 and the management server 5 according to the first embodiment. Hardware components of the terminal apparatus 3 are denoted by reference signs in 300s. Hardware components of the management server 5 are denoted by reference signs in 500s in parentheses.

[0028] FN202502341The terminal apparatus 3 includes a central processing unit (CPU) 301, a read-only memory (ROM) 302, a random access memory (RAM) 303, a hard disk (HD) 304, a hard disk drive (HDD) 305, a recording medium 306, a medium interface (I / F) 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a compact disc-rewritable (CD-RW) drive 314, and a bus line 310.

[0029] The CPU 301 controls the overall operation of the terminal apparatus 3. The ROM 302 stores a program used for driving the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various types of data such as a program. The HDD 305 controls reading or writing of various types of data from or to the HD 304 under control of the CPU 301.

[0030] The medium I / F 307 controls reading or writing (storing) of data from or to the recording medium 306 such as a flash memory. The display 308 displays various types of information such as a cursor, a menu, a window, text, or an image. The network I / F 309 is an interface for performing data communication using the communication network 100.

[0031] The keyboard 311 is an example of an input means that includes multiple keys for inputting characters, numerical values, various instructions, etc. The mouse 312 is an example of an input means for selecting or executing various instructions, selecting a target for processing, or moving a cursor.

[0032] The CD-RW drive 314 controls reading or writing of various types of data from or to a CD-RW 313, which is an example of a removable recording medium. The terminal apparatus 3 may further include a component for controlling reading or writing (storing) of data from or to an external personal computer (PC) or external device connected with a cable or connected wirelessly by wireless fidelity (Wi-Fi®) or the like.

[0033] The management server 5 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD 505, a recording medium 506, a medium I / F 507, a display 508, a network I / F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. Since the components of the management server 5 are substantially the same as the respective components of the terminal apparatus 3 described above (i.e., the CPU 301, the ROM 302, the RAM 303, the HD 304, the HDD 305, the recording medium 306, the medium I / F 307, the display 308, the network I / F 309, the keyboard 311, the mouse 312, the CD-RW drive 314, and the bus line 310), a description thereof is omitted. The CD-RW drive 514 controls reading or writing of various types of data from or to a CD-RW 513, which is an example of a removable recording medium.

[0034] FN202502341The CD-RW drive 314 (514) may be replaced with a CD-R drive or the like. In one embodiment, each of the terminal apparatus 3 and the management server 5 is implemented by a single computer. In another embodiment, each of the terminal apparatus 3 and the management server 5 is implemented by multiple computers to which the components (functions, means, or storages) are distributed as desired.

[0035] FIG. 3 is a diagram illustrating an example of functional blocks of the information processing system 1 according to the first embodiment.

[0036] Functional Configuration of Terminal ApparatusAs illustrated in FIG. 3, the terminal apparatus 3 includes a transmitting and receiving unit 31, a reception unit 32, a display control unit 34, and a storing and reading unit 39. Each of these units is a function implemented by or a means caused to function by a corresponding one of the hardware components illustrated in FIG. 2 operating in accordance with instructions from the CPU 301 according to a program loaded to the RAM 303 from the HD 304. The terminal apparatus 3 further includes a storage unit 3000 implemented by the RAM 303 and the HD 304 illustrated in FIG. 2.

[0037] Functional Components of Terminal ApparatusFunctional components of the terminal apparatus 3 will be described.

[0038] The transmitting and receiving unit 31 is an example of a receiving means and is implemented by instructions from the CPU 301 and by the network I / F 309 illustrated in FIG. 2. The transmitting and receiving unit 31 transmits and receives various types of data (or information) to and from other terminals, apparatuses, or systems via the communication network 100.

[0039] The reception unit 32 is an example of a reception means and is implemented by instructions from the CPU 301 and by the keyboard 311 and the mouse 312 illustrated in FIG. 2. The reception unit 32 receives various inputs from the user.

[0040] The display control unit 34 is an example of a display control means and is implemented by instructions from the CPU 301 illustrated in FIG. 2. The display control unit 34 causes the display 308, which is an example of a display, to display various images or screens.

[0041] The storing and reading unit 39 is an example of a storage control means and is implemented by instructions from the CPU 301 and by the HDD 305, the medium I / F 307, and the CD-RW drive 314 illustrated in FIG. 2 and by an external PC or an external device. The storing and reading unit 39 performs processing of storing various types of data to the storage unit 3000, the recording medium 306, the CD-RW 313, and an external PC or an external device andFN202502341reading various types of data from the storage unit 3000, the recording medium 306, the CD-RW 313, and the external PC or the external device.

[0042] Functional Configuration of Management ServerThe management server 5 includes a transmitting and receiving unit 51, a processing unit 53, a determination unit 55, a generation unit 57, and a storing and reading unit 59. Each of these units is a function implemented by or a means caused to function by a corresponding one of the hardware components illustrated in FIG. 2 operating in accordance with instructions from the CPU 501 according to a program loaded to the RAM 503 from the HD 504. The management server 5 further includes a storage unit 5000 implemented by the HD 504 illustrated in FIG. 2. The storage unit 5000 is an example of a storage means.

[0043] Functional Components of Management ServerFunctional components of the management server 5 will be described. The management server 5 may be implemented such that the functions of the management server 5 are distributed to multiple computers. The management server 5 is described as a server computer residing in a cloud environment but may be a server residing in an on-premises environment.

[0044] The transmitting and receiving unit 51 is an example of a transmitting means and is implemented by instructions from the CPU 501 and by the network I / F 509 illustrated in FIG.2. The transmitting and receiving unit 51 transmits and receives various types of data (or information) to and from other terminals, apparatuses, or systems via the communication network 100.

[0045] The processing unit 53 is implemented by instructions from the CPU 501 illustrated in FIG. 2. The processing unit 53 performs various processes including a 3D model processing process (described later). The processing unit 53 is an example of a processing means.

[0046] The determination unit 55 is implemented by instructions from the CPU 501 illustrated in FIG. 2. The determination unit 55 makes various determinations.

[0047] The generation unit 57 is implemented by instructions from the CPU 501 illustrated in FIG. 2. The generation unit 57 generates various display screens (described later). The generation unit 57 generates display screens for displaying a three-dimensional point cloud, a 3D mesh, and a 3D model, for example. The generation unit 57 can generate a graphical user interface (GUI) screen for correcting a 3D model.

[0048] The storing and reading unit 59 is an example of a storage control means and is implemented by instructions from the CPU 501 and by the HDD 505, the medium I / F 507, and the CD-RW drive 514 illustrated in FIG. 2 and by an external PC or an external device. The storing andFN202502341reading unit 59 performs processing of storing various types of data to the storage unit 5000, the recording medium 506, the CD-RW 513, and an external PC or an external device and reading various types of data from the storage unit 5000, the recording medium 506, the CD-RW 513, and the external PC or the external device. The storage unit 5000, the recording medium 506, the CD-RW 513, and the external PC or the external device are an example of a storage means.

[0049] The storage unit 5000 includes a user information management database (DB) 5001, a setting information management DB 5002, a storage processing management DB 5003, a point cloud management DB 5004, and a processing result management DB 5005.

[0050] The user information management DB 5001 stores and manages a file name of three-dimensional point cloud data in association with user information. The setting information management DB 5002 stores and manages various types of setting information. The storage processing management DB 5003 stores and manages various processing programs and data for executing point cloud processing. The point cloud management DB 5004 stores and manages point cloud data. The processing result management DB 5005 stores and manages processing result information indicating a processing result of executing the point cloud processing on point cloud data.

[0051] FIGs. 4A and 4B are diagrams each illustrating an example of a three-dimensional point cloud. FIGs. 4A and 4B illustrate an example of a three-dimensional point cloud that is acquired from a structure, is visualized in an XYZ space, and viewed from directly above. FIG. 4A illustrates a three-dimensional point cloud acquired from a wide area of the structure. FIG. 4B illustrates a three-dimensional point cloud acquired from a portion of the structure in an area narrower than that in FIG. 4A. The present embodiment describes an example of registering these three-dimensional point clouds.

[0052] Functional Configuration of Processing UnitFIG. 5 is a diagram illustrating an example of functional blocks of the processing unit 53 according to the first embodiment. As illustrated in FIG. 5, the processing unit 53 includes a tilt correction unit 531, a two-dimensional data creation unit 532, an orientation changing unit 533, a two-dimensional registration candidate generation unit 534, a three-dimensional registration information generation unit 535, and a registration unit 536.

[0053] The tilt correction unit 531 corrects a tilt of a three-dimensional point cloud. A three-dimensional point cloud acquired from a frame structure has rectangular parallelepiped or rectangular structures (hereinafter referred to as rectangularity) for rooms and windows. The tilt correction unit 531 performs tilt correction to make faces or sides of these structuresFN202502341included in the three-dimensional point cloud parallel to one of axes of the three-dimensional coordinate system.

[0054] The tilt correction unit 531 uses, for example, an algorithm such as random sample consensus (RANSAC) to obtain a rectangle that approximates a ceiling or a floor from a point cloud of the ceiling or the floor, and corrects the rectangle to make a side of the rectangle parallel to an X-axis or a Y-axis. This allows the orientations of the respective point clouds to be matched by rotation of multiple three-dimensional point clouds of the frame structure having rectangularity in units of 90 degrees in the XY plane. The tilt correction unit 531 may obtain a rectangle that approximates a wall from a point cloud of the wall and may correct the rectangle to make a vertical side of the rectangle parallel to the Z-axis.

[0055] The storing and reading unit 59 stores, in the processing result management DB 5005, transformation information of each three-dimensional point cloud used by the tilt correction unit 531 in the tilt correction process (rotation process), for example, as a transformation matrix.

[0056] FIGs. 6A and 6B are diagrams each illustrating an example of a tilt-corrected three-dimensional point cloud. FIGs. 6A and 6B illustrate examples of tilt-corrected three-dimensional point clouds of the three-dimensional point clouds illustrated in FIGs. 4A and 4B, respectively. As illustrated in FIGs. 6A and 6B, the tilt correction unit 531 corrects the tilt of a three-dimensional point cloud such that a side of a rectangle of a ceiling or a floor of a structure is parallel to the X-axis or the Y-axis.

[0057] The two-dimensional data creation unit 532 creates two-dimensional data based on a three-dimensional point cloud and a projection plane. The projection plane is a plane that is parallel to a plane that approximates one of a ceiling, a floor, or a wall included in the frame structure. The two-dimensional data is a two-dimensional point cloud or a two-dimensional image that is a projection of the three-dimensional point cloud onto the projection plane.

[0058] The two-dimensional data creation unit 532 removes a three-dimensional point cloud included in an area parallel to the projection plane, and projects the resulting three-dimensional point cloud onto the projection plane to create the two-dimensional point cloud or the two-dimensional image. In the present embodiment, since a registration process using two-dimensional data (i.e., two-dimensional registration process) is performed, noise that hinders the two-dimensional registration process is removed by removing the three-dimensional point cloud included in the area parallel to the projection plane.

[0059] For example, when the projection plane is a plane (i.e., the XY plane) parallel to the plane that approximates the ceiling or floor, the three-dimensional point cloud included in the area of theFN202502341ceiling or floor is removed. In this case, each point of the three-dimensional point cloud from which points included in the ceiling or floor are removed is projected onto the XY plane, so that the two-dimensional point cloud is created. Each point of the three-dimensional point cloud from which points included in the ceiling or floor are removed is projected onto the XY plane, pixel values of pixels including the projected points are set to white, and pixel values of the rest of the pixels are set to black, so that the two-dimensional image is created. The two-dimensional image thus created is a binary image including white pixels and black pixels. The two-dimensional image may be created as a multi-value image having pixel values corresponding to the number of points projected to the respective pixels.

[0060] The three-dimensional point cloud included in the area of the ceiling or floor has a high frequency distribution in the Z-axis direction. Thus, the point cloud of the area of the ceiling or floor can be removed by extracting a point cloud having Z coordinate values that are in a range between the peak values of the frequency distribution (that are smaller than the peak value for the ceiling and greater than the peak value for the floor). When the projection plane is a plane (i.e., an XZ plane or a YZ plane) parallel to a plane that approximates the wall, a three-dimensional point cloud included in the area of the wall parallel to the projection plane is removed in the same manner as described above.

[0061] The storing and reading unit 59 stores, in the processing result management DB 5005, information (e.g., information such as the projection plane, the image size, and the resolution) used by the two-dimensional data creation unit 532 for creating the two-dimensional data.

[0062] FIGs. 7A and 7B are diagrams each illustrating an example of two-dimensional data. FIGs. 7 A and 7B illustrate examples of two-dimensional images (e.g., binary images) created from the three-dimensional point clouds illustrated in Figs. 6A and 6B, respectively.

[0063] The orientation changing unit 533 changes the orientation of two-dimensional data based on an instruction from the user. The user checks a display screen (described later), and inputs an instruction to change the orientation of the two-dimensional data to enable registration of the two-dimensional data by translation, for example. The orientation is changed by rotation in units of 90 degrees in the XY plane, for example.

[0064] The storing and reading unit 59 stores, in the processing result management DB 5005, information (e.g., information on a rotation angle) used by the orientation changing unit 533 for changing the orientation of the two-dimensional data.

[0065] The two-dimensional registration candidate generation unit 534 generates one or more candidates in which the two-dimensional data is relatively moved and thus is registered with other two-dimensional data. Relative movement includes translation and rotation.FN202502341

[0066] The two-dimensional registration candidate generation unit 534 uses, for example, template matching to generate registration candidates. Template matching is a technique for detecting a portion that matches a previously defined template (that has a small matching error). The matching error is determined based on a distance between two-dimensional point clouds or a sum of squared differences between pixels of two-dimensional images, for example.

[0067] Among two pieces of two-dimensional data, a smaller piece of two-dimensional data is used as a template and is relatively moved to search for a matching portion within the other piece of two-dimensional data. Template matching can reduce the processing amount of registration since registration does not involve feature extraction processing. Since the matching error is evaluated including small unevenness in the two-dimensional data, template matching can achieve accurate matching even for a point cloud of a building including multiple similar shapes.

[0068] On the other hand, it may be difficult to cope with rotation, scaling, and a change in illumination conditions with template matching. However, since these difficulties rarely become problematic in registration of point clouds of a building because of the reasons below, template matching is suitable for registration of point cloud data.As for rotation, since the above-described rectangularity is usable, it is easy to match the orientations of pieces of two-dimensional data by rotation in units of 90 degrees.As for scaling, since point cloud data is generally acquired using a laser scanner, it is easy to unify the scale.As for illumination conditions, the laser scanner is hardly affected by illumination.

[0069] When there are multiple relative movements for which the matching error is less than or equal to a predetermined threshold value, the two-dimensional registration candidate generation unit 534 generates a registration candidate (i.e., two-dimensional registration candidate) corresponding to each relative movement. The predetermined threshold value is set in advance based on an experiment or the like.

[0070] The storing and reading unit 59 stores, in the processing result management DB 5005, information on two-dimensional registration (i.e., information on translation) selected by the user from among the candidates generated by the two-dimensional registration candidate generation unit 534.

[0071] The three-dimensional registration information generation unit 535 generates information for registering the three-dimensional point clouds (i.e., three-dimensional registration information) based on the two-dimensional registration candidate selected by the user.Specifically, the three-dimensional registration information generation unit 535 integratesFN202502341together the pieces of information stored in the processing result management DB 5005 at the time of the tilt correction, the creation of the two-dimensional data, the change of the orientation of the two-dimensional data, and the registration of the two-dimensional data to generate three-dimensional registration information. The information at the time of tilt correction (i.e., information on rotation), at the time of the change of the orientation of the two-dimensional data (i.e., information on rotation in units of 90 degrees), or the registration of the two-dimensional data (i.e., information on translation) is an example of information on the relative movement.

[0072] The registration unit 536 registers the three-dimensional point clouds based on the three-dimensional registration information generated by the three-dimensional registration information generation unit 535 and outputs registered three-dimensional point cloud data.

[0073] A procedure of a registration process according to the present embodiment will be described with reference to FIGs. 8 to 12. The registration process for two three-dimensional point clouds (i.e., a first three-dimensional point cloud and a second three-dimensional point cloud) will be described.

[0074] FIG. 8 is a flowchart illustrating an example of the procedure of the registration process according to the first embodiment.

[0075] The reception unit 32 of the terminal apparatus 3 receives an input operation from a user. The transmitting and receiving unit 31 transmits information on the received input operation to the transmitting and receiving unit 51 of the management server 5. In step SI 00, the storing and reading unit 59 of the management server 5 reads three-dimensional point cloud data from the point cloud management DB 5004 based on the information on the input operation of the user received by the transmitting and receiving unit 51.

[0076] In step S 101, the tilt correction unit 531 corrects the tilt of a three-dimensional point cloud represented by the read three-dimensional point cloud data. In step SI 02, the two-dimensional data creation unit 532 creates two-dimensional data from the tilt-corrected three-dimensional point cloud.

[0077] FIG. 9 is an illustration of an example of a display screen 1000 including an operation screen 1300 for the registration process. The operation screen 1300 is a user interface (UI) screen that is generated by the generation unit 57 of the management server 5, is transmitted to the terminal apparatus 3, and is displayed on the display 308 by the display control unit 34 of the terminal apparatus 3.

[0078] FN202502341As illustrated in FIG. 9, the operation screen 1300 includes a first display area 1310, a second display area 1320, a candidate display area 1330, point cloud input buttons 1311 and 1321, orientation change buttons 1312 and 1322, a matching button 1331, a pull-down menu 1332, a point cloud output button 1340, and a pointer 1350. The user uses the pointer 1350, which is operated with the mouse 312, to press various buttons or select to-be-displayed data with the pull-down menu 1332.

[0079] The user presses the point cloud input button 1311 to perform an input operation of the first three-dimensional point cloud. The user presses the point cloud input button 1321 to perform an input operation of the second three-dimensional point cloud. Specifically, the user selects or inputs a file name of point cloud data on an input screen or the like that is displayed in response to pressing of the point cloud input buttons 1311 and 1321 to designate data of the first three-dimensional point cloud and data of the second three-dimensional point cloud, respectively.

[0080] FIG. 10 is an illustration of an example of the operation screen 1300 that displays two pieces of two-dimensional data.In this example, the first display area 1310 displays first two-dimensional data created from the first three-dimensional point cloud. The second display area 1320 displays second two-dimensional data created from the second three-dimensional point cloud. The first two-dimensional data is a black-and-white binary image. The second two-dimensional data is a black-and-gray binary image.

[0081] The user presses the orientation change button 1312 to change the orientation of the first two-dimensional data. The user presses the orientation change button 1322 to change the orientation of the second two-dimensional data. For example, the orientation of the corresponding two-dimensional data is changed by 90 degrees each time the orientation change button 1312 or 1322 is pressed.

[0082] Referring back to FIG. 8, when the orientation of the first two-dimensional data or the second two-dimensional data is to be changed (Yes in step S103), the orientation changing unit 533 changes the orientation of the two-dimensional data in step SI 04.When neither the orientation of the first two-dimensional data nor the orientation of the second two-dimensional data is to be changed (No in step SI 03), the process proceeds to step S105.

[0083] FIG. 11 is an illustration of an example of the operation screen 1300 in which the orientation of two-dimensional data is changed. In this example, the second display area 1320 displays the second two-dimensional data whose orientation is rotated by 90 degrees to the right. ThisFN202502341allows the second two-dimensional data to be relatively moved to and registered with a portion of the first two-dimensional data.

[0084] Referring back to FIG. 8, in response to the user pressing the matching button 1331, the two-dimensional registration candidate generation unit 534 generates a two-dimensional registration candidate in step SI 05.

[0085] FIG. 12 is an illustration of an example of the operation screen 1300 that displays a two-dimensional registration candidate.In this example, the candidate display area 1330 displays an image in which the relatively moved second two-dimensional data is superimposed on the first two-dimensional data. In this example, a registration result (superimposed image) of a "Candidate 1" is displayed among multiple two-dimensional registration candidates, and the second two-dimensional data is superimposed at a position of a rectangle 1320-1 in the candidate display area 1330.

[0086] The user uses the pull-down menu 1332 to change the two-dimensional registration candidate to be displayed in the candidate display area 1330 from the "Candidate 1" and checks the registration results of the other candidates. When the desired two-dimensional registration candidate is displayed in the candidate display area 1330, the user presses the point cloud output button 1340 to output registered three-dimensional point cloud data corresponding to the two-dimensional registration candidate.

[0087] Referring back to FIG. 8, in response to the user pressing the point cloud output button 1340, the three-dimensional registration information generation unit 535 generates three-dimensional registration information based on the two-dimensional registration candidate selected by the user in step S106. The registration unit 536 registers the three-dimensional point clouds based on the generated three-dimensional registration information in step SI 07, and outputs the registered three-dimensional point cloud data.

[0088] As described above, in the present embodiment, even when point cloud data acquired from an office building or the like has multiple similar shapes, registration can be accurately performed on the point cloud data.

[0089] The target of the registration process in the present embodiment is not limited to the ceiling, the floor, and the walls and may be other portions of the frame structure such as doors and window frames or members inside a building such as tables and chairs.

[0090] Second EmbodimentA second embodiment of the present disclosure will be described.

[0091] FN202502341In the second embodiment, the management server 5 includes the functions of the terminal apparatus 3 described in the first embodiment. In the description of the second embodiment below, the description of a portion overlapping that of the first embodiment is omitted and differences from the first embodiment will be described.

[0092] FIG. 13 is a diagram illustrating an example of functional blocks of the management server 5 according to the second embodiment. Differences from the first embodiment (FIG. 3) are that the management server 5 further includes a reception unit 60 and a display control unit 61. Since operations of the other functional units (such as the transmitting and receiving unit 51 and the processing unit 53) of the management server 5 are substantially the same as those described in FIG. 3, the description thereof is omitted.

[0093] The reception unit 60 has substantially the same function as the reception unit 32 illustrated in FIG. 3 and receives input operations (registration instructions) from a user. The registration instructions include the input operation of a three-dimensional point cloud, the instruction to change the orientation of two-dimensional data, and the instruction to generate or select a two-dimensional registration candidate described above.

[0094] The display control unit 61 has substantially the same function as the display control unit 34 illustrated in FIG. 3 and causes the display 508, which is an example of a display, to display various images and screens. The display 508 displays the display screen 1000 including the operation screen 1300 described above. The user uses the pointer 1350, which is operated with the mouse 512, to press various buttons or select a to-be-displayed candidate with the pull-down menu 1332.

[0095] As described above, in the present embodiment, registration can be accurately performed on the point cloud data as in the first embodiment via a UI screen displayed on the display 508 of the management server 5.

[0096] The program to be executed by the information processing apparatus according to each of the embodiments described above is recorded on and provided through a non-transitory computer-readable recording medium such as a compact disc read-only memory (CD-ROM), a flexible disk (FD), a compact disc recordable (CD-R), or a digital versatile disc (DVD), in an installable or executable file format.

[0097] The program to be executed by the information processing apparatus according to each of the embodiments may be stored on a computer connected to a network such as the Internet, and downloaded and thus provided via the network. The program to be executed by the information processing apparatus according to each of the embodiments may be provided or distributed via a network such as the Internet.FN202502341

[0098] The program according to each of the embodiments may be preinstalled and provided on the ROM or the like.

[0099] The program to be executed by the information processing apparatus according to each of the embodiments has a module configuration including the above-described functional units (such as the tilt correction unit 531, the orientation changing unit 533, and the two-dimensional data creation unit 532). The CPU (i.e., a processor) that is actual hardware reads the program from the recording medium and executes the program, so that the above-described functional units are loaded to and generated in a main storage device.

[0100] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality.There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0101] The apparatuses or devices described in the embodiments are merely one example of multiple computing environments that implement the embodiments disclosed herein.

[0102] In some embodiments, the management server 5 includes multiple computing devices, such as a server cluster. The multiple computing devices communicate with one another through any type of communication link including a network, a shared memory, or the like and perform the processes disclosed herein. Likewise, the terminal apparatus 3 may also include multiple computing devices that communicate with one another.

[0103] The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may beFN202502341combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0104] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Internet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD-ROM), a magnetic tape device, or a solid state memory device.

[0105] For example, aspects of the present disclosure include the following.According to Aspect 1, an information processing apparatus includes a processing unit and a registration unit. The processing unit creates first two-dimensional data based on a first three-dimensional point cloud and a projection plane and creates second two-dimensional data based on a second three-dimensional point cloud and the projection plane. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.According to Aspect 2, in the information processing apparatus of Aspect 1, the processing unit creates as the first two-dimensional data, a two-dimensional point cloud that is a projection of the first three-dimensional point cloud onto the projection plane, and as the second two-dimensional data, a two-dimensional point cloud that is a projection of the second three-dimensional point cloud onto the projection plane.According to Aspect 3, in the information processing apparatus of Aspect 1, the processing unit creates as the first two-dimensional data, a two-dimensional image that is a projection of the first three-dimensional point cloud onto the projection plane, and creates as the second two-dimensional data, a two-dimensional image that is a projection of the second three-dimensional point cloud onto the projection plane.FN202502341According to Aspect 4, in the information processing apparatus of any one of Aspects 1 to 3, the first three-dimensional point cloud and the second three-dimensional point cloud include three-dimensional point clouds representing a frame structure including multiple portions including a ceiling, a floor, and a wall. The projection plane includes a plane parallel to an approximation plane that approximates one of the multiple portions of the frame structure. According to Aspect 5, in the information processing apparatus of Aspect 4, the processing unit corrects a tilt of the first three-dimensional point cloud and a tilt of the second three-dimensional point cloud to make a side of a rectangle that approximates one of the multiple portions of the frame structure parallel to one of axes of a three-dimensional coordinate system.According to Aspect 6, in the information processing apparatus of Aspect 4 or 5, the processing unit removes, from the first three-dimensional point cloud and the second three-dimensional point cloud, a three-dimensional point cloud included in an area where the approximation plane is parallel to the projection plane among the three-dimensional point clouds corresponding to the multiple portions of the frame structure, and creates the first two-dimensional data and the second two-dimensional data.According to Aspect 7, the information processing apparatus of any one of Aspect 1 to 6 further includes a display control unit and a reception unit. The display control unit causes a display to display the first two-dimensional data and the second two-dimensional data. The reception unit receives a registration instruction. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud based on the registration instruction received by the reception unit.According to Aspect 8, in the information processing apparatus of Aspect 7, the display control unit causes the display to display the first two-dimensional data and the second two-dimensional data that are relatively moved based on the registration instruction.According to Aspect 9, in the information processing apparatus of Aspect 7 or 8, the registration instruction includes an operation instruction to relatively move the first two-dimensional data or the second two-dimensional data. The display control unit causes the display to display the first two-dimensional data or the second two-dimensional data that is relatively moved based on the operation instruction. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud based on information on the relative movement.According to Aspect 10, an information processing method to be executed by an information processing apparatus, includes: creating first two-dimensional data based on a first three-dimensional point cloud and a projection plane and creating second two-dimensional data based on a second three-dimensional point cloud and the projection plane; and registering the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data. According to Aspect 11, a program causing a computer to function as: processing means configured to create first two-dimensional data based on a first three-dimensional point cloudFN202502341and a projection plane, and second two-dimensional data based on a second three-dimensional point cloud and the projection plane; and registration means configured to register the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.According to Aspect 12, an information processing system includes an information processing apparatus and a terminal apparatus to communicate with the information processing apparatus. The information processing system includes a processing unit, a registration unit, a display control unit, and a reception unit. The processing unit creates first two-dimensional data based on a first three-dimensional point cloud and a projection plane and creates second two-dimensional data based on a second three-dimensional point cloud and the projection plane. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data. The display control unit causes a display to display the first two-dimensional data and the second two-dimensional data. The reception unit receives a registration instruction. The registration unit registers the first three-dimensional point cloud and the second three-dimensional point cloud based on the registration instruction received by the reception unit.

[0106] This patent application is based on and claims priority to Japanese Patent Application No. 2025-026918, filed on February 21, 2025, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.[Reference Signs List]

[0107] 1 information processing system3 terminal apparatus5 management server32 reception unit34 display control unit39 storing and reading unit53 processing unit57 generation unit100 communication network308, 508 display311, 511 keyboard312, 512 mouse531 tilt correction unit532 two-dimensional data creation unit533 orientation changing unit534 two-dimensional registration candidate generation unit535 three-dimensional registration information generation unitFN202502341536 registration unit 1000 display screen 1300 operation screen

Claims

FN202502341[CLAIMS]1. An information processing apparatus comprising:a processing unit configured to create first two-dimensional data based on a first three-dimensional point cloud and a projection plane and create second two-dimensional data based on a second three-dimensional point cloud and the projection plane; anda registration unit configured to register the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.

2. The information processing apparatus according to claim 1, wherein the first two-dimensional data is a two-dimensional point cloud that is a projection of the first three-dimensional point cloud onto the projection plane, andthe second two-dimensional data is a two-dimensional point cloud that is a projection of the second three-dimensional point cloud onto the projection plane.

3. The information processing apparatus according to claim 1, wherein the first two-dimensional data is a two-dimensional image that is a projection of the first three-dimensional point cloud onto the projection plane, andthe second two-dimensional data is a two-dimensional image that is a projection of the second three-dimensional point cloud onto the projection plane.

4. The information processing apparatus according to any one of claims 1 to 3, whereinthe first three-dimensional point cloud and the second three-dimensional point cloud include three-dimensional point clouds representing a frame structure including multiple portions including a ceiling, a floor, and a wall, andthe projection plane includes a plane parallel to an approximation plane that approximates one of the multiple portions of the frame structure.

5. The information processing apparatus according to claim 4, wherein the processing unit is configured to correct a tilt of the first three-dimensional point cloud and a tilt of the second three-dimensional point cloud to make a side of a rectangle that approximates one of the multiple portions of the frame structure parallel to one of axes of a three-dimensional coordinate system.

6. The information processing apparatus according to claim 4 or 5, wherein the processing unit is configured to:remove, from the first three-dimensional point cloud and the second three-dimensional point cloud, a three-dimensional point cloud included in an area where the approximationFN202502341plane is parallel to the projection plane among the three-dimensional point clouds corresponding to the multiple portions of the frame structure; andcreate the first two-dimensional data and the second two-dimensional data.

7. The information processing apparatus according to any one of claims 1 to 6, further comprising:a display control unit configured to cause a display to display the first two-dimensional data and the second two-dimensional data; anda reception unit configured to receive a registration instruction, whereinthe registration unit is configured to register the first three-dimensional point cloud and the second three-dimensional point cloud based on the registration instruction received by the reception unit.

8. The information processing apparatus according to claim 7, wherein the display control unit is configured to cause the display to display the first two-dimensional data and the second two-dimensional data that are relatively moved based on the registration instruction.

9. The information processing apparatus according to claim 7 or 8, wherein the registration instruction includes an operation instruction to relatively move the first two-dimensional data or the second two-dimensional data,the display control unit is configured to cause the display to display the first two-dimensional data or the second two-dimensional data that is relatively moved based on the operation instruction, andthe registration unit is configured to register the first three-dimensional point cloud and the second three-dimensional point cloud based on information on the relative movement.

10. An information processing method performed by an information processing apparatus, comprising:creating first two-dimensional data based on a first three-dimensional point cloud and a projection plane;creating second two-dimensional data based on a second three-dimensional point cloud and the projection plane; andregistering the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.

11. Carrier means carrying computer readable code for controlling a computer system to carry out an information processing method, the information processing method comprising:FN202502341creating first two-dimensional data based on a first three-dimensional point cloud and a projection plane;creating second two-dimensional data based on a second three-dimensional point cloud and the projection plane; andregistering the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data.

12. An information processing system comprising:a processing unit configured to create first two-dimensional data based on a first three-dimensional point cloud and a projection plane and create second two-dimensional data based on a second three-dimensional point cloud and the projection plane;a registration unit configured to register the first three-dimensional point cloud and the second three-dimensional point cloud through relative movement of the first two-dimensional data and the second two-dimensional data;a display control unit configured to cause a display to display the first two-dimensional data and the second two-dimensional data; anda reception unit configured to receive a registration instruction, whereinthe registration unit is configured to register the first three-dimensional point cloud and the second three-dimensional point cloud based on the registration instruction received by the reception unit.

Images