Method and system for the automated localisation of objects using 3D scans, and computer program product
The method uses panoramic images and point clouds to accurately localize objects in 3D scans by combining angular measurements and distances, addressing the challenges of large data size and noise in point clouds, enabling precise localization for factory planning.
Patent Information
- Application Number
- PCT/DE2026/100499
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-04-17
- Publication Date
- 2026-11-12
AI Technical Summary
Current methods struggle to consistently and accurately localize hall columns in 3D scans due to large data size, noise in point clouds, low resolution, and obstruction by other objects, hindering reliable detection and classification.
A method using panoramic images and point clouds to identify objects, determine their positions in 2D maps, and span solids in point clouds to precisely locate objects by combining angular measurements and distances, enabling precise localization through object and text recognition.
Enables reliable, automated, and precise localization of objects within 3D scans, allowing for efficient navigation and utilization of hall columns as reference points in factory planning.
Smart Images

Figure DE2026100499_12112026_PF_FP_ABST
Abstract
Description
[0001] 5-1221 DE
[0002] Method and system for the automated localization of objects using 3D scans and computer program product
[0003] The invention relates to a method for the automated localization of objects using 3D scans comprising at least one panoramic image and at least one point cloud. Furthermore, the invention relates to a system for the automated localization of objects using 3D scans and a computer program product.
[0004] Nowadays, the automated analysis of images, for example by artificial intelligence, plays an increasingly important role in the recognition of objects and / or areas. For instance, German patent DE 102015205855 A1 describes a method for the automatic detection of parking zones in at least one street. This method uses a processing unit, reads information in the form of panoramic images of the at least one street from an external data storage device, and reads information in the form of street data for the at least one street from a map database. An internal database is also created to store the panoramic images. The read panoramic images are analyzed for the presence of vehicles, and further analyzed for the presence of street signs and traffic signs.Based on analyses of vehicle presence and the presence of road signs and traffic signs, the expected existing parking / no-stopping zones are determined for at least one selected street segment. A dataset is then generated and saved, containing the information on detected vehicles, road signs, and the markings of parking / no-stopping zones and parking areas. Finally, the information contained in the dataset is visualized on a terminal device. 25-1221 DE 2.
[0005] Factory planning is also increasingly relying on the evaluation of existing images. For example, DE 102020204921 A1 discloses a method and an arrangement for the digital acquisition of rooms within a building, wherein a scanning device scans a corresponding room in the building and captures it in a digital point cloud and / or by means of an image recording, wherein object recognition is performed using artificial intelligence based on the digital point cloud and / or the image recording, wherein after successful object recognition the digital point cloud and / or the image recording is mapped into a digital building model, wherein when capturing defined objects within the building, the respective defined object is specifically captured by the scanning device, and wherein attributes are assigned to the respective defined object via voice input.
[0006] In factory planning, the factory hall columns serve as reference points for various planning activities. Each column is marked with a clearly legible column number, consisting of a combination of letters and numbers. This column number is based on a hall grid spanning the entire hall, with the columns positioned at the grid's intersections. In planning, the hall column designation is frequently used to describe specific locations within the factory hall, for example, to locate problems. Currently, however, it is not possible to consistently use the hall columns as reference points across all planning systems because a system for capturing the columns' coordinates within the factory coordinate system is lacking. Nevertheless, 3D scans are available for all plants, encompassing both point clouds and panoramic images.
[0007] Detecting the columns in the point clouds presents a significant challenge for the following reasons. The extremely large data size of the point cloud necessitates complex processing. Furthermore, the point cloud is often noisy, which hinders reliable detection of the columns. In addition, other objects are often located around the columns, obstructing their detection. The resolution of the point cloud is too low to recognize the column labels. Finally, objects are sometimes incorrectly classified as columns or not detected at all. 25-1221 DE 3
[0008] The present invention is based on the objective of creating a method, a system and a computer program product that enable precise, automated localization of objects in 3D scans.
[0009] To solve the problem, a method with the features of claim 1, a system with the features of claim 9 and a computer program product with the features of claim 10 are proposed.
[0010] Advantageous configurations of the process are the subject of the dependent claims.
[0011] According to a first aspect of the invention, a method for the automated localization of objects using 3D scans comprising at least one panoramic image and at least one point cloud is proposed. In the method, in a first step, at least one object is identified in at least one panoramic image showing at least one area of the interior of a building, and the position of the identified object within the panoramic image is determined. In a second step, the position of the identified object in a 2D (two-dimensional) map of the interior is determined based on the identified position of the object. In a third step, a solid is spanned in a point cloud of the panoramic image at the determined position, and the coordinates of the object within the spanned solid are determined in the point cloud.
[0012] This method enables reliable, precise, and automated localization of an object by using a panoramic image in combination with a point cloud from a 3D scan. A significant advantage of this method is the ability to localize a large number of objects within 3D scans.
[0013] In this method, panoramic images, in particular georeferenced panoramic images, and a two-dimensional (2D) map generated from this data, in particular an automatically generated two-dimensional (2D) map, are advantageously used as input data for locating an object. 25-1221 DE 4
[0014] The object can be a column or a hall column. The building can be a factory hall. The method can also be used to locate objects outside of a building.
[0015] In an advantageous embodiment, at least one object is identified in the at least one panoramic image by means of object recognition and / or text recognition. Advantageously, several objects, for example, columns, can be identified in the panoramic image. Object recognition can be performed using image processing based on artificial intelligence. Text recognition can capture the object's number, for example, a column number. Text recognition can be performed using optical character recognition (OCR).
[0016] In an advantageous embodiment, to determine the position of the at least one identified object in the at least one panoramic image, an angular measure is defined. This measure comprises an angle whose vertex is a position at which the panoramic image was taken, and which extends from the vertex between a first ray that intersects an edge of the panoramic image and a second ray that intersects the identified object. Thus, the position of the object in the panoramic image is described by means of an angular measure. Advantageously, the panoramic image is a circular image covering 360°. Therefore, the angular measure can be between 0*TT and 2*TT.
[0017] In an advantageous embodiment, the distance of at least one object to a position where the panoramic image was taken is determined. This allows the distances of the object, for example a column, to the positions where panoramic images were taken to be evaluated, since these are georeferenced in the point clouds.
[0018] In an advantageous embodiment, the position, column number, angle measurement, and / or distance to the position where the panoramic image was taken are stored in a database for at least one object identified in at least one panoramic image. Thus, the following information can be stored for each panoramic image: Position im25-1221 DE 5
[0019] 3D space, object numbers, in particular column numbers, the angular dimensions of the objects, for example, the columns, and optionally the distance from the position where the panoramic image was taken to the object, for example, the column. In an advantageous embodiment, the angular dimension belonging to the at least one identified object is entered as a ray into the 2D map at each position where a panoramic image was taken, wherein an intersection of the rays corresponds to an x-coordinate and a y-coordinate of the identified object. Advantageously, the 2D map of the interior of the building, in particular the hall, is loaded from a database. On this map, the previously recorded angular dimensions of the objects, in particular the columns, are plotted as rays. The object, for example, the column, is located at the intersections of these rays, whereby the x- and y-coordinates of the object, for example, the column, can be roughly determined.
[0020] In an advantageous embodiment, the identification of at least one object is verified by checking it against a large number of panoramic images. The recognition of the object(s), for example, the columns, is verified by checking multiple panoramic images to identify incorrectly identified objects. Furthermore, the recognized object numbers can be verified by considering the numbers of neighboring objects. For example, column D9 is a neighbor of column D8. However, the D could also be interpreted as 0.
[0021] In an advantageous embodiment, a solid body with a predefined base area and a predefined height in the z-direction is spanned in the point cloud at the x-coordinate and the y-coordinate. The point cloud is loaded. At the detected x- and y-positions of the object, for example, the column, a solid body, such as a box with a base area of, for example, 2 m x 2 m, is spanned. The z-height of the box extends from the floor to the ceiling of the hall, so that the box roughly encloses the object, for example, the column. Within this box, the point cloud is analyzed to determine the object's position more precisely, in particular its x-, y-, and z-coordinates, thereby significantly narrowing the search space. 25-1221 DE 6
[0022] In an advantageous embodiment, a technical identification number, an object number, and / or the x, y, and z coordinates of the identified object are stored in a database. Consequently, the columns are recorded in a database with their technical ID, column number, and x, y, and z coordinates. Various applications can access this database to utilize the position of the hall columns, for example, to quickly navigate to a specific hall column in a 3D model of the factory.
[0023] According to a further aspect of the invention, a system for the automated localization of objects using 3D scans, comprising at least one panoramic image and at least one point cloud, is proposed. The system comprises at least one data processing unit that includes means for carrying out the method according to the invention.
[0024] The data processing device can be a computer or a mobile device, such as a tablet or a smartphone. The data processing device can include a storage device for storing the 3D scans, in particular the panoramic images and point clouds of the 3D scans, the 2D maps, and the object's position in 3D space, its angular measurement, its distance, its technical identification number, its object number, and / or its x, y, and z coordinates. The storage device can contain an object recognition program, a text recognition program, and / or a computer program product that is loaded into the processing unit and used to perform the steps of the method according to the invention.Furthermore, the data processing device can include a computing unit, for example a CPU, and an interface for connecting the data processing device to a database, such as a server. The database can contain the 3D scans, in particular the panoramic images and the point clouds of the 3D scans, and the 2D maps. The computing unit is advantageously configured to perform steps a) to c) of the method according to the invention. The data processing device can also include an input device, such as a keyboard and / or a mouse, and an output device, such as a monitor. 25-1221 DE 7.
[0025] According to another aspect of the invention, a computer program product is proposed which includes commands that cause the system according to the invention to execute the method according to the invention.
[0026] According to another aspect of the invention, a computer-readable storage medium on which a computer program product according to the invention is stored is proposed.
[0027] The following section describes a system and method for the automated localization of objects using 3D scans, along with further features and advantages, using an exemplary embodiment that is schematically represented in the figures. This shows:
[0028] Fig. 1 shows a schematic representation of a system for the automated localization of objects using 3D scans;
[0029] Fig. 2 shows a schematic representation of the process of a method for the automated localization of objects using 3D scans;
[0030] Fig. 3 shows a panoramic image of an area of the interior of a building, in which objects are identified using object and text recognition;
[0031] Fig. 4 a 2D map of the interior with rays marked;
[0032] and
[0033] Fig. 5 shows a point cloud from the panoramic image with solid bodies around the identified objects.
[0034] Fig. 1 shows a system 10 for the automated localization of objects 11 using 3D scans.
[0035] System 10 comprises a data processing unit 12 and a database 14, which are connected to each other via a data line 16, for example a data cable or wirelessly. 25-1221 DE 8
[0036] The data processing device 12 has a computing unit 18, for example a CPU, a storage device 20 for storing a program for object recognition, a program for text recognition and a computer program product for carrying out the procedure schematically shown in Fig. 2.
[0037] The data processing unit 12 also has an input device 22 in the form of a keyboard 24 and an output device 26 in the form of a monitor 28, which are connected to the computing unit 18 via data lines 16.
[0038] The database 14 is, in this case, a server 30 on which the 3D scans, including georeferenced panoramic images 32 and point clouds 34 of an interior 36 of a factory hall (not shown), as well as 2D maps 36 of the interior 36, which were generated from the panoramic images 32 and the point clouds 34, are stored. As can be seen particularly in Fig. 3, the panoramic images 32 cover a viewing angle of 360°, i.e., a range between O*TT and 2*TT.
[0039] As can be seen particularly in Figure 3, the objects to be identified are 11 columns (40) of the factory hall. In factory planning, the columns (40) of the factory hall serve as reference points for various planning activities. Each column (40) is marked with a clearly legible column number (42), which consists of a combination of letters and numbers. This column number (42) is based on a hall grid that spans the entire hall, with the columns (40) positioned at the grid's intersections. In planning, the column (40) number is frequently used to describe specific locations within the factory hall, for example, to locate problems.
[0040] The following section explains in more detail a method for the automated localization of objects using System 10, based on the method schematically depicted in Fig. 2. 25-1221 DE 9
[0041] As can be seen in Fig. 2, the procedure comprises three main steps, namely a first main step S1, which includes the identification of the columns 40 in the panoramic images 32, a second main step S2, which includes the determination of the position of the columns 40 in the 2D map 38, and a third main step S3, which includes the analysis of the point cloud 34.
[0042] The first main step S1 is divided into several sub-steps, as can be seen in Fig. 2. In a first sub-step S1.1, the panoramic images 32 are loaded from the server 30 into the computing unit 18.
[0043] In a second substep S1.2, the object recognition and text recognition programs stored in the memory unit 20 are loaded into the processing unit 18 to identify the columns 40 in the panoramic images 32. For this purpose, the column number 42 of the columns 40 is identified using text recognition. In the example schematically depicted in Fig. 3, the columns 40 have the numbers E9, E10 and D9.
[0044] Subsequently, in a third substep S1.3, the position of the identified columns 40 is described by an angular dimension. Each column 40 is assigned an angular dimension depending on its horizontal position on the panoramic image 32. For example, the right edge of the panoramic image 32 has an angular dimension of π*0 and the left edge 2*π. Thus, depending on the position of the column 40 on the panoramic image 32, a value between 0*π and 2*π is assigned. In the example shown in Fig. 3, column 40 numbered E9 has an angular dimension of 2.10 rad, column 40 numbered E10 has an angular dimension of 1.77 rad, and column 40 numbered D9 has an angular dimension of 0.64 rad.
[0045] Additionally, the distances D of the columns 40 to the position P can be determined, since the panoramic images 32 are georeferenced, as exemplified in Fig.
[0046] 3 is shown schematically in the 2D map 38 shown in the upper right.
[0047] In a fourth substep S1.4, the following information is stored for each column 40 in the storage device 20 and / or the server 30: the position of column 40, the column number 42 of column 40, an angle measurement and / or 25-1221 DE 10
[0048] a distance D to the position P where the panorama image 32 was taken.
[0049] In a fifth sub-step, S1.5, it is checked whether all panorama images 32 have been evaluated. If this is not the case, then those panorama images 32 that have not yet been evaluated are loaded into the processing unit 18, and sub-steps S1.1 to S1.4 are performed again for these panorama images 32. If all panorama images 32 have been evaluated, then the process proceeds to the second main step, S2.
[0050] The detection of columns 40 is verified by checking several panoramic images 32 to identify incorrectly detected columns 40. Furthermore, the detected column numbers 42 can be verified by examining the column numbers 42 of neighboring columns. For example, column D9 is a neighbor of column D8. However, the D could also be interpreted as 0.
[0051] The second main step S2 is also divided into several sub-steps, as can be seen in Fig. 2. In a first sub-step S2.1, the 2D map 38 of the factory hall is loaded from the server 30 into the computing unit 18.
[0052] In a second sub-step S2.2, starting from position P, where the panoramic image 32 was taken and which forms the vertex of the angle, the previously recorded angular dimensions of the columns 40 are plotted as rays S, optionally using the distances D, on the 2D map 38, as shown schematically in Fig. 4. The columns 40 are located at the intersection points ST of the rays S.
[0053] In a third sub-step S2.3, the x-coordinate and the y-coordinate of column 40 are roughly determined based on the intersection points ST.
[0054] In a fourth substep S2.4, the x-coordinate and y-coordinate for each column 40 are stored in the storage device 20 and / or the server 30. 25-1221 DE 11
[0055] The third main step S3 is also divided into several sub-steps, as can be seen in Fig. 2. In a first sub-step S3.1, the point cloud 34 is loaded into the processing unit 18.
[0056] In a second sub-step S3.2, solid bodies V in the form of boxes B with a base area of 2 m x 2 m are defined at the identified x and y coordinates of the columns 40, as schematically illustrated in Fig. 4. The height in the z-direction extends from the floor to the ceiling of the factory hall. Thus, the boxes B roughly enclose the columns 40. Within the box B, the point cloud 34 is analyzed to determine the position of the column 40 more precisely, in particular its x, y, and z coordinates, thereby significantly narrowing the search space.
[0057] In a third sub-step S3.3, each column 40 is recorded in a database 14 with a technical identification (ID), the column number 42 of column 40 and the x, y and z coordinates.
[0058] Various applications can access this database 14 to use the position of the columns 40, for example to quickly navigate to a specific column 40 in a 3D model of the factory hall.
[0059] The method enables reliable, precise and automated localization of columns 40 by using a panoramic image 32 in combination with a point cloud 34 of a 3D scan.-1221 DE 12
[0060] Reference symbol list
[0061] system
[0062] object
[0063] Data processing facility
[0064] database
[0065] Data line
[0066] computing unit
[0067] Storage device
[0068] Input device
[0069] Keyboard
[0070] Output device
[0071] monitor
[0072] server
[0073] Panoramic image
[0074] Point cloud
[0075] interior
[0076] 2D map
[0077] column
[0078] Column number
[0079] first main step
[0080] .1 first substep
[0081] .2 second substep
[0082] .3 third substep
[0083] .4 fourth substep
[0084] .5 fifth substep
[0085] second main step
[0086] .1 first substep
[0087] .2 second substep
[0088] .3 third substep25-1221 DE 13
[0089] S2.4 fourth step
[0090] S3 third main step 53.1 first substep 53.2 second substep 53.3 third substep
[0091] D distance
[0092] P Position
[0093] S beam
[0094] ST interface
[0095] V Solid body B Box
Claims
25-1221 DE 14 Claims 1. Method for the automated localization of objects (11) using 3D scans comprising at least one panoramic image (32) and at least one point cloud (34), wherein the method comprises the following steps: a. Identification of at least one object (11 ) in at least one panoramic image (34) showing at least one area of an interior space (36) of a building, and determination of a position of the identified object (11) in the panoramic image (32); b. Determining a position of the identified object (11) in a 2D map (38) of the interior (36) based on the identified position of the object (11); and c. Stretching a solid (V) in a point cloud (34) of the panorama image (32) at the determined position and determining the coordinates of the object (11) within the stretched solid (V) in the point cloud (34).
2. Method according to claim 1, characterized in that the at least one object (11) is identified in the at least one panoramic image (32) by means of object recognition and / or text recognition.
3. Method according to claim 1 or 2, characterized in that, to determine the position of the at least one identified object (11) in the at least one panoramic image (32), an angular measure is determined which comprises an angle whose vertex is a position (P) at which the panoramic image (32) was taken and which is defined starting from the vertex between a first ray that intersects an edge of the panoramic image (32) and a second ray that intersects the identified object (11). 25-1221 DE 15 4. Method according to one of the preceding claims, characterized in that a distance (D) of the at least one object (11) to a position (P) at which the panoramic image (32) was taken is determined.
5. Method according to one of the preceding claims, characterized in that the position, column number (42), angle measurement and / or distance (D) to the position (P) at which the panoramic image (32) was taken is stored in a database (14) for the at least one object (11) identified in the at least one panoramic image (32).
6. Method according to one of claims 3 to 5, characterized in that the angular measure belonging to the at least one identified object (11) is entered as a ray (S) into the 2D map (38) at each position (P) where a panoramic image (P) was taken, wherein an intersection point (ST) of the rays (S) corresponds to an x-coordinate and a y-coordinate of the identified object (11).
7. Method according to claim 6, characterized in that a solid body (V) with a predetermined base area and a predetermined height in the z-direction is spanned in the point cloud (34) at the x-coordinate and the y-coordinate.
8. Method according to claim 7, characterized in that a technical identification number, an object number and / or the x, y and z coordinates are stored in a database (14) for the identified object (11).
9. System (10) for the automated localization of objects (11) by means of 3D scans comprising at least one panoramic image (32) and at least one point cloud (34), comprising at least one data processing device (12) comprising means for carrying out the method according to any one of claims 1 to 8. 25-1221 DE 16 10. Computer program product comprising instructions that cause the system (10) according to claim 9 to execute the method according to any one of claims 1 to 8.
