Apparatus or method for estimating position of robot

The method uses a sensor-based semantic point cloud comparison with a pre-created map to overcome GPS errors and mapping costs, providing accurate robot positioning in diverse environments.

WO2026235217A1PCT designated stage Publication Date: 2026-11-12LG ELECTRONICS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2025/006157
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing autonomous driving technologies face limitations due to GPS-based positioning errors in urban areas and the high costs and difficulties in updating pre-surveyed maps, especially in complex environments with dynamic obstacles.

Method used

A robot position estimation method using a sensor group comprising an image sensor or LiDAR sensor to generate a semantic point cloud, comparing it with a previously created map to estimate position and orientation, utilizing rotation-invariant and rotation-variable semantic class feature information for precise positioning.

Benefits of technology

Enables accurate and cost-effective positioning of robots without GPS signals, reducing the need for extensive mapping and periodic updates, suitable for both indoor and outdoor environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2025006157_12112026_PF_FP_ABST
    Figure KR2025006157_12112026_PF_FP_ABST
Patent Text Reader

Abstract

According to one embodiment of the present invention, an apparatus for estimating a position of a robot is proposed. The apparatus may comprise: a sensor group including an image sensor or a LiDAR sensor; and a processor configured to obtain, from sensor data acquired by the sensor group, a semantic point cloud corresponding to a region of a predetermined size surrounding the robot, and to estimate the position of the robot by comparing the obtained semantic point cloud corresponding to the region surrounding the robot with semantic point clouds corresponding to a plurality of candidate regions each having the predetermined size.
Need to check novelty before this filing date? Find Prior Art

Description

Device for position estimation for a robot or method for the same

[0001] The present invention relates to an apparatus or method for position estimation for a robot, and more specifically, to an apparatus or method configured to perform position estimation of a robot using a pre-created map.

[0002] Autonomous driving technology is continuously evolving to improve the performance of robots moving in various environments. Generally, autonomous robots determine their location and plan routes by utilizing pre-surveyed mapping information or by using the Global Positioning System (GPS) and Global Navigation Satellite System (GNSS). However, existing technologies have several limitations.

[0003] First, the reliability of the GPS (GNSS) method can be degraded in urban areas due to signal zoning issues. As illustrated in Fig. 1, between high-rise buildings, satellite signals are blocked or positional accuracy is reduced due to multipath effects. Furthermore, the positional error range of GPS itself is generally in the range of a few meters, which can cause errors in narrow spaces or complex environments where precise driving is required, thereby increasing the risk of collision. Therefore, there are limitations to performing high-precision autonomous driving using GPS alone.

[0004] Second, to overcome the limitations of GPS or GNSS mentioned above, the method of utilizing pre-surveyed map information requires surveying extensive areas during the map construction process, which results in excessive time and cost. Such map data requires periodic updates in response to environmental changes, which significantly increases maintenance costs. In particular, there is a problem in that accurate driving is difficult using only pre-surveyed maps in indoor environments or environments with many dynamic obstacles.

[0005] As such, existing autonomous driving robot technology is showing limitations due to issues such as GPS-based positioning errors or limitations, high costs, and difficulties in updating maps. Accordingly, new technology is required to enable more efficient and precise autonomous driving.

[0006] The present invention aims to provide a device for estimating the position or orientation of a robot and a control method for the same.

[0007] According to one embodiment of the present invention, a robot position estimation device is proposed, wherein the device may include a sensor group comprising an image sensor or a LiDAR sensor; and a processor configured to obtain a semantic point cloud for an area of ​​a preset size surrounding the robot from sensor data obtained through the sensor group, and to estimate the position of the robot by comparing the obtained semantic point cloud surrounding the robot with a plurality of candidate areas having the preset size. The plurality of candidate areas may include only candidate areas where the robot can move, and the semantic point cloud of the plurality of candidate areas may include semantic class information for each point or each point group, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information obtained from a previously created original map.

[0008] Additionally or alternatively, the processor may be configured to extract the semantic class of the robot's current position, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information from the semantic point cloud around the acquired robot.

[0009] Additionally or alternatively, the processor may be configured to obtain a first-level set of candidate regions by selecting, from among the semantic point clouds of the plurality of candidate regions, a semantic point cloud of a candidate region having the same semantic class as the semantic class of the robot's current position.

[0010] Additionally or alternatively, the processor may be configured to obtain a second level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the first level set of candidate regions, a semantic point cloud of a candidate region having rotation-invariant semantic class feature information similar to the rotation-invariant semantic class feature information extracted from the semantic point cloud around the robot.

[0011] Additionally or alternatively, the processor may be configured to obtain a third-level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the second-level set of candidate regions, a semantic point cloud of a candidate region having rotationally variable semantic class feature information similar to the rotationally variable semantic class feature information extracted from the semantic point cloud around the robot.

[0012] Additionally or alternatively, the processor may be configured to select a preset number (N1) of candidate regions included in the set of candidate regions of the second level.

[0013] Additionally or alternatively, the processor may be configured to select a preset number (N2) of candidate regions included in the set of candidate regions of the third level.

[0014] Additionally or alternatively, the processor may be configured to obtain primary estimated position and orientation information of the robot based on rotational variable semantic class feature information of a semantic point cloud of candidate regions belonging to the set of candidate regions of the third level.

[0015] Additionally or alternatively, the processor may be configured to obtain secondary estimated position and orientation information of the robot by comparing each point of the semantic point group around the acquired robot with each point of the semantic point group of the candidate region belonging to the set of candidate regions of the third level one-to-one.

[0016] Additionally or alternatively, the processor may be configured to re-acquire the second level candidate region set when the similarity between the extracted rotationally variable semantic feature information and the rotationally variable semantic class feature information of the semantic point cloud of the third level candidate region set is below a threshold.

[0017] Additionally or alternatively, the processor may be configured to lower the similarity threshold for the rotation-invariant semantic class feature information or increase the number of candidate regions belonging to the second-level candidate region set when reacquiring the second-level candidate region set.

[0018] Additionally or alternatively, the rotation-invariant semantic class feature information may include semantic class ratio information in the area of ​​the preset size.

[0019] Additionally or alternatively, the rotationally variable semantic class feature information may include semantic class information in a preset angle range and distance range in the preset size area.

[0020] According to another embodiment of the present invention, a method for estimating the position of a robot is proposed, the method comprising: a step of obtaining a semantic point cloud for an area of ​​a preset size around the robot from sensor data obtained through one or more sensors; and a step of estimating the position of the robot by comparing the obtained semantic point cloud around the robot with a semantic point cloud of a plurality of candidate areas having the preset size, wherein the plurality of candidate areas include only candidate areas where the robot can move, and the semantic point cloud of the plurality of candidate areas may include semantic class information for each point or each point cloud, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information obtained from a previously created original map.

[0021] Additionally or alternatively, the method may include the step of extracting semantic class of the robot's current position, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information from the semantic point cloud around the robot acquired.

[0022] Additionally or alternatively, the method may include the step of obtaining a first-level set of candidate regions by selecting, from among the semantic point clouds of the plurality of candidate regions, a semantic point cloud of a candidate region having the same semantic class as the semantic class of the robot's current position.

[0023] Additionally or alternatively, the method may include the step of obtaining a second-level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the first-level set of candidate regions, a semantic point cloud of a candidate region having rotation-invariant semantic class feature information similar to the rotation-invariant semantic class feature information extracted from the semantic point cloud around the robot.

[0024] Additionally or alternatively, the method may include the step of obtaining a third-level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the second-level set of candidate regions, a point cloud of candidate regions having rotationally variable semantic class feature information similar to the rotationally variable semantic class feature information extracted from the semantic point cloud around the robot.

[0025] Additionally or alternatively, the method may include the step of obtaining the final estimated position and orientation information of the robot by comparing each point of the semantic point group around the robot obtained above with each point of the semantic point group of the candidate region belonging to the set of candidate regions of the third level one-to-one.

[0026] According to another embodiment of the present invention, a computer-readable medium is proposed that stores code configured to execute a robot position estimation method by a computer or processor.

[0027] The above-mentioned problem-solving methods are merely some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by those skilled in the art based on the detailed description of the present invention to be described below.

[0028] The present invention has the following technical effects.

[0029] Since the present invention utilizes a previously created original map or its data, the position or direction of a robot can be obtained at low cost without a GPS signal.

[0030] In addition, the present invention does not require prior mapping of the area where the robot can move.

[0031] The effects according to the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the following detailed description of the invention.

[0032] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and explain the technical concept of the present invention together with the detailed description.

[0033] Figure 1 illustrates the problem of positioning based on GPS signals.

[0034] Figure 2 shows a semantic map of the entire candidate region obtained from a previously created original map according to the present invention.

[0035] FIG. 3 illustrates a plurality of candidate regions according to the present invention.

[0036] FIG. 4 illustrates a method or procedure for obtaining a semantic point group according to the present invention.

[0037] FIG. 5 illustrates a specific method or procedure for obtaining a semantic point group according to the present invention.

[0038] FIG. 6 illustrates three-dimensional and two-dimensional point cloud data according to the present invention.

[0039] FIGS. 7 and 8 illustrate a specific method or procedure for obtaining a semantic point group according to the present invention.

[0040] FIG. 9 illustrates a matching procedure between a semantic point cloud for a candidate region and a semantic point cloud for a robot surrounding region according to the present invention.

[0041] FIG. 10 illustrates a flowchart of a method for estimating the position or orientation of a robot according to the present invention.

[0042] FIG. 11 illustrates a block diagram of a device for estimating the position or orientation of a robot according to the present invention.

[0043] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0044] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0045] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0046] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0047] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0048]

[0049] FIG. 2 shows a semantic map of all candidate regions obtained from a previously created original map according to the present invention. It can be obtained sequentially from FIG. 2(a), FIG. 2(b), and FIG. 2(c). In this specification, the semantic map may include information in which feature points, such as semantic classes associated with the regions, are assigned to at least some regions of the original map.

[0050] FIG. 2(a) illustrates the original map of a pre-set area. The present invention uses the original map created earlier without creating a separate map for estimating the position or orientation of a robot.

[0051] In this specification, the original map includes a map in which semantic class distinctions, such as roads, buildings, sidewalks, crosswalks, tunnels, and bridges, which are generally widely used in outdoor spaces nationwide and are accessible or available to anyone, are indicated in a polygonal form. However, since the original map cannot be recognized or understood by a robot, it is necessary to obtain a semantic map from the original map and its data.

[0052] Figure 2(a) is for a portion of the original map corresponding to 2km*3km, and the size of the area does not limit the invention.

[0053] FIG. 2(b) shows a sampled point cloud in which a point is sampled from the original map in the form of a point cloud with uniform intervals, which is advantageous for feature processing and exploration, and a semantic class is assigned to each point. FIG. 2(b) was obtained by sampling the original map of FIG. 2(a) at intervals of 0.5 m, and the sampling interval does not limit the invention. FIG. 2(b) is a part of FIG. 2(a) and shows the sampling results for a part of the area. In addition, each point within each point cloud has Universal Transverse Mercator (UTM) coordinates (X,Y coordinates).

[0054] Additionally, in FIG. 2(b), semantic classes usable for position or direction estimation can be extracted from the original map and mapped to each point group or each point. Although the original images in FIG. 2 are all color images, they are represented in grayscale because it is impossible to submit color images as drawings in international applications; however, in the original images of FIG. 2(b) and (c), each class is mapped and displayed in different colors. FIG. 2(b) is assigned four semantic classes, which are roadway, sidewalk, crosswalk, and building, and one of the four semantic classes is assigned to each point. In FIG. 2(b) and (c), each semantic class is indicated with different hatching. Unlike conventional autonomous driving maps for vehicles, the present invention includes not only roadways but also sidewalks and crosswalks as semantic classes, thereby including a space where a robot can drive or move. The number of semantic classes used in the present invention also does not limit the invention. Areas where the robot cannot drive or move are represented in white (no hatching) on ​​the drawing.

[0055] FIG. 2(c) shows the result of filtering semantic classes for the same area as FIG. 2(a). That is, it shows the class corresponding to a location where a robot can move or drive among the semantic classes, and the result of sampling those locations at uniform intervals. Among the four semantic classes mentioned earlier, buildings, which are areas where a robot cannot drive or move, were excluded. Movement within a building does not fall within the scope of the present invention. Additionally, FIG. 2(c) shows the result of sampling at 2m intervals.

[0056] As shown in FIG. 2(c), only regions mapped with specific hatching (i.e., points or point groups having specific semantic classes), that is, regions where the robot can drive or move, can be candidate regions for estimating the position or orientation of the robot described later. Similar to FIG. 2(b), in the original image of FIG. 2(c), each class was mapped and displayed in different colors; accordingly, in FIG. 2(c), regions where the robot can drive or move were indicated by hatching, and regions where the robot cannot drive or move were indicated in white. However, due to the drawing scale, unlike FIG. 2(b), it was not possible to display different hatchings corresponding to the four semantic classes.

[0057]

[0058] FIG. 3 illustrates a plurality of candidate regions according to the present invention.

[0059] A semantic point cloud with some semantic classes filtered out, such as sampling as in (c) of FIG. 2, is obtained and can be referred to as the “entire candidate region.” The entire candidate region can be sampled into sub-regions of a preset size. Accordingly, the entire candidate region can be divided or extracted into multiple sub-regions of a preset size, and hereinafter, multiple sub-regions of a preset size will be referred to as “candidate regions.”

[0060] In other words, the entire candidate region can be divided or extracted into multiple candidate regions. However, multiple candidate regions may overlap with neighboring candidate regions; for example, the candidate regions may be set such that the center of each candidate region is separated from the center of a neighboring candidate region by an actual distance of 0.5m. The distance between the centers of each candidate region does not limit the present invention. The center of each candidate region may correspond to a candidate location of a robot. Furthermore, although each candidate region is set to a radius of 20m in the present invention, the size of the candidate region also does not limit the present invention.

[0061] Figure 3 shows a map of multiple candidate regions obtained in this way, and various feature points can be extracted or obtained for each candidate region.

[0062] To explain in more detail, by extracting semantic point clouds of individual candidate regions from a semantic map sampled in the form of a point cloud, it is possible to pre-extract semantic point clouds and feature points that are observable when the actual robot is near the candidate region. That is, the candidate region may be one of the regions where the robot will move or drive, and accordingly, the semantic point clouds of the candidate region correspond to the semantic point clouds of the regions around the robot acquired while the robot is moving or driving. Therefore, semantic point clouds for each candidate region can be pre-acquired or extracted and stored in a storage medium such as the robot's internal memory, or in a storage medium of an external device such as a server that the robot can access.

[0063] After semantic point clouds are extracted for each candidate region, various types of semantic feature points can be pre-computed using the semantic point clouds as base data, depending on the target computational load, positioning accuracy, or the type of matching algorithm to be applied.

[0064] In the present invention, four semantic feature points are defined to achieve computational efficiency while sequentially reducing the size of candidate regions, but various additional semantic feature points may be added as needed.

[0065] 1. Semantic Class: The semantic class with the highest number among the nearest class labels around the robot is set as the semantic class of the current robot position, and candidate regions identical to this are selected.

[0066] (Example: If the robot is determined to be located on the sidewalk, candidate locations that are not sidewalks, such as crosswalks or roadways, can be excluded.)

[0067] 2. Rotation-invariant feature points: Feature points independent of the robot's facing direction (yaw). While this results in low matching accuracy and makes orientation estimation impossible when applied to positioning, it has the advantage of low computational load.

[0068] (Example: Class distribution ratio of the entire surrounding semantic point cloud, or class distribution ratio by distance)

[0069] 3. Rotationally Variable Feature Points: Feature points that vary depending on the direction (yaw) the robot is facing. These offer relatively high matching accuracy and enable orientation estimation when applied to positioning, but require a large amount of computation.

[0070] (Example: Class distribution or proportion for each section when the area around a robot is divided into sections of a certain size based on two axes of direction and distance.)

[0071] When the surrounding point cloud is divided every 2m from the reference point and by 6 degrees in each direction, the class distribution or proportion for 20 (distance) * 60 (angle) = 1200 sections based on a candidate region radius of 20m)

[0072] 4. Original Semantic Point Cloud: This method directly utilizes the surrounding semantic point cloud itself for matching, offering the highest matching accuracy and the largest computational load. It is also a rotationally variable feature point.

[0073] (The number of semantic point clouds within a candidate region can range from thousands to tens of thousands depending on the spatial shape.)

[0074] In addition, additional semantic feature points may be established in each candidate region, and these may include feature points that can be information identifiable by a sensor or unique identification information of a specific candidate region.

[0075]

[0076] FIG. 4 illustrates a method or procedure for obtaining a semantic point cloud of a robot surrounding area according to the present invention.

[0077] The present invention estimates the position or orientation of a robot using semantic feature information obtained from sensor data, rather than directly using location information such as GPS signals. Therefore, it is necessary to obtain a semantic point cloud around the robot to attempt comparison or matching with the semantic point cloud (reference data) of the semantic map or the entire candidate region described above. The location information of the candidate region with the highest similarity to the semantic point cloud around the robot corresponding to the robot's current position can ultimately be estimated as the robot's current position.

[0078] The results of perceiving the surrounding environment can be converted into a semantic point cloud through sensors capable of perceiving three-dimensional structures, such as image sensors (i.e., cameras) or LiDAR sensors mounted on the robot, or through the fusion of multiple sensors. As such, the semantic point cloud around the robot is not information obtained based on the original map or semantic map.

[0079] It is desirable to obtain the semantic point cloud around the robot for an area around the robot that has a preset size. As explained with reference to FIG. 3, since the entire candidate area was divided or extracted into multiple candidate areas, it is desirable to set the target area of ​​the semantic point cloud around the robot to be the same size as the multiple candidate areas.

[0080] To this end, a robot or a sensor system of the robot, etc., may be configured to generate a semantic point cloud (300) by combining semantic segmentation (100) and geometric analysis (200). In this case, semantic segmentation is performed using an image acquired through an image sensor, and geometric analysis is performed using sensor values ​​acquired through a LiDAR sensor. This is because, in the case of an image sensor, the acquisition of distance information to an object is inaccurate and it is difficult to recognize a three-dimensional structure.

[0081] The semantic point cloud around the robot or the extraction or acquisition of its feature points can be performed in the same manner as the acquisition of the semantic map described above or the extraction or acquisition of the semantic point cloud and its feature points for the entire candidate region.

[0082] In addition, semantic point clouds around the robot and their feature points can be acquired whenever the robot moves a certain distance.

[0083] Meanwhile, as another embodiment, a semantic point cloud may be generated using either an image sensor or a LiDAR sensor. This will be described later.

[0084]

[0085] FIGS. 5, 7 to 8 illustrate a specific method or procedure for obtaining a semantic point group or feature points around a robot according to the present invention.

[0086] FIG. 5 illustrates a method for extracting semantic point clouds and their feature points by fusing an image sensor and a LiDAR sensor. The following method may be performed by a robot, a device or system for estimating the position or orientation of a robot, or a processor thereof. Hereinafter, it will be described that a “robot” performs the method.

[0087] The robot can obtain semantic segmentation results from a 2D image acquired through an image sensor using a deep learning model that performs semantic segmentation, and can extract four types of semantic classes (roadway, sidewalk, crosswalk, building) mentioned above to be used for semantic matching from the results (101).

[0088] The robot can obtain a 3D point cloud of the surrounding environment from a LiDAR sensor and then extract a plane through geometric analysis of the point cloud (201). This is intended to remove noise such as dynamic objects and to robustly extract horizontal planes (roadways, sidewalks, crosswalks) visible from an aerial view of the map and building structures that are generally composed mainly of planes.

[0089] Afterward, the robot can fuse the two results (101, 102) above to generate a three-dimensional semantic point cloud (301), and convert it into an aerial view (or bird's-eye view) that is a view looking down from above, thereby converting it into a two-dimensional semantic point cloud that is the same view as the map (S302). FIG. 6 (a) is an example of a three-dimensional semantic point cloud, and FIG. 6 (b) is an example of a three-dimensional semantic point cloud converted into a two-dimensional semantic point cloud.

[0090] The robot can additionally extract various semantic feature points from the semantic point cloud (303).

[0091]

[0092] FIGS. 7 and FIGS. 8 illustrate a method for acquiring a semantic point cloud or its feature points using a LiDAR sensor or an image sensor, respectively. FIGS. 7 and FIGS. 8 are intended to illustrate an embodiment that can replace the process of generating a semantic point cloud from the robot's surrounding environment described in FIG. 5.

[0093] The following method may be performed by a robot, a device or system for estimating the position or orientation of a robot, or a processor thereof. Hereinafter, it will be described that a “robot” performs the said method.

[0094] In the embodiment described with reference to FIG. 5, a semantic point cloud around the robot was generated by fusing the semantic segmentation result of a 2D image acquired through an image sensor with a 3D point cloud acquired through a LiDAR sensor; however, without being restricted to this methodology, semantic point clouds can be obtained in various ways, such as direct semantic segmentation of the original LiDAR point cloud, SfM (Structure from Motion), VIO (Visual-Inertial Odometry) for the 2D image, and 3D point cloud generation techniques utilizing deep learning-based depth estimation techniques.

[0095] Referring to FIG. 7, the robot acquires a 3D point cloud through a LiDAR sensor (311), and from this, can acquire a 3D semantic segmentation result and a 3D semantic point cloud (312). The robot can convert the 3D semantic point cloud into an aerial view (or bird's-eye view) and convert it into a 2D semantic point cloud with the same view as the map (S313). The robot can additionally extract various semantic feature points from the semantic point cloud (314).

[0096] Referring to Fig. 8, a method for obtaining a three-dimensional semantic point cloud using a two-dimensional image obtained through an image sensor is presented.

[0097] The robot can obtain semantic segmentation results from a 2D image obtained through an image sensor using a deep learning model that performs semantic segmentation, and can extract the four types of classes mentioned above (roadway, sidewalk, crosswalk, building) to be used for semantic matching from the results (101).

[0098] The robot can convert 2D image data into 3D data by applying techniques such as SfM, VIO, and Depth Estimation to the acquired 2D image (211).

[0099] Afterwards, the robot can fuse the two results (101, 211) above to generate a 3D semantic point cloud (321), and convert it into an aerial view (or bird's-eye view) that is a view looking down from above, and convert it into a 2D semantic point cloud that is the same view as the map (S322).

[0100] The robot can additionally extract various semantic feature points from the semantic point cloud (323).

[0101] As described above, there may be various methods for generating semantic point clouds of the robot's surrounding area depending on various conditions, such as the type of sensor adopted in the robot or computational constraints.

[0102]

[0103] FIG. 9 illustrates a matching procedure between a point cloud for a candidate region and a point cloud for a region surrounding a robot according to the present invention. The following procedure may be performed by a robot, a device or system for estimating the position or orientation of a robot, or a processor thereof. Hereinafter, it will be described that a “robot” performs the procedure.

[0104] The matching procedure illustrated in Fig. 9 proposes a computationally efficient position estimation through stepwise matching of the semantic point cloud of the robot's surrounding area observed by the robot and the semantic point cloud of the candidate area based on the original map.

[0105] Using the matching procedure illustrated in FIG. 9, the present invention estimates the most similar location as the current location of the robot by comparing the semantic point cloud of the robot's surrounding environment with the semantic point cloud of the previously generated candidate region. Since the space where the robot can drive or move (hereinafter referred to as the “service target space”) is very large and there are many candidate regions, a method is adopted to reduce the candidate region step by step by matching the semantic point cloud of the robot's surrounding region with the semantic point cloud of the candidate region based on the original map using various types of semantic feature points for real-time location estimation of the robot.

[0106] The matching methodology proposed in the present invention consists of four steps and can be defined as follows. The matching of the following four steps corresponds to the four semantic feature point information described with reference to FIG. 3.

[0107] Meanwhile, as explained above, prior to the four steps described below, candidate regions consisting only of points corresponding to classes where the robot cannot drive or move may be excluded from the matching target. In this regard, please refer to the explanation of (c) in FIG. 2.

[0108]

[0109] 1st Matching (S100)

[0110] The first matching (S100) involves checking whether the semantic class of a candidate region matches the semantic class of a region surrounding the robot. For example, if it is determined that the robot is located on a sidewalk, candidate regions whose class information does not include a sidewalk can be excluded from the matching target. That is, as a result of the first matching, a set of candidate regions of the first level can be obtained.

[0111] 2nd Matching (S200)

[0112] Second matching (S200) involves checking whether rotation-invariant feature point information of candidate regions and rotation-invariant feature point information of robot surrounding regions are similar. For example, the class distribution ratio of point clouds of all candidate regions belonging to the first level of candidate region set is compared with the class distribution ratio of point clouds of robot surrounding regions, and if it falls outside a preset similarity range, the corresponding candidate region can be excluded from the matching target. That is, as a result of the second matching, a second level of candidate region set can be obtained.

[0113] 3rd Matching (S300)

[0114] Third matching (S300) involves checking whether the rotation-invariant feature point information of the candidate region and the rotation-variable feature point information of the robot's surrounding region are similar. For example,

[0115] The class information by angle and distance of the point clouds of all candidate regions belonging to the second-level candidate region set is compared with the class information by angle and distance of the point clouds of the robot surrounding region, and if it falls outside a preset similarity range, the corresponding candidate region can be excluded. As a result of the third-level matching, a very small number of candidate regions similar to the robot surrounding region can be obtained (e.g., 1). Accordingly, as a result of the third-level matching, approximate position or orientation information of the robot (i.e., first-level estimated position or orientation information) can be obtained. That is, as a result of the third-level matching, a third-level candidate region set can be obtained.

[0116] Meanwhile, through the results of the third matching, candidate regions within a preset similarity range with respect to the robot's surrounding region may not be extracted. In this case, since candidate regions corresponding to the robot's surrounding region were excluded (filtered) in the second matching, the possibility of an error in the second matching cannot be ruled out. Accordingly, the second matching may be retried, but the similarity range used in the second matching may be relaxed. That is, the criteria for similarity judgment may be relaxed so that a larger number of candidate regions are left (so that a smaller number of candidate regions are excluded). Alternatively, the number of candidate regions belonging to the second level of candidate region set obtained as a result of the second matching may be configured to be set to a preset number (N1).

[0117] Alternatively, the second matching may not be retried, but the third matching may be retried, and the similarity range used in the third matching may be more relaxed. That is, the criteria for similarity judgment may be relaxed so that a larger number of candidate regions are left (so that a smaller number of candidate regions are excluded). Alternatively, the number of candidate regions belonging to the third level of the set of candidate regions obtained as a result of the third matching may be configured to be set to a preset number (N2).

[0118]

[0119] 4th Matching (S400)

[0120] Fourth-order matching (S400) involves directly comparing the original semantic point cloud with the semantic point cloud of the robot's surrounding area point by point. For example, using the ICP (Iterative Closest Point) algorithm, the semantic point cloud of all candidate regions belonging to the third-level set of candidate regions can be aligned or matched by comparing each point of the semantic point cloud of the robot's surrounding area with each other. Accordingly, as a result of the fourth-order matching, more accurate position or orientation information of the robot (i.e., second-order estimated position or orientation information) can be obtained, and the second-order estimated position and orientation information may be closer to the actual position of the robot than the first-order estimated position and orientation information corresponding to the result of the third-order matching.

[0121] In conclusion, the number of candidate regions can be reduced through first or second-order matching, and the position or orientation of the robot can be estimated through third or fourth-order matching, thereby simultaneously improving the accuracy and efficiency (speed) of the computation for estimating the position or orientation of the robot.

[0122]

[0123] FIG. 10 illustrates a flowchart of a method for estimating the position or orientation of a robot according to the present invention. The following method may be performed by a robot, a device or system for estimating the position or orientation of a robot, or a processor thereof. Hereinafter, it will be described that a “robot” performs the method.

[0124] A robot may be configured to acquire semantic point clouds of multiple candidate regions (S1010). Semantic point clouds of multiple candidate regions are acquired from a previously created original map and may include semantic class information, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information for each point or point group acquired therefrom. The multiple candidate regions may not include candidate regions composed only of points having a class in which the robot cannot move or drive according to the semantic class. That is, the multiple candidate regions may include only candidate regions in which the robot can move or drive.

[0125] The robot may be configured to acquire a semantic point cloud for an area of ​​a preset size around the robot from sensor data acquired through one or more sensors (S1020).

[0126] Then, the robot can be configured to compare the semantic point cloud of a preset size area around the robot acquired with the semantic point cloud of a plurality of candidate areas (S1030).

[0127] The comparison of the semantic point cloud of the region around the robot and the semantic point cloud of multiple candidate regions can be performed through the four-step matching described above.

[0128]

[0129] FIG. 11 illustrates a block diagram of a device for estimating the position or orientation of a robot according to the present invention.

[0130] A device (100) for estimating the position or orientation of a robot (hereinafter referred to as “device (100)”) may include a sensor group (1100) composed of one or more sensors.

[0131] The device (100) may include a memory (1200) configured to store an algorithm for estimating the position or orientation of a robot, a code for acquiring a semantic point cloud around the robot, or a code for detecting a candidate region with the highest similarity to the semantic point cloud around the robot by comparing the semantic point cloud around the robot with the semantic point cloud of a plurality of candidate regions.

[0132] The device (100) may include a processor (1300) configured to execute code stored in memory (1200).

[0133] The processor (1300) may be configured to obtain a semantic point cloud for an area of ​​a preset size around the robot from sensor data obtained through a sensor group. Additionally, the processor (1300) may be configured to estimate the position of the robot by comparing the obtained semantic point cloud around the robot with a semantic point cloud of a plurality of candidate areas having a preset size.

[0134] Multiple candidate regions may include only candidate regions where the robot can move. Additionally, the semantic point clouds of the multiple candidate regions may include semantic class information for each point or each point group, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information obtained from a previously created original map.

[0135] The processor (1300) may be configured to extract the semantic class of the point or point group corresponding to the current position of the robot or the current position, rotation invariant semantic class feature information, or rotation variable semantic class feature information from the semantic point group around the robot acquired.

[0136] The processor (1300) may be configured to obtain a first-level set of candidate regions by selecting, from among a plurality of candidate regions, a semantic point cloud of a candidate region having the same semantic class as the semantic class of the robot's current position.

[0137] The processor (1300) may be configured to obtain a second-level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to a first-level set of candidate regions, a semantic point cloud of a candidate region having rotation-invariant semantic class feature information similar to rotation-invariant semantic class feature information extracted from a semantic point cloud around the robot. The rotation-invariant semantic class feature information may include semantic class ratio information in a region of a preset size. The processor (1300) may be configured to select a preset number (N1) of candidate regions included in the second-level set of candidate regions.

[0138] Additionally, the processor (1300) may be configured to obtain a third-level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the second-level set of candidate regions, a semantic point cloud of a candidate region having rotationally variable semantic class feature information similar to the rotationally variable semantic class feature information extracted from the semantic point cloud around the robot. The rotationally variable semantic class feature information may include semantic class information in a preset angle range and distance range within a preset size region. The processor (1300) may be configured to select a preset number (N2) of candidate regions included in the third-level set of candidate regions.

[0139] The processor (1300) may be configured to obtain primary estimated position and orientation information of the robot based on rotational variable semantic class feature information of a semantic point group of candidate regions belonging to a third-level set of candidate regions. Additionally, the processor (1300) may be configured to obtain secondary estimated position and orientation information of the robot by comparing each point of the semantic point group around the robot obtained with each point of the semantic point group of candidate regions belonging to a third-level set of candidate regions one-to-one.

[0140] The processor (1300) may be configured to re-acquire a second-level set of candidate regions when the similarity between rotation-variable semantic feature information extracted from a semantic point cloud around the robot and rotation-variable semantic class feature information of a point cloud of candidate regions belonging to a third-level set of candidate regions is below a threshold. In re-acquiring a second-level set of candidate regions, the processor (1300) may be configured to lower the similarity threshold for rotation-invariant semantic class feature information or increase the number of candidate regions belonging to a second-level set of candidate regions.

[0141] Additionally, the device (100) may further include a transceiver (1400) for transmitting position or orientation information of the robot.

[0142]

[0143] Meanwhile, even if not described with reference to FIG. 11, the robot (100) of the present invention may be able to perform the operation according to the present invention as described above in FIG. 2 to 10.

[0144] According to the present invention, information on a semantic point cloud that is understandable by a robot or a corresponding map can be obtained by utilizing a general original map that is not constructed exclusively for robots.

[0145] In addition, according to the present invention, the cost of constructing and managing a precise map of a wide range of outdoor robot service target spaces can be reduced.

[0146] In addition, the present invention can be differentiated from existing high-definition (HD) maps for vehicles, which are unsuitable for robot driving because they are primarily created for roadways, by including most outdoor spaces where robots can drive, such as roadways, sidewalks, and crosswalks, in the location candidate pool when converted into a semantic map.

[0147] In addition, the present invention enables the additional extraction of various types of features or feature points that can be utilized for multiple purposes by converting a general map into a semantic point cloud form to which semantic classes are assigned.

[0148] In addition, the present invention can ensure the robustness of the positioning technique by selecting classes such as roads, sidewalks, crosswalks, and buildings as targets, which have a high observation frequency and large scale in outdoor urban areas and undergo little change over time.

[0149] In addition, outdoor spatial maps produced by public institutions or individual companies generating geographic information data are generally updated on a quarterly, semi-annual, or annual basis, which has the advantage of periodically reflecting large-scale environmental changes sufficient for semantic matching.

[0150] In addition, if there is no original map to which the necessary type of semantic class is assigned for the target space, the present invention can obtain the original map defined in the present invention by performing semantic segmentation on the aerial photograph to assign the necessary semantic class.

[0151] In addition, the present invention can perform positioning without prior investigation in outdoor sections by defining, extracting, and using various types of four-stage semantic features for each location to perform matching.

[0152] In addition, by converting the surrounding environment perceived by the robot through sensors such as cameras into a semantic point cloud, extracting various types of semantic feature points from it, and matching them with semantic feature points for the previously generated semantic map and location candidate groups, high-accuracy positioning can be performed across outdoor sections nationwide without costly preliminary surveys.

[0153] In addition, as an example of the present invention, a semantic point cloud around the robot was generated by fusing the semantic segmentation result of a 2D image acquired by a camera with a LiDAR point cloud; however, without being restricted to this methodology, a semantic point cloud can be obtained in various ways, such as direct semantic segmentation of the original LiDAR point cloud, and 3D point cloud generation techniques utilizing SfM, VIO, and deep learning-based depth estimation techniques for the 2D image.

[0154]

[0155] In addition, as another aspect of the present invention, the operation of the above-described proposal or invention may be provided as code that can be implemented, practiced, or executed by a "computer" (a comprehensive concept including a system on chip (SoC) or (micro)processor, etc.), or as a computer-readable storage medium or computer program product that stores or contains said code, and the scope of the present invention may be extended to said code or as a computer-readable storage medium or computer program product that stores or contains said code.

[0156]

[0157] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention as described in the following claims. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. In a robot position estimation device, A sensor group including an image sensor or a LiDAR sensor; and A processor configured to obtain a semantic point cloud for an area of ​​a preset size around the robot from sensor data obtained through the sensor group, and to estimate the position of the robot by comparing the obtained semantic point cloud around the robot with the semantic point cloud of a plurality of candidate areas having the preset size. The above plurality of candidate regions includes only candidate regions where the robot can move, and A device comprising semantic point groups of the plurality of candidate regions, each point or each point group, semantic class information, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information obtained from a previously created original map.

2. In paragraph 1, the processor, A device configured to extract semantic class of the robot's current position, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information from the semantic point cloud around the robot obtained above.

3. In paragraph 2, the processor is, A device configured to obtain a first-level set of candidate regions by selecting, from among the semantic point clouds of the plurality of candidate regions, a semantic point cloud of a candidate region having the same semantic class as the semantic class of the robot's current position.

4. In paragraph 3, the processor, A device configured to obtain a second level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the first level set of candidate regions, a semantic point cloud of a candidate region having rotation-invariant semantic class feature information similar to rotation-invariant semantic class feature information extracted from the semantic point cloud around the robot.

5. In paragraph 4, the processor, A device configured to obtain a third level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the second level set of candidate regions, a semantic point cloud of a candidate region having rotationally variable semantic class feature information similar to rotationally variable semantic class feature information extracted from the semantic point cloud around the robot.

6. In paragraph 4, the processor, A device configured to select a preset number (N1) of candidate regions included in the set of candidate regions of the second level.

7. In paragraph 5, the above processor, A device configured to select a preset number (N2) of candidate regions included in the set of candidate regions of the third level.

8. In paragraph 5, the above processor, A device configured to acquire primary estimated position and orientation information of the robot based on rotational variable semantic class feature information of a semantic point cloud of candidate regions belonging to the set of candidate regions of the third level.

9. In paragraph 8, the processor, A device configured to obtain secondary estimated position and orientation information of the robot by comparing each point of the semantic point group around the robot obtained above with each point of the semantic point group of the candidate region belonging to the set of candidate regions of the third level in a one-to-one manner.

10. In paragraph 5, the processor, A device configured to re-acquire the second level of candidate region set when the similarity between the extracted rotationally variable semantic feature information and the rotationally variable semantic class feature information of the semantic point cloud of the third level of candidate region set is below a threshold.

11. In Clause 10, the above processor, A device configured to lower the similarity threshold for the rotation-invariant semantic class feature information or increase the number of candidate regions belonging to the second level of candidate region set when reacquiring the second level of candidate region set.

12. In Paragraph 4, A device comprising the above rotation-invariant semantic class feature information including semantic class ratio information in the above preset size area.

13. In Paragraph 5, A device comprising the above-mentioned rotational variable semantic class feature information including semantic class information in a preset angle range and distance range in a preset size area.

14. In a method for estimating the position of a robot, A step of obtaining a semantic point cloud for an area of ​​a preset size around the robot from sensor data obtained through one or more sensors; and The method includes the step of estimating the position of the robot by comparing the semantic point cloud around the robot obtained above with the semantic point cloud of a plurality of candidate regions having a preset size, and The above plurality of candidate regions includes only candidate regions where the robot can move, and A method in which the semantic point groups of the plurality of candidate regions include semantic class information for each point or each point group, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information obtained from a previously created original map.

15. In Paragraph 14, A method comprising the step of extracting semantic class of the robot's current position, rotation-invariant semantic class feature information, or rotation-variable semantic class feature information from the semantic point cloud around the robot obtained above.

16. In Paragraph 15, A method comprising the step of obtaining a first-level set of candidate regions by selecting, from among the semantic point clouds of the plurality of candidate regions, a semantic point cloud of a candidate region having the same semantic class as the semantic class of the robot's current position.

17. In Paragraph 16, A method comprising the step of obtaining a second level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the first level set of candidate regions, a semantic point cloud of a candidate region having rotation-invariant semantic class feature information similar to rotation-invariant semantic class feature information extracted from the semantic point cloud around the robot.

18. In Paragraph 17, A method comprising the step of obtaining a third level set of candidate regions by selecting, from among the semantic point clouds of candidate regions belonging to the second level set of candidate regions, a point cloud of a candidate region having rotationally variable semantic class feature information similar to rotationally variable semantic class feature information extracted from the semantic point cloud around the robot.

19. In Paragraph 18, A method comprising the step of obtaining final estimated position and orientation information of the robot by comparing each point of the semantic point group around the robot obtained above with each point of the semantic point group of the candidate region belonging to the set of candidate regions of the third level in a one-to-one manner.

20. A computer-readable medium storing code configured to execute a method according to any one of paragraphs 14 through 19 by a computer or processor.