Visual relocalization method and apparatus, computer-readable storage medium, and robot
By semantic perception and angle adjustment of the robot's vision sensor, the problems of relocation loss and poor stability in visual relocation methods are solved, and stable visual relocation under a wider field of view is achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- UBTECH ROBOTICS CORP LTD
- Filing Date
- 2024-12-28
- Publication Date
- 2026-05-21
AI Technical Summary
Existing visual relocalization methods are prone to relocalization loss and have poor stability.
By performing semantic perception on images acquired by the robot's vision sensors, visual semantic perception information is determined, and the angle of the vision sensors is adjusted based on this information to perform visual relocalization in a wider field of view.
It effectively reduces the loss of relocation and improves the stability of visual relocation.
Smart Images
Figure CN2024143524_21052026_PF_FP_ABST
Abstract
Description
Visual relocalization methods, devices, computer-readable storage media, and robots
[0001] This application claims priority to Chinese Patent Application No. 202411635173.0, filed on November 15, 2024, entitled "Visual Relocation Method, Apparatus, Computer-Readable Storage Medium and Robot", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application belongs to the field of robotics technology, and in particular relates to a visual relocation method, apparatus, computer-readable storage medium, and robot. Background Technology
[0003] In the field of robotics, visual repositioning is a crucial fundamental problem. Visual repositioning refers to determining a robot's pose within an existing visual prior map based on real-time data captured by visual sensors. While numerous visual repositioning methods exist, they are often limited by the finite information available within a fixed field of view, making them prone to repositioning loss and exhibiting poor stability. Technical issues
[0004] In view of this, embodiments of this application provide a visual relocation method, apparatus, computer-readable storage medium, and robot to solve the problems of easy relocation loss and poor stability in existing visual relocation methods. Technical solutions
[0005] A first aspect of this application provides a visual relocation method, which may include:
[0006] In the event of visual relocalization failure, semantic perception is performed on the first image acquired by the robot's visual sensor to obtain visual semantic perception information.
[0007] The angle adjustment information of the visual sensor is determined based on the visual semantic perception information;
[0008] The visual sensor is adjusted according to the angle adjustment information;
[0009] Based on the second image acquired by the visual sensor after angle adjustment, the robot is visually relocalized to obtain the visual relocalization result.
[0010] In one specific implementation of the first aspect, determining the angle adjustment information of the visual sensor based on the visual semantic perception information may include:
[0011] Determine the scene differences between the visual semantic perception information and the preset visual prior map information;
[0012] If the scene difference is greater than a preset scene difference threshold, then the first angle limit and the second angle limit of the visual sensor are determined; wherein, the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions;
[0013] The angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.
[0014] In one specific implementation of the first aspect, after determining the scene difference between the visual semantic perception information and the preset visual prior map information, it may further include:
[0015] If the scene difference is less than or equal to the scene difference threshold, then the bias angle of the visual sensor is determined based on the visual semantic perception information.
[0016] The bias angle is determined as the angle adjustment information.
[0017] In one specific implementation of the first aspect, determining the bias angle of the visual sensor based on the visual semantic perception information may include:
[0018] Based on the visual semantic perception information, the regional scene differences in each area of the robot's field of vision are determined respectively;
[0019] A first target region is selected based on the differences in the regional scenes, wherein the first target region is the region with the smallest differences in the regional scenes;
[0020] The angle corresponding to the first target area is determined as the bias angle.
[0021] In one specific implementation of the first aspect, determining the bias angle of the visual sensor based on the visual semantic perception information may include:
[0022] Based on the visual semantic perception information, determine the static semantic objects in the robot's field of vision;
[0023] The angle corresponding to the static semantic object is determined as the bias angle.
[0024] In one specific implementation of the first aspect, determining the bias angle of the visual sensor based on the visual semantic perception information may include:
[0025] Based on the visual semantic perception information, the texture richness of each region in the robot's field of vision is determined respectively;
[0026] A second target region is selected based on the texture richness, wherein the second target region is the region with the highest texture richness;
[0027] The angle corresponding to the second target area is determined as the bias angle.
[0028] In one specific implementation of the first aspect, after performing visual relocalization on the robot and obtaining the visual relocalization result, the method may further include:
[0029] If the visual relocation result is visual relocation failure, then return to the step of performing semantic perception on the first image acquired by the robot's visual sensor and subsequent steps, until the visual relocation result is visual relocation success.
[0030] A second aspect of the embodiments of this application provides a visual relocation device, which may include:
[0031] The visual semantic perception module is used to perform semantic perception on the first image acquired by the robot's visual sensor in the event of visual relocalization failure, and obtain visual semantic perception information.
[0032] An angle adjustment information determination module is used to determine the angle adjustment information of the visual sensor based on the visual semantic perception information.
[0033] An angle adjustment module is used to adjust the angle of the vision sensor according to the angle adjustment information;
[0034] The visual relocalization module is used to perform visual relocalization on the robot based on the second image acquired by the visual sensor after angle adjustment, and obtain the visual relocalization result.
[0035] In one specific implementation of the second aspect, the angle adjustment information determination module may include:
[0036] The scene difference determination submodule is used to determine the scene difference between the visual semantic perception information and the preset visual prior map information;
[0037] An angle traversal adjustment information determination submodule is used to determine the first angle limit and the second angle limit of the visual sensor if the scene difference is greater than a preset scene difference threshold; wherein, the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions; the angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.
[0038] In one specific implementation of the second aspect, the angle adjustment information determination module may further include:
[0039] The bias angle adjustment information determination submodule is used to determine the bias angle of the visual sensor based on the visual semantic perception information if the scene difference is less than or equal to the scene difference threshold; and to determine the bias angle as the angle adjustment information.
[0040] In one specific implementation of the second aspect, the bias angle adjustment information determination submodule may include:
[0041] The first bias angle determination unit is used to determine the regional scene differences of each region in the robot's field of vision based on the visual semantic perception information; select a first target region based on the regional scene differences, wherein the first target region is the region with the smallest regional scene differences; and determine the angle corresponding to the first target region as the bias angle.
[0042] In one specific implementation of the second aspect, the bias angle adjustment information determination submodule may include:
[0043] The second bias angle determination unit is used to determine the static semantic object in the robot's field of vision based on the visual semantic perception information; and to determine the angle corresponding to the static semantic object as the bias angle.
[0044] In one specific implementation of the second aspect, the bias angle adjustment information determination submodule may include:
[0045] The third bias angle determination unit is used to determine the texture richness of each region in the robot's field of vision based on the visual semantic perception information; select a second target region based on the texture richness, wherein the second target region is the region with the largest texture richness; and determine the angle corresponding to the second target region as the bias angle.
[0046] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described visual relocation methods.
[0047] A fourth aspect of this application provides a robot including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described visual relocation methods.
[0048] The fifth aspect of this application provides a computer program product that, when run on a robot, causes the robot to perform the steps of any of the above-described visual relocation methods. Beneficial effects
[0049] The beneficial effects of this application embodiment compared with the prior art are as follows: In the case of visual relocalization failure, this application embodiment performs semantic perception on the first image acquired by the robot's visual sensor to obtain visual semantic perception information; determines the angle adjustment information of the visual sensor based on the visual semantic perception information; adjusts the angle of the visual sensor based on the angle adjustment information; and performs visual relocalization on the robot based on the second image acquired by the angle-adjusted visual sensor to obtain the visual relocalization result. Through this application embodiment, the angle adjustment information of the visual sensor can be determined based on visual semantic perception information, and richer information from a wider field of view can be obtained through angle adjustment. Visual relocalization performed on this basis can effectively reduce relocalization loss and improve the stability of visual relocalization. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 is a flowchart of an embodiment of a visual relocation method according to this application;
[0052] Figure 2 is a schematic flowchart of determining the angle adjustment information of the visual sensor based on visual semantic perception information;
[0053] Figure 3 is a schematic diagram of the rotational degrees of freedom of the robot's head;
[0054] Figure 4 is a schematic diagram of angle traversal;
[0055] Figure 5 is a structural diagram of an embodiment of a visual relocation device according to the present application;
[0056] Figure 6 is a schematic block diagram of a robot according to an embodiment of this application. Embodiments of the present invention
[0057] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0058] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0059] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0060] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0061] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."
[0062] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0063] In the field of robotics, visual repositioning is a crucial fundamental problem. Visual repositioning refers to determining a robot's pose within an existing visual prior map based on real-time data captured by visual sensors. While numerous visual repositioning methods exist, they are often limited by the finite information available within a fixed field of view, making them prone to repositioning loss and exhibiting poor stability.
[0064] In view of this, embodiments of this application provide a visual relocation method, apparatus, computer-readable storage medium, and robot to solve the problems of easy relocation loss and poor stability in existing visual relocation methods.
[0065] In this embodiment, the angle adjustment information of the visual sensor can be determined based on visual semantic perception information, and richer information under a wider field of view can be obtained through angle adjustment. Visual relocalization can be performed on this basis, which can effectively reduce the phenomenon of relocalization loss and improve the stability of visual relocalization.
[0066] The executing entity in the embodiments of this application can be a robot, including but not limited to industrial robots, home service robots, commercial service robots, and other types of robots.
[0067] Please refer to Figure 1. One embodiment of a visual relocation method in this application may include:
[0068] Step S101: In the event of visual relocalization failure, perform semantic perception on the first image acquired by the robot's visual sensor to obtain visual semantic perception information.
[0069] Visual relocalization is typically implemented by the robot's Visual Simultaneous Localization and Mapping (VSLAM) module, which can include states such as initial relocalization, continuous frame localization, and relocalization. Each state can include both success and failure.
[0070] When the robot begins to perform a task, it first performs initial relocalization, matching the images captured by the robot's visual sensors with a preset visual prior map to obtain the robot's pose in the map coordinate system, thus realizing the relocalization function.
[0071] During the robot's task execution, continuous frame localization can be performed. At this time, the robot's vision sensors will continue to collect images and match these images with the visual prior map in real time to obtain real-time relocalization results.
[0072] In the event of continuous frame localization loss, the robot can obtain short-term pose using devices such as inertial measurement units (IMUs) and kinematic odometry. However, cumulative errors exist over time. Therefore, the robot can perform relocalization, and the relocalization process is the same as the initial relocalization.
[0073] If the robot successfully performs visual relocalization through the above process, the angle of the visual sensor does not need to be adjusted, and subsequent visual relocalization can proceed according to the same process. However, if the robot's visual relocalization fails, the angle of the visual sensor can be adjusted to obtain richer information from a wider field of view, and visual relocalization can be performed based on this information.
[0074] To enable more targeted angle adjustments to the vision sensor, embodiments of this application can perform semantic perception on images (denoted as the first image) acquired by the robot's vision sensor when visual relocalization fails, thereby obtaining corresponding visual semantic perception information. This visual semantic perception information may include, but is not limited to, semantic objects, text information, and texture information.
[0075] Taking semantic objects as an example, a preset semantic segmentation model can be used to perform semantic segmentation on the first image, thereby obtaining various semantic objects in the first image, including but not limited to pedestrians, vehicles, tables, cabinets, and other types of semantic objects. The specific semantic segmentation model used can be flexibly set according to the actual situation, and may include but not limited to YOLO, U-Net, FCN, SegNet, PSPNet, DeepLab, and other models. This application embodiment does not specifically limit this.
[0076] Taking text information as an example, the first image can be recognized by a preset optical character recognition (OCR) algorithm to obtain the text information in the first image, including but not limited to various types of text information such as Chinese and English. The specific optical character recognition algorithm used can be flexibly set according to the actual situation, and may include but not limited to rule-based recognition algorithms, template matching-based recognition algorithms, deep learning-based recognition algorithms, and other recognition algorithms. This application does not specifically limit this.
[0077] Taking texture information as an example, feature points can be detected in the first image based on a preset feature point detection algorithm to obtain feature points with significant characteristics in the first image, such as corner points, edge points, or other significant texture points. The specific feature point detection algorithm used can be flexibly set according to the actual situation, and may include, but is not limited to, Harris, SIFT, SURF, ORB, FAST, BRISK, and other detection algorithms. This application embodiment does not specifically limit this.
[0078] Step S102: Determine the angle adjustment information of the visual sensor based on the visual semantic perception information.
[0079] As shown in Figure 2, step S102 may include the following process:
[0080] Step S1021: Determine the scene difference between the visual semantic perception information and the preset visual prior map information.
[0081] In this embodiment of the application, feature point matching can be performed between feature points in visual semantic perception information and feature points in visual prior map information, and the ratio between the number of feature points that failed to match in visual semantic perception information and the total number of feature points is determined as the scene difference.
[0082] Step S1022: Determine whether the scene difference is greater than the preset scene difference threshold.
[0083] The specific value of the scene difference threshold can be flexibly set according to the actual situation, and this application embodiment does not impose a specific limitation on it. If the scene difference is greater than the preset scene difference threshold, it means that the scene difference is too large and the current visual semantic perception information is not very meaningful. At this time, an unbiased angle traversal adjustment can be performed, i.e., step S1023 is executed. If the scene difference is less than or equal to the scene difference threshold, it means that the scene difference is not large and the current visual semantic perception information is still meaningful. At this time, a biased angle adjustment can be performed based on the visual semantic perception information, i.e., step S1024 is executed.
[0084] Step S1023: Determine the first angle limit and the second angle limit of the vision sensor, and determine the angle traversal between the first angle limit and the second angle limit as angle adjustment information.
[0085] Wherein, the first angle limit is the angle limit in the first direction, the second angle limit is the angle limit in the second direction, and the first direction and the second direction are opposite directions.
[0086] With the vision sensor located at the robot's head, the vision sensor will move with the head, as shown in Figure 3. The robot's head can have three rotational degrees of freedom: pitch around the Y-axis, yaw around the Z-axis, and roll around the X-axis. The direction in front of the head is the positive X-axis, the direction to the right of the head is the positive Y-axis, and the direction above the head is the positive Z-axis. Thus, the vision sensor can rotate along with the robot's head in these three rotational degrees of freedom.
[0087] In this embodiment, the robot's field of view can be changed by adjusting the yaw angle to perform visual relocalization under the new field of view. The first direction can be a leftward rotation direction, and the second direction can be a rightward rotation direction. The first angle limit is the leftward rotation angle limit, with a negative angle value, and the second angle limit is the rightward rotation angle limit, with a positive angle value. The specific values of the first and second angle limits can be flexibly set according to actual conditions, and this embodiment does not impose specific limitations on them. As an example, the first angle limit can be set to -90 degrees, and the second angle limit can be set to 90 degrees. In this case, as shown in Figure 4, the robot's head can be gradually rotated from the leftmost to the rightmost, that is, the angle traversal from -90 degrees to 90 degrees is determined as the angle adjustment information. Of course, the opposite setting can also be performed, gradually rotating the robot's head from the rightmost to the leftmost, that is, the angle traversal from 90 degrees to -90 degrees is determined as the angle adjustment information.
[0088] Step S1024: Determine the bias angle of the visual sensor based on the visual semantic perception information, and set the bias angle as the angle adjustment information.
[0089] In the embodiments of this application, the bias angle of the visual sensor can be determined based on at least one of scene differences, static semantic objects, and texture richness.
[0090] Taking scene differences as an example, the scene differences of each region in the robot's field of vision can be determined based on visual semantic perception information. The specific division method of each region can be flexibly set according to the actual situation, and this application embodiment does not impose specific limitations on this. For example, the robot's field of vision can be divided equally from left to right to divide each region. For any region, the ratio between the number of feature points that failed to match in that region and the total number of feature points in that region can be determined as the region scene difference. After determining the scene differences of each region, a first target region can be selected based on the scene differences, and the angle corresponding to the first target region can be determined as the bias angle. The first target region is the region with the smallest scene difference. If the robot adjusts the angle of the vision sensor according to this bias angle, the vision sensor can be oriented towards the region with the smallest scene difference, which can effectively improve the probability of successful relocalization.
[0091] Taking static semantic objects as an example, static semantic objects in the robot's field of vision can be determined based on visual semantic perception information, and the angle corresponding to the static semantic object can be defined as a bias angle. Static semantic objects can include, but are not limited to, fixed-position objects such as text, tables, and cabinets. If the robot adjusts the angle of its visual sensor according to this bias angle, the visual sensor can be directed towards the fixed-position static semantic object, effectively increasing the probability of successful relocalization.
[0092] Taking texture richness as an example, the texture richness of each region in the robot's field of vision can be determined based on visual semantic perception information. For any given region, the total number of feature points in that region can be defined as its texture richness. After determining the texture richness of each region, a second target region can be selected based on the texture richness, and the angle corresponding to the second target region can be defined as the bias angle. The second target region is the region with the highest texture richness. If the robot adjusts the angle of the vision sensor according to this bias angle, the vision sensor can be oriented towards the region with the highest texture richness, effectively increasing the probability of successful relocalization.
[0093] Scene differences, static semantic objects, and texture richness can have different priorities. For example, scene differences can be set to the highest priority, static semantic objects to the second highest priority, and texture richness to the lowest priority. That is, the bias angle is determined first based on scene differences, then based on static semantic objects, and finally based on texture richness. Of course, other different priority settings can be made according to the actual situation, and this application embodiment does not specifically limit this.
[0094] Step S103: Adjust the angle of the vision sensor according to the angle adjustment information.
[0095] With the vision sensor located at the robot's head, if the angle adjustment information is an angle traversal, the robot's head can be controlled to rotate gradually from the leftmost to the rightmost, or from the rightmost to the leftmost, thereby causing the vision sensor to rotate gradually as well. If the angle adjustment information is a directional angle, the robot's head can be controlled to rotate to that directional angle, thereby causing the vision sensor to rotate to that directional angle as well.
[0096] Step S104: Based on the second image acquired by the visual sensor after angle adjustment, perform visual relocalization on the robot to obtain the visual relocalization result.
[0097] Because the angle of the robot's vision sensor has been adjusted, the robot's field of view has changed accordingly. In this embodiment, the robot can be visually repositioned based on the image captured by the vision sensor in the new field of view (denoted as the second image), thereby obtaining the corresponding visual repositioning result. If the visual repositioning result is a failure, the angle can be readjusted again according to the process shown in Figure 1, and visual repositioning can be performed again in the new field of view until the visual repositioning result is successful.
[0098] In summary, in the embodiments of this application, the angle adjustment information of the visual sensor can be determined based on visual semantic perception information, and richer information under a wider field of view can be obtained through angle adjustment. On this basis, visual relocalization can be performed, which can effectively reduce the phenomenon of relocalization loss and improve the stability of visual relocalization.
[0099] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0100] Corresponding to the visual relocation method described in the above embodiments, Figure 5 shows a structural diagram of an embodiment of a visual relocation device provided in this application.
[0101] In this embodiment, a visual relocalization device may include:
[0102] The visual semantic perception module 501 is used to perform semantic perception on the first image acquired by the robot's visual sensor in the event of visual relocalization failure, and obtain visual semantic perception information.
[0103] Angle adjustment information determination module 502 is used to determine the angle adjustment information of the visual sensor based on the visual semantic perception information;
[0104] Angle adjustment module 503 is used to adjust the angle of the vision sensor according to the angle adjustment information;
[0105] The visual repositioning module 504 is used to perform visual repositioning on the robot based on the second image acquired by the visual sensor after angle adjustment, and obtain the visual repositioning result.
[0106] In one specific implementation of this application embodiment, the angle adjustment information determination module may include:
[0107] The scene difference determination submodule is used to determine the scene difference between the visual semantic perception information and the preset visual prior map information;
[0108] An angle traversal adjustment information determination submodule is used to determine the first angle limit and the second angle limit of the visual sensor if the scene difference is greater than a preset scene difference threshold; wherein, the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions; the angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.
[0109] In one specific implementation of this application embodiment, the angle adjustment information determination module may further include:
[0110] The bias angle adjustment information determination submodule is used to determine the bias angle of the visual sensor based on the visual semantic perception information if the scene difference is less than or equal to the scene difference threshold; and to determine the bias angle as the angle adjustment information.
[0111] In one specific implementation of this application embodiment, the bias angle adjustment information determination submodule may include:
[0112] The first bias angle determination unit is used to determine the regional scene differences of each region in the robot's field of vision based on the visual semantic perception information; select a first target region based on the regional scene differences, wherein the first target region is the region with the smallest regional scene differences; and determine the angle corresponding to the first target region as the bias angle.
[0113] In one specific implementation of this application embodiment, the bias angle adjustment information determination submodule may include:
[0114] The second bias angle determination unit is used to determine the static semantic object in the robot's field of vision based on the visual semantic perception information; and to determine the angle corresponding to the static semantic object as the bias angle.
[0115] In one specific implementation of this application embodiment, the bias angle adjustment information determination submodule may include:
[0116] The third bias angle determination unit is used to determine the texture richness of each region in the robot's field of vision based on the visual semantic perception information; select a second target region based on the texture richness, wherein the second target region is the region with the largest texture richness; and determine the angle corresponding to the second target region as the bias angle.
[0117] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0118] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0119] Figure 6 shows a schematic block diagram of a robot provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.
[0120] As shown in Figure 6, the robot 6 in this embodiment includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the various visual relocation method embodiments described above, such as steps S101 to S104 shown in Figure 1. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the various device embodiments described above, such as the functions of modules 501 to 504 shown in Figure 5.
[0121] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the robot 6.
[0122] Those skilled in the art will understand that Figure 6 is merely an example of robot 6 and does not constitute a limitation on robot 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, robot 6 may also include input / output devices, network access devices, buses, etc.
[0123] The processor 60 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0124] The memory 61 can be an internal storage unit of the robot 6, such as a hard drive or memory. The memory 61 can also be an external storage device of the robot 6, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 61 can include both internal and external storage units of the robot 6. The memory 61 is used to store the computer program and other programs and data required by the robot 6. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0128] In the embodiments provided in this application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0131] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.
[0132] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A visual relocalization method, characterized by, include: In the event of visual relocalization failure, semantic perception is performed on the first image acquired by the robot's visual sensor to obtain visual semantic perception information. The angle adjustment information of the visual sensor is determined based on the visual semantic perception information; The visual sensor is adjusted according to the angle adjustment information; Based on the second image acquired by the visual sensor after angle adjustment, the robot is visually relocalized to obtain the visual relocalization result.
2. The visual relocalization method of claim 1, wherein, Determining the angle adjustment information of the visual sensor based on the visual semantic perception information includes: Determine the scene differences between the visual semantic perception information and the preset visual prior map information; If the scene difference is greater than a preset scene difference threshold, then the first angle limit and the second angle limit of the visual sensor are determined; wherein, the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions; The angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.
3. The visual relocalization method of claim 2, wherein, After determining the scene difference between the visual semantic perception information and the preset visual prior map information, the method further includes: If the scene difference is less than or equal to the scene difference threshold, then the bias angle of the visual sensor is determined based on the visual semantic perception information. The bias angle is determined as the angle adjustment information.
4. The visual relocalization method of claim 3, wherein, Determining the bias angle of the visual sensor based on the visual semantic perception information includes: Based on the visual semantic perception information, the regional scene differences in each area of the robot's field of vision are determined respectively; A first target region is selected based on the differences in the regional scenes, wherein the first target region is the region with the smallest differences in the regional scenes; The angle corresponding to the first target area is determined as the bias angle.
5. The visual relocalization method of claim 3, wherein, Determining the bias angle of the visual sensor based on the visual semantic perception information includes: Based on the visual semantic perception information, determine the static semantic objects in the robot's field of vision; The angle corresponding to the static semantic object is determined as the bias angle.
6. The visual relocalization method of claim 3, wherein, Determining the bias angle of the visual sensor based on the visual semantic perception information includes: Based on the visual semantic perception information, the texture richness of each region in the robot's field of vision is determined respectively; A second target region is selected based on the texture richness, wherein the second target region is the region with the highest texture richness; The angle corresponding to the second target area is determined as the bias angle.
7. The visual relocalization method according to any of claims 1 to 6, wherein, After performing visual relocalization on the robot and obtaining the visual relocalization result, the process further includes: If the visual relocation result is visual relocation failure, then return to the step of performing semantic perception on the first image acquired by the robot's visual sensor and subsequent steps, until the visual relocation result is visual relocation success.
8. A visual repositioning device, characterized in that include: The visual semantic perception module is used to perform semantic perception on the first image acquired by the robot's visual sensor in the event of visual relocalization failure, and obtain visual semantic perception information. An angle adjustment information determination module is used to determine the angle adjustment information of the visual sensor based on the visual semantic perception information. An angle adjustment module is used to adjust the angle of the vision sensor according to the angle adjustment information; The visual relocalization module is used to perform visual relocalization on the robot based on the second image acquired by the visual sensor after angle adjustment, and obtain the visual relocalization result.
9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. When the computer program is executed by a processor, it implements the steps of the visual relocation method as described in any one of claims 1 to 7.
10. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the visual relocation method as described in any one of claims 1 to 7.