Image processing method and apparatus, and agent

Abstract

Provided in the present application are an image processing method and apparatus, and an agent. The method relates to the field of image processing, and comprises: on the basis of a vision sensor, acquiring original images of a physical environment where the vision sensor is located; and on the basis of the original images, obtaining reflectance maps, wherein the reflectance maps are used for determining pose and motion parameters of the vision sensor. By means of processing the reflectance maps that can reflect intrinsic properties of an environmental object, the impact of non-ideal lighting conditions can be reduced, such that the calculated pose and motion parameters of the vision sensor are more accurate.

Description

An image processing method, apparatus and intelligent agent Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, and intelligent agent. Background Technology

[0002] With the development of artificial intelligence-related technologies, the demands on the perception capabilities of intelligent agents are increasing. Perception systems, represented by visual sensors (such as pinhole cameras and fisheye cameras), are one of the most fundamental and extensively researched effective environmental information inputs in the fields of intelligent driving and even robotics. Various processing and analysis methods are applied to the images acquired by visual sensors to achieve complex visual tasks. Taking the field of intelligent driving as an example, a multi-view camera system deployed around the vehicle body can provide the vehicle with all-around environmental perception capabilities.

[0003] Based on some known image processing methods, the accuracy of an intelligent agent's perception of the environment is not high. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and intelligent agent to improve the accuracy of the intelligent agent's perception of the environment.

[0005] In a first aspect, this application provides an image processing method that can be applied to an intelligent agent. The steps of the method can be executed by the intelligent agent, or the method can be executed by a component (such as a chip, chip system, etc.) configured in the intelligent agent, or it can be implemented by a logic module or software capable of realizing all or part of the functions of the intelligent agent. This application does not limit the scope of the method.

[0006] In this application, the intelligent agent may include vehicles, robots, or drones, etc.

[0007] For example, the method includes: acquiring an original image of the physical environment in which the visual sensor is located based on the visual sensor; and obtaining a reflection map based on the original image, the reflection map being used to determine the pose motion parameters of the visual sensor.

[0008] Based on the above technical solution, by obtaining the reflection map from the raw image acquired by the visual sensor, and directly processing the reflection map that reflects the intrinsic properties of objects in the environment (e.g., feature-based methods, optical flow tracing methods, or direct methods), the influence of non-ideal lighting conditions can be reduced, making the calculated pose and motion parameters of the visual sensor more accurate. This reduces the dependence on the intensity of light in the environment, thereby improving the perception accuracy of the intelligent agent.

[0009] In conjunction with the first aspect, in some possible implementations, acquiring raw images of the physical environment in which the visual sensor is located based on the visual sensor includes: acquiring M raw images based on the visual sensor, where M is an integer greater than or equal to 2; and obtaining reflection maps based on the raw images includes: obtaining N reflection maps based on the M raw images, where N is an integer greater than or equal to 2 and less than or equal to M.

[0010] In conjunction with the first aspect, in some possible implementations, obtaining N reflection images based on the M original images includes: estimating the brightness of a first image among the M original images to obtain a brightness map of the first image, wherein the first image is a keyframe image among the M original images, and the M original images include the N keyframe images; and separating the brightness map from the first image to obtain a reflection map of the first image.

[0011] In conjunction with the first aspect, in some possible implementations, obtaining N reflection images based on the M original images includes: estimating the brightness of a second image among the M original images to obtain a brightness map of the second image, wherein the second image is any one of the M original images; separating the brightness map from the second image to obtain a reflection map of the second image; and determining N reflection maps from the M reflection maps.

[0012] In conjunction with the first aspect, in some possible implementations, the method further includes: extracting features from a first reflection map among the N reflection maps to obtain a first set of feature points, wherein the first reflection map is any one of the N reflection maps.

[0013] Feature extraction from reflectance maps can effectively increase the number of feature points.

[0014] In conjunction with the first aspect, in some possible implementations, the method further includes: obtaining a first weight set of the first brightness map based on the first brightness map, the first weight set including the weight of the pixel brightness corresponding to each pixel point in the first brightness map, the first brightness map being any one of the N brightness maps used to obtain the N reflection maps.

[0015] The feature point method discards some image information. By using the first weight set obtained based on the brightness map to optimize and filter feature points, it can effectively utilize image brightness information and improve the accuracy of feature point extraction.

[0016] In conjunction with the first aspect, in some possible implementations, in the first set of weights, the weight corresponding to the first pixel is 1, and the weight corresponding to the second pixel is 0. The first pixel is a pixel whose pixel brightness is greater than or equal to the first threshold and less than or equal to the second threshold, and the second pixel is a pixel whose pixel brightness is less than the first threshold or greater than the second threshold.

[0017] In conjunction with the first aspect, in some possible implementations, the method further includes: optimizing and filtering the feature points in the first feature point set based on the first weight set to obtain a second feature point set, wherein the first feature point set is the feature point set obtained after feature extraction based on the first reflection map, and the first reflection map is any one of the N reflection maps.

[0018] In conjunction with the first aspect, in some possible implementations, the method further includes: matching feature points in the second feature set of the nth reflection map among the N reflection maps with feature points in the second feature point set of the (n+1)th reflection map to obtain a first feature point pair set, where n is an integer greater than or equal to 1 and less than or equal to N-1; obtaining a second weight set based on the difference in pixel brightness corresponding to each pair of feature points in the first feature point pair set, the second weight set including the weight corresponding to each pair of feature points in the first feature point pair set; and optimizing and filtering the feature points in the first feature point pair set based on the second weight set to obtain a second feature point pair set.

[0019] In conjunction with the first aspect, in some possible implementations, in the second weight set, the weight corresponding to the first feature point pair is 1, and the weight corresponding to the second feature point pair is 0. The first feature point pair is a feature point pair in the first feature point pair set where the difference in pixel brightness between the two feature points is less than or equal to a third threshold, and the second feature point pair is a feature point pair in the first feature point pair set where the difference in pixel brightness between the two feature points is greater than the third threshold.

[0020] In conjunction with the first aspect, in some possible implementations, the method further includes: estimating the pose motion parameters of the visual sensor based on the second set of feature point pairs: initial values ​​R0 of the rotation matrix and t0 of the translation vector; and constructing a reprojection error based on the first set of weights and the second set of feature point pairs, the reprojection error satisfying:

[0021] Where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p jLet j represent the j-th feature point of the first reflection map in the set of second feature point pairs corresponding to the first reflection map, where 1≤j≤k, and k indicates that the set of second feature point pairs corresponding to the first reflection map includes k feature point pairs; This represents the pose motion parameters R of the visual sensor at the target time. j and t j Regarding p j The reprojected point, where the target time is the time when the original image of the first reflection map was acquired; W k This represents the weight of the k-th feature point pair in the set of second feature point pairs corresponding to the first reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0022] In conjunction with the first aspect, in some possible implementations, the method further includes: estimating the pose motion parameters of the visual sensor based on the second set of feature point pairs: initial values ​​R0 of the rotation matrix and t0 of the translation vector; and constructing a reprojection error based on the first set of weights and the second set of feature point pairs, the reprojection error satisfying:

[0023] Where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p i Let i represent the feature points in the feature point pair in the set of the second feature point pairs corresponding to the i-th reflection map, where 1≤i≤n; This represents the pose motion parameters R of the visual sensor at the target time. i and t i Regarding p i The reprojected point, where the target time is the time when the original image of the i-th reflection map was acquired; W i This represents the first weight set corresponding to the i-th reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0024] Constructing reprojection error based on the first set of weights can effectively utilize brightness information and increase brightness-based constraints, thereby improving robustness.

[0025] In conjunction with the first aspect, in some possible implementations, the method further includes: processing the reflection image based on optical flow tracing to obtain the pose motion parameters R and t of the visual sensor.

[0026] In conjunction with the first aspect, in some possible implementations, the method further includes: processing the reflection image based on a direct method to obtain the pose motion parameters R and t of the visual sensor.

[0027] In conjunction with the first aspect, in some possible implementations, this method is applied to the calibration of the external parameters of the vision sensor.

[0028] Online calibration of vision sensors based on the image processing scheme provided in this application can overcome scene limitations under non-ideal lighting conditions (such as nighttime), enabling rapid and high-precision online calibration of vision sensors. This avoids safety risks caused by large angular deviations in the extrinsic parameters of vision sensors. In the field of intelligent driving, it can also improve the safety of vehicles performing intelligent driving at night. Furthermore, vehicles off the production line already have the function of online calibration of vision sensors at night, eliminating the need to enter a calibration room for calibration, and there are no requirements regarding open road lighting conditions.

[0029] In conjunction with the first aspect, in some possible implementations, this method is applied to visual odometry.

[0030] Secondly, this application provides an intelligent agent comprising execution steps for performing the first aspect and any possible implementation thereof. The intelligent agent includes corresponding modules for performing the above-described methods. The modules included in the intelligent agent can be implemented in software and / or hardware.

[0031] Thirdly, this application provides an intelligent agent including a processor. The processor is coupled to a memory and can be used to execute a program in the memory to implement the execution steps in the first aspect and any possible implementation thereof.

[0032] Optionally, the agent may also include a memory.

[0033] Optionally, the agent also includes a communication interface, with the processor coupled to the communication interface.

[0034] Optionally, in this application, the intelligent agent may include vehicles, robots, or drones, etc.

[0035] Fourthly, this application provides a chip system including at least one processor for supporting the implementation of the functions involved in the first aspect and any possible implementation of the first aspect, such as receiving or processing data and / or instruction information involved in the above methods.

[0036] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0037] The chip system can consist of chips or include chips and other discrete components.

[0038] Fifthly, this application provides a computer-readable storage medium storing a program (also referred to as code or instructions) that, when executed by a processor, causes the methods in the first aspect and any possible implementation thereof to be performed.

[0039] Sixthly, this application provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes the methods in the first aspect and any possible implementation thereof to be executed.

[0040] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0041] Figure 1 is a schematic structural block diagram of an image processing device provided in an embodiment of this application;

[0042] Figure 2 is a schematic flowchart of the image processing method provided in an embodiment of this application;

[0043] Figure 3 is a schematic comparison of the original image, brightness map, and reflection map provided in the embodiments of this application;

[0044] Figure 4 is a comparative diagram of the pixel brightness distribution histogram under non-ideal lighting conditions, the pixel brightness distribution histogram under ideal lighting conditions, and the brightness weight response function provided in the embodiments of this application.

[0045] Figure 5 is a comparative schematic diagram of feature extraction based on the original image and the reflectance image respectively, provided in an embodiment of this application;

[0046] Figure 6 is a comparative schematic diagram of feature matching based on the original image and the reflectance image respectively, provided in an embodiment of this application;

[0047] Figure 7 is a schematic hand-eye calibration diagram for external parameter calibration of a vision sensor provided in an embodiment of this application;

[0048] Figure 8 is a schematic diagram of triangulation constraints;

[0049] Figure 9 is another schematic flowchart of the image processing method provided in the embodiments of this application;

[0050] Figure 10 is a schematic block diagram of an image processing apparatus provided in an embodiment of this application;

[0051] Figure 11 is another schematic block diagram of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0053] First, in this application, the terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, an apparatus, system, product or device that includes a series of modules, units or units is not necessarily limited to those modules, units or units that are explicitly listed, but may include other modules, units or units that are not explicitly listed or that are inherent to such apparatus, system, product or device.

[0054] Second, in this application, the words "exemplarily" and "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design that is described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] Third, in this application, "when," "under the circumstances," "if," and "if" all refer to the device making a corresponding action under certain objective circumstances, and are not time-limited, nor do they require the device to make a judgment action when it is implemented, nor do they mean that there are other limitations.

[0056] Fourth, in this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first feature point set and the second feature point set are used to distinguish different feature point sets, and the first feature point pair set and the second feature point pair set are used to distinguish different feature point pair sets, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0057] Fifth, in this application, "preset" can be understood as predefined, defined, pre-defined, stored, pre-stored, pre-negotiated, or pre-configured, etc.

[0058] Sixth, in this application, "at least one (kind, type)" refers to one (kind, type) or multiple (kinds, types). "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects, but it does not exclude the possibility of indicating an "and" relationship. The specific meaning can be understood in conjunction with the context.

[0059] Seventh, in this application, information C is used to determine information D, including both determining information D based solely on information C and determining it based on information C and other information. Furthermore, information C can also be used to determine information D indirectly, for example, in the case where information D is determined based on information E, and information E is determined based on information C.

[0060] To facilitate understanding of the embodiments of this application, some technical terms or vocabulary involved in this application will be briefly explained below.

[0061] 1. Visual sensors: These are devices or systems capable of capturing, processing, and analyzing visual information. They are widely used in various fields, including robotics, autonomous driving, industrial automation, security monitoring, medical imaging, augmented reality (AR), and virtual reality (VR). The core function of visual sensors is to convert light signals into electrical signals and process them through algorithms and software to achieve perception and understanding of the environment or objects.

[0062] The visual sensors involved in this application may include, but are not limited to, pinhole cameras, fisheye cameras, red (R) green (G) blue (B) (RGB)-depth (D) (RGB-D) cameras, etc.

[0063] 2. Pose motion parameters of the vision sensor: These describe the changes in the position and orientation (i.e., direction) of the vision sensor in three-dimensional space.

[0064] 3. Visual odometry (VO): This is a technique that uses images captured by a camera to estimate the trajectory of a camera or robot in space. It has wide applications in computer vision, robot navigation, augmented reality, and autonomous driving.

[0065] The basic principle of visual odometry is to detect and match feature points in a series of continuously captured images, calculate the motion of these feature points between different image frames, and thus infer the motion trajectory of the camera or robot.

[0066] For example, the algorithm steps can typically be as follows:

[0067] 1) Image acquisition: Acquiring continuous image frames through a camera;

[0068] 2) Feature detection: Detect significant feature points, such as corners and edges, in each frame (or keyframe) of the image;

[0069] 3) Feature matching: Matching identical feature points between consecutive image frames;

[0070] 4) Motion estimation: Based on the matching results of feature points, estimate the camera's motion parameters (translation and rotation) using geometric methods (such as triangulation, PnP problem, etc.);

[0071] 5) Trajectory optimization: The estimated trajectory is globally optimized using optimization algorithms (such as bundle adjustment (BA) algorithms) to improve accuracy.

[0072] 4. BA Algorithm: This is an algorithm used in computer vision and photogrammetry to optimize 3D reconstruction and camera pose estimation. Its core idea is to simultaneously optimize the camera's intrinsic and extrinsic parameters and the coordinates of 3D points by minimizing the reprojection error, thereby improving the accuracy of the reconstruction results.

[0073] The goal of bundle adjustment is to find a set of camera parameters and 3D point coordinates such that the positions of these 3D points projected onto the image plane are as close as possible to the actual observed feature point positions.

[0074] For example, the algorithm steps can be as follows:

[0075] 1) Initial estimation: Obtain initial camera parameters and 3D point coordinates through other methods (such as visual odometry, triangulation, etc.);

[0076] 2) Error definition: Define reprojection error as the difference between the position of a 3D point projected onto the image plane by the camera and the actual observed position of the feature point;

[0077] 3) Optimization objective: Construct an objective function, which is typically the sum of squares of all reprojection errors;

[0078] 4) Iterative optimization: Use nonlinear optimization algorithms (such as the Levenberg-Marquardt (LM) algorithm) to iteratively adjust camera parameters and 3D point coordinates to minimize the objective function.

[0079] 5. The LM algorithm is an iterative optimization algorithm for solving nonlinear least squares problems. It combines the advantages of gradient descent and Gauss-Newton methods, providing better convergence and stability when dealing with nonlinear problems.

[0080] The LM algorithm achieves optimization by balancing gradient descent and the Gauss-Newton method. For example, the algorithm steps can be as follows:

[0081] 1) Initialization: Select the initial parameter vector, damping factor, and convergence threshold;

[0082] 2) Iterative update: Calculate the error function under the current parameter vector; calculate the Jacobian matrix; construct an approximate Hessian matrix and gradient vector; update the parameter vector; calculate the new error function; adjust the damping factor according to the change of the error function.

[0083] 3) Convergence judgment: If the change in the parameter vector is very small, the algorithm is considered to have converged and the iteration stops.

[0084] 6. Key frame: In the field of image processing, a key frame usually refers to a representative or important frame selected from a video or image sequence.

[0085] 7. Optical flow tracking: This is a method used in image processing and computer vision to track the motion of objects in an image sequence. Optical flow tracking estimates the temporal motion vector field of each pixel by analyzing changes in pixel intensity within the image sequence.

[0086] 8. Direct method: Also known as direct matching method, it is a fundamental method in image processing, typically used for basic image operations and processing. The direct method usually involves direct manipulation of image pixels without requiring complex mathematical transformations or frequency domain processing.

[0087] 9. Morphological operations: These mainly include dilation, erosion, opening, and closing operations. A brief explanation of these operations follows.

[0088] 1) Dilation: Dilation expands bright areas (usually the foreground) in an image, filling small black holes.

[0089] 2) Erosion: The erosion operation shrinks bright areas in an image and removes small white noise.

[0090] 3) Opening: The opening operation involves first performing erosion and then dilation. This operation can remove small white noise while preserving the overall shape of the image.

[0091] 4) Closing: Closing involves performing dilation followed by erosion. This operation can fill small black holes while preserving the overall shape of the image.

[0092] 10. Histogram equalization: This is an image processing technique used to enhance image contrast. By adjusting the image's gray-level histogram, it makes the distribution of gray-level values ​​more uniform, thereby improving the image's visual effect. Histogram equalization has wide applications in image enhancement, image analysis, and computer vision.

[0093] The basic idea of ​​histogram equalization is to redistribute the gray values ​​of an image so that each gray value has approximately the same probability of occurrence, thereby enhancing the image's contrast. For example, the steps of histogram equalization can be as follows:

[0094] 1) Calculate the grayscale histogram of the original image: count the number of pixels for each grayscale value (0 to 255) in the image to obtain the grayscale histogram.

[0095] 2) Calculate the cumulative distribution function (CDF): The cumulative distribution function represents the proportion of pixels with gray values ​​less than or equal to a certain value. The cumulative distribution function can be obtained by summing the values ​​of the gray-level histogram.

[0096] 3) Construct the mapping function: Based on the cumulative distribution function, map the original grayscale value to a new grayscale value. The mapping function is usually linear, that is, the new grayscale value is equal to the value of the cumulative distribution function multiplied by the maximum grayscale value (255).

[0097] 4) Apply the mapping function: Replace the gray value of each pixel in the original image with the new gray value after mapping to obtain the equalized image.

[0098] 11. Non-ideal and ideal lighting conditions: In this application, non-ideal lighting conditions may include low-light conditions (e.g., nighttime) and / or high-light conditions. A light intensity of 50 lux or less can be defined as a low-light condition, and a light intensity of 50,000 lux or more can be defined as a high-light condition. Conversely, in this application, a light intensity greater than 50 lux and less than 50,000 lux can be defined as an ideal lighting condition.

[0099] 12. Brute-force feature matching: This is a fundamental method for feature point matching in computer vision and image processing. This method finds the best matching pair by comparing all feature points in two images one by one.

[0100] 13. Intelligent Agent: In computer science and artificial intelligence, an intelligent agent is an entity that can perceive its environment and take actions to achieve a certain goal. Examples of intelligent agents include vehicles, robots, and drones.

[0101] With the development of artificial intelligence-related technologies, people have increasingly higher requirements for the perception capabilities of intelligent agents. Perception systems, represented by visual sensors (such as pinhole cameras and fisheye cameras), are one of the most fundamental and extensively researched effective environmental information inputs in the fields of intelligent driving and even robotics. Various processing and analysis are performed on the images acquired by visual sensors to achieve complex visual tasks. Taking intelligent driving as an example, a multi-view camera system deployed around the vehicle body can provide the vehicle with all-around environmental perception capabilities. Through research and analysis, the inventors of this application have found that, based on currently known image processing methods, the perception accuracy of intelligent agents is low under conditions of very weak light intensity (e.g., at night) or very strong light intensity (i.e., under non-ideal lighting conditions). In other words, based on currently known image processing methods, there is a significant dependence on light intensity; only under suitable light intensity will the vehicle's environmental perception accuracy not be too low.

[0102] To address the aforementioned problems, this application provides an image processing method, apparatus, and intelligent agent. By obtaining a reflection map from the original image acquired by a visual sensor, and then analyzing and processing it, the dependence on the light intensity in the environment can be reduced, thereby improving the accuracy of the intelligent agent's perception of the environment.

[0103] Before detailing the image processing method provided in the embodiments of this application, the image processing apparatus applied to the embodiments of this application will now be described exemplarily with reference to FIG1.

[0104] Figure 1 is a schematic structural block diagram of an image processing device provided in an embodiment of this application.

[0105] As shown in Figure 1, the image processing device 100 may include a preprocessing module 101, a matching module 102, and a fusion module 103.

[0106] The preprocessing module 101 can be used to separate the reflectance map and the brightness map from the original image acquired by the vision sensor; the preprocessing module 101 or the matching module 102 can be used to obtain a first weight set based on the brightness map, which includes the weight of the pixel brightness corresponding to each pixel in the brightness map; the matching module 102 can also be used to extract features based on the reflectance map and track and match feature points; or, the matching module 102 can also be used to track and match pixels based on the reflectance map; the fusion module 103 can be used to obtain the reprojection error based on the first weight set, and then obtain the extrinsic parameters (which can be simply referred to as the extrinsic matrix) of the vision sensor based on the reprojection error.

[0107] It is understood that the module division of the image processing device 100 in Figure 1 is merely exemplary. In practical applications, different functional modules can be divided according to different functional requirements. This application does not limit the form and number of functional modules in practical applications, and Figure 1 does not limit this application in any way. As an example and not a limitation, in some embodiments, the image processing device may include more or fewer modules than shown, or combine some modules, or split some modules, or have different module arrangements. For example, the preprocessing module 101 and the matching module 102 may be combined into one module; as another example, the matching module 102 and the fusion module 103 may be combined into one module. This application does not limit this in any way. The functions of each functional module can be implemented in whole or in part by software, hardware, firmware, or any combination thereof.

[0108] The image processing method provided in the embodiments of this application will be described exemplarily below with reference to the accompanying drawings.

[0109] Figure 2 is a schematic flowchart of the image processing method provided in an embodiment of this application.

[0110] The method 200 shown in Figure 2 may include steps 210 and 220.

[0111] The steps of this method can be executed by an intelligent agent, or by a component (such as a chip, chip system, etc.) configured in the intelligent agent, or by a logic module or software capable of implementing all or part of the functions of the intelligent agent, or by an image processing device; this application does not limit this. The following describes in detail each step of method 200 using the example of an intelligent agent executing method 200.

[0112] In step 210, the original image of the physical environment in which the visual sensor is located is acquired based on the visual sensor.

[0113] For example, at least one visual sensor may be deployed on the agent. The agent can use the visual sensor to acquire raw images of the physical environment in which the visual sensor is located. For instance, the agent can acquire M raw images based on the visual sensor, where M is an integer greater than or equal to 2.

[0114] Understandably, an agent can acquire M raw images based on any visual sensor deployed on the agent.

[0115] In step 220, a reflection map is obtained based on the original image, which is used to determine the pose motion parameters of the vision sensor.

[0116] As an example, the agent can obtain N reflection maps based on the M original images. Here, N is an integer greater than or equal to 2 and less than or equal to M, and the above N reflection maps are used to determine the pose and motion parameters of the vision sensor.

[0117] For example, after the agent acquires M raw images based on any one of the vision sensors on the agent, the agent can process these M raw images to obtain N reflection maps, which are used to determine the pose and motion parameters of the vision sensor.

[0118] As an example and not a limitation, the agent may be equipped with the image processing device shown in FIG1, so that the agent can process the M images based on the preprocessing module 101 to obtain the N reflection images.

[0119] In one possible implementation, N reflection images are obtained based on the M original images, including: estimating the brightness of a first image among the M original images to obtain a brightness map of the first image, wherein the first image is a keyframe image among the M original images, and the M original images include the N keyframe images; and separating the brightness map from the first image to obtain a reflection map of the first image.

[0120] For example, the agent can determine N keyframe images from these M original images and perform brightness estimation on each of these N keyframe images.

[0121] For ease of description, we will use any one of these N keyframe images as an example, and denote any one of these N keyframe images as the first image. The agent can perform brightness estimation on each of these N keyframe images; that is, the agent performs brightness estimation on the first image to obtain the brightness map of the first image. As an example and not a limitation, the agent's brightness estimation of the first image to obtain the brightness map of the first image includes: the agent converts the blue (B), green (G), and red (R) (RGB) color channels of the first image into hue (H), saturation (S), and value (V) (HSV) representations, and takes the V channel as the brightness map of the first image.

[0122] Furthermore, the agent can separate the brightness map of the first image from the first image based on log(I(x,y))=log(R(x,y))+log(B(x,y)) (referred to as Formula 1 for ease of description), to obtain the reflection map of the first image. Here, I(x,y) can represent the pixel at pixel position (x,y) in the original image, B(x,y) represents the brightness value of the pixel at pixel position (x,y) in the brightness map, and R(x,y) represents the pixel at pixel position (x,y) in the reflection map.

[0123] Figure 3 is a schematic comparison of the original image, brightness map, and reflection map provided in the embodiments of this application.

[0124] As shown in Figure 3, by separating the brightness map B(x,y) of the first image from the original image I(x,y) corresponding to the first image, the reflection map R(x,y) of the first image can be obtained.

[0125] In another possible implementation, N reflection images are obtained based on the M original images, including: estimating the brightness of a second image among the M original images to obtain a brightness map of the second image, wherein the second image is any one of the M original images; separating the brightness map from the second image to obtain a reflection map of the second image; and determining N reflection maps from the M reflection maps.

[0126] For example, the agent can perform brightness estimation on each of the M original images. For ease of description, let's take any one of the M original images as an example, and denote any one of the M original images as the second image. The agent performs brightness estimation on each of the M original images, that is, the agent performs brightness estimation on the second image to obtain the brightness map of the second image. As an example and not a limitation, the agent's brightness estimation on the second image to obtain the brightness map of the second image includes: the agent converts the RGB three color channels of the second image to HSV representation, and takes the V channel as the brightness map of the second image. Furthermore, the agent can separate the brightness map of the second image from the second image based on the above formula one to obtain the reflection map of the second image. Further, the agent can determine N representative or important reflection maps from these M reflection maps.

[0127] After obtaining the first reflection image, the agent can further optimize it. For example, morphological operations can be performed on the first reflection image, noise can be removed through filtering, and methods such as histogram equalization can be used to enhance local contrast. For explanations of morphological operations and histogram equalization, please refer to the relevant descriptions in the technical terminology section above; for brevity, they will not be repeated here.

[0128] The filtering methods applicable to the image processing methods in the embodiments of this application may include, but are not limited to: mean filtering, median filtering, Gaussian filtering, bilateral filtering, non-local means filtering, wavelet transform denoising, convolutional neural networks (CNN) denoising, etc.

[0129] In one possible implementation, the method 200 further includes: obtaining a first weight set of the first brightness map based on the first brightness map, the first weight set including the weight of the pixel brightness corresponding to each pixel point in the first brightness map, the first brightness map being any one of the N brightness maps used to obtain the N reflection maps.

[0130] For example, the agent can process the first brightness map based on a predefined or pre-trained brightness weight response function.

[0131] It can be understood that the brightness value of each pixel in the first brightness map is the input data of the brightness weight response function, and the weight corresponding to the brightness value of each pixel in the first brightness map is the output data of the brightness weight response function. That is, the first weight set is the output data of the brightness weight response function.

[0132] Optionally, the algorithm logic for processing the brightness map using the brightness weight response function to obtain the first weight set is as follows: a first threshold L is preset. lower Second threshold L upper And L lower <L upper In L lower The brightness value corresponding to the pixel in the brightness map is ≤L upper In the case of 0 ≤ luminance value < L, output high weights; lower In the case of, or, in L upper If the brightness value of a pixel in the brightness map is less than or equal to 255, output a low weight.

[0133] For illustrative purposes only, not as a limitation, an example, the first threshold L lower =80, second threshold L upper =180. Another example: the luminance map includes a*b pixels. The a*b luminance values ​​corresponding to these a*b pixels are sorted from low to high without deduplication. The 16th percentile luminance value is determined as the first threshold L. lower The value of L is determined by setting the lower 84% of brightness values ​​as the second threshold. upper The values ​​are determined by the given information. For example, if a = 100 and b = 100, the luminance map has 10,000 pixels. These 10,000 luminance values ​​are sorted from low to high without deduplication. The 1600th luminance value (obtained from 10,000 × 16%) is then determined as the first threshold L. lower The value of the 8400th luminance value (obtained from 10000 × 84%) is determined as the second threshold L. upper The values ​​of the first and second thresholds are not limited in this application. However, in practical applications, this application does not limit the specific values ​​of the first and second thresholds, and they can be set according to different needs.

[0134] Figure 4 is a comparative diagram of the pixel brightness distribution histogram under non-ideal lighting conditions, the pixel brightness distribution histogram under ideal lighting conditions, and the brightness weight response function provided in the embodiments of this application.

[0135] Figure 4a) shows a histogram of pixel brightness distribution of an original image acquired under non-ideal lighting conditions. As shown in Figure 4a), the pixel brightness of the original image acquired under non-ideal lighting conditions is mostly distributed at both ends (left and right ends, with pixel brightness close to 0 on the left and close to 255 on the right), with a small portion distributed in the middle. Figure 4b) shows a histogram of pixel brightness distribution of an original image acquired under ideal lighting conditions. As shown in Figure 4b), the pixel brightness of the original image acquired under ideal lighting conditions is mostly distributed in the middle, with a small portion distributed at both ends. Figure 4c) shows a schematic diagram of the brightness weight response function. As shown in Figure 4c), the brightness weight response function aims to optimize the performance of the image by applying a weighted average of the two pixels that satisfy the L... lower The brightness value corresponding to the pixel in the brightness map is ≤L upper Pixels with high weights are assigned to pixels with a brightness value less than L in the brightness map. lower Pixels are assigned low weights.

[0136] It is understandable that the high and low weights mentioned above are relative values.

[0137] For example, taking integer values ​​between 0 and 10 as an example, low weights can be values ​​between 0 and 5 (inclusive), and high weights can be values ​​between 6 and 10 (inclusive). Alternatively, the agent can also use a normalization operation to map high and low weights to the range of 0 to 1.

[0138] Another example, taking rational numbers between 0 and 1, is that low weights can satisfy: 0 ≤ low weights ≤ 0.5, and high weights can satisfy: 0.5 < high weights ≤ 1.

[0139] Another example, using 0 and 1, shows that a low weight can be 0 and a high weight can be 1. That is, in this implementation, in the first weight set, the weight corresponding to the first pixel is 1, and the weight corresponding to the second pixel is 0. The first pixel is a pixel whose brightness is greater than or equal to a first threshold and less than or equal to a second threshold, and the second pixel is a pixel whose brightness is less than the first threshold or greater than the second threshold. In other words, in L... lower The brightness value corresponding to the pixel in the brightness map is ≤L upper In the case where the output weight is 1, the brightness value corresponding to the pixel in the brightness map is <L. lower In the case of, or, in L upperWhen the brightness value corresponding to a pixel in the brightness map is less than the output weight, the weight is 0. It can be understood that in this implementation, the first weight set contains only two values: 0 and 1. Optionally, in this implementation, the aforementioned brightness weight response function can be called a 0-1 brightness weight function, which satisfies the following: (For ease of description, this formula is referred to as Formula 2), where B(x, y) can represent the brightness value of the pixel at pixel position (x, y), 0≤L lower <L upper ≤255, f(x,y) represents the weight of the brightness value at pixel position (x,y).

[0140] As an example and not a limitation, the agent may be equipped with the image processing device shown in FIG1, so that the agent can obtain the first weight set of the first brightness map based on the preprocessing module 101 or the matching module 102.

[0141] In one possible implementation, the method 200 further includes: extracting features from a first reflection map among the N reflection maps to obtain a first set of feature points, wherein the first reflection map is any one of the N reflection maps.

[0142] For example, the agent can extract features from the first reflection map to obtain a first set of feature points.

[0143] As an example and not a limitation, an agent may extract features from a first reflection map using a corner detection method. Corner detection methods applicable to embodiments of this application may include, but are not limited to: Harris corner detection algorithm, Shi-Tomasi corner detection algorithm, features from accelerated segment test (FAST) corner detection algorithm, scale-invariant feature transform (SIFT) corner detection algorithm, oriented accelerated segment test (FAST) features, and binary robust independent elementary features (BRIEF) (ORB), etc.

[0144] In another example, the agent can extract features from the first reflection map using a pre-trained feature detection network. The feature detection network applicable to the embodiments of this application may include, but is not limited to: SuperPoint feature point detection and description algorithm, learning local features net (LF-Net) algorithm, reliable and repeatable detectors and descriptors (R2D2) algorithm, end-to-end feature point detection and description net (D2-Net) algorithm, key net (KeyNet) algorithm, etc.

[0145] As an example and not a limitation, the agent may be equipped with the image processing device shown in Figure 1, so that the agent can extract features from the first reflection map of the N reflection maps based on the matching module 102 to obtain a first feature point set.

[0146] In one possible implementation, the method 200 further includes: optimizing and filtering the feature points in the first feature point set based on the first weight set to obtain a second feature point set, wherein the first feature point set is the feature point set obtained after feature extraction based on the first reflection map, and the first reflection map is any one of the N reflection maps.

[0147] For example, as described above, the agent can extract features from the first reflection map to obtain a first set of feature points, and the agent can also process the first brightness map to obtain a first set of weights. It is understood that the first reflection map and the first brightness map are separated from the same original image. Furthermore, the agent can optimize and filter the first set of feature points based on the first set of weights to obtain a second set of feature points.

[0148] As an example and not a limitation, the agent may be equipped with the image processing device shown in Figure 1, so that the agent can optimize and filter the feature points in the first feature point set based on the matching module 102 to obtain the second feature point set.

[0149] Figure 5 is a schematic diagram comparing the effects of feature extraction based on the original image and the reflection map, respectively, according to the embodiments of this application.

[0150] Figure 5a) shows the effect of feature extraction based on the original image; Figure 5b) shows the effect of feature extraction based on the reflection map and the feature points optimized and filtered by the first weight set. As can be seen from Figure 5, the feature points in Figure 5b) are much richer than those in Figure 5a).

[0151] In one possible implementation, the method 200 further includes: matching feature points in the second feature set of the nth reflection map among the N reflection maps with feature points in the second feature point set of the (n+1)th reflection map to obtain a first feature point pair set, where n is an integer greater than or equal to 1 and less than or equal to N-1; obtaining a second weight set based on the difference in pixel brightness corresponding to each pair of feature points in the first feature point pair set, the second weight set including the weight corresponding to each pair of feature points in the first feature point pair set; and optimizing and filtering the feature points in the first feature point pair set based on the second weight set to obtain a second feature point pair set.

[0152] For example, based on the time of the M original images acquired, these N reflection maps also have a chronological order, meaning that these N reflection maps can be arranged according to their chronological order. The agent can perform feature matching on every two adjacent reflection maps among these N reflection maps; that is, it matches the feature points in the second feature set of the nth reflection map with the feature points in the second feature point set of the (n+1)th reflection map to obtain a first feature point pair set.

[0153] Optionally, the agent can use brute-force feature matching to match feature points in the second feature set of the nth reflection map with feature points in the second feature point set of the (n+1)th reflection map. For a description of brute-force feature matching, please refer to the relevant description in the technical terminology section above; for brevity, it will not be repeated here.

[0154] Optionally, the agent can use a fast approximate nearest neighbor matching method to match feature points in the second feature set of the nth reflection map with feature points in the second feature point set of the (n+1)th reflection map. As an example and not a limitation, the fast approximate nearest neighbor matching method applicable to embodiments of this application may include, but is not limited to: locality-sensitive hashing (LSH), fast library for approximate nearest neighbors (FLANN), approximate nearest neighbors oh yeah (Annoy), etc.

[0155] Furthermore, after obtaining the first set of feature point pairs, the agent can optimize and filter the first set of feature point pairs based on the difference in pixel brightness corresponding to each pair of feature points in the first set to obtain a second set of weights. In one possible implementation, the agent can process the first set of feature point pairs based on a predefined or pre-trained inter-frame pixel brightness consistency weight function and the difference in pixel brightness corresponding to each pair of feature points in the first set to obtain the second set of weights. It is understood that the difference in pixel brightness corresponding to each pair of feature points in the first set is the input data of the inter-frame pixel brightness consistency weight function, and the weights corresponding to the difference in pixel brightness corresponding to each pair of feature points in the first set are the output data.

[0156] Optionally, the algorithm logic for processing the first feature point set based on the inter-frame pixel brightness consistency weight function is as follows: a preset third threshold L diff , in d n*(n+1) (x, y) ≤ L diff In the case of d, output the high weight; n*(n+1) (x, y) > L diff In this case, output the lower weight. d n*(n+1) (x, y) represents the difference (absolute value) between the pixel brightness at pixel position (x, y) in the nth reflection map and the (n+1)th reflection map.

[0157] As an example, and not a limitation, the third threshold L is an example. diff =100. In another example, the pixel brightness differences corresponding to each pair of feature points in the first feature point set are sorted from high to low without deduplication. The 25th and 75th differences are then compared again, and the difference between the 25th and 75th differences is determined as the third threshold L. diff However, in practical applications, this application does not limit the specific value of the third threshold, which can be set according to different needs.

[0158] It is understandable that the high and low weights mentioned above are relative values.

[0159] For example, taking integer values ​​between 0 and 10 as an example, low weights can be values ​​between 0 and 5 (inclusive), and high weights can be values ​​between 6 and 10 (inclusive). Alternatively, the agent can also use a normalization operation to map high and low weights to the range of 0 to 1.

[0160] Another example, taking rational numbers between 0 and 1, is that low weights can satisfy: 0 ≤ low weights ≤ 0.5, and high weights can satisfy: 0.5 < high weights ≤ 1.

[0161] Another example, using 0 and 1, can have a low weight of 0 and a high weight of 1. That is, in this implementation, in the second weight set, the weight corresponding to the first feature point pair is 1, and the weight corresponding to the second feature point pair is 0. The first feature point pair is a pair in the first feature point pair set where the difference in pixel brightness between the two feature points is less than or equal to a third threshold, and the second feature point pair is a pair in the first feature point pair set where the difference in pixel brightness between the two feature points is greater than the third threshold. In other words, in d... n*(n+1) (x, y) ≤ L diff In the case of d, the weight of the output is 1; n*(n+1) (x, y) > L diff In this case, the output weight is 0. Where d n*(n+1) (x, y) = |B n (x, y)-B (n+1) (x, y)|,B n (x, y) represents the pixel brightness value at pixel position (x, y) of the nth reflectance map, B (n+1) (x, y) represents the pixel brightness value at pixel position (x, y) of the (n+1)th reflectance image.

[0162] Understandably, in this implementation, the values ​​in the second weight set are only 0 and 1. Optionally, in this implementation, the aforementioned inter-frame pixel brightness consistency weight function can be called the 0-1 inter-frame pixel brightness consistency weight function, which satisfies the following: (For ease of description, this formula will be referred to as Formula 3), where B n (x, y) represents the pixel brightness value at pixel position (x, y) of the nth reflectance map, B (n+1) (x, y) represents the pixel brightness value at pixel position (x, y) of the (n+1)th reflectance image, d n*(n+1) (x, y) represents the absolute value of the difference in brightness between the pixel at position (x, y) in the nth and (n+1)th reflection maps, g n*(n+1) (x, y) represents the brightness consistency weight at pixel position (x, y) in the nth reflectance map corresponding to the (n+1)th reflectance map.

[0163] Furthermore, after obtaining the second set of weights, the agent can optimize and filter the first set of feature point pairs based on the second set of weights to obtain the second set of feature point pairs. For example, if the second set of weights contains only 0 and 1 values, then feature point pairs in the second set of weights that have a value of 1 can be retained, while feature point pairs in the second set of weights that have a value of 0 can be removed, thus obtaining the second set of feature point pairs.

[0164] As an example and not a limitation, the agent may be equipped with the image processing device shown in FIG1, so that the agent can optimize and filter the feature points in the first feature point pair set based on the matching module 102 to obtain the second feature point pair set.

[0165] Figure 6 is a comparative schematic diagram of feature matching based on the original image and the reflection map respectively, provided by an embodiment of this application.

[0166] Figure 6a) shows the effect of feature matching based on the original image; Figure 6b) shows the effect of feature matching based on the reflection map and optimized filtering of feature point pairs by the second weight set. As can be seen from Figure 6, the matching results in Figure 6b) are richer than those in Figure 6a).

[0167] In one possible implementation A, the method 200 further includes: estimating the pose motion parameters of the visual sensor based on the second set of feature point pairs: initial value R0 of the rotation matrix and initial value t0 of the translation vector; constructing a reprojection error based on the first set of weights and the second set of feature point pairs, the reprojection error satisfying: (For ease of description, this formula is referred to as Formula 4), where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p j Let j represent the j-th feature point of the first reflection map in the set of second feature point pairs corresponding to the first reflection map, where 1≤j≤k, and k indicates that the set of second feature point pairs corresponding to the first reflection map includes k feature point pairs; This represents the pose motion parameters R of the visual sensor at the target time. j and t j Regarding p j The reprojected point, where the target time is the time when the original image of the first reflection map was acquired; W k This represents the weight of the k-th feature point pair in the set of second feature point pairs corresponding to the first reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0168] For example, after obtaining the second set of feature point pairs, the agent can estimate the initial values ​​R0 and t0 of the pose motion parameters of the visual sensor based on this second set of feature point pairs. As an example, and not a limitation, for pinhole cameras or fisheye cameras, the agent can use a 2-dimensional (2D)-2D epipolar geometry method to coarsely estimate the pose motion parameters of the inter-frame camera: the initial value R0 of the rotation matrix and the initial value t0 of the translation vector. For RGB-D cameras, the agent can use a 3-dimensional (3D)-3D iterative closest point (ICP) algorithm to coarsely estimate the pose motion parameters of the inter-frame camera: the initial value R0 of the rotation matrix and the initial value t0 of the translation vector.

[0169] Understandably, after matching inter-frame feature point pairs, the problem can be transformed into a reprojection problem to solve for the pose of the visual sensor and the coordinates of the feature world points. More specifically, the agent can construct the reprojection error based on the first set of weights and the second set of feature point pairs.

[0170] Furthermore, the agent can substitute the initial values ​​R0 and t0 of the pose motion parameters from the visual sensor into Formula 4 above for optimization iteration (the optimization method can be the LM method). When the preset convergence condition is met, the result is considered converged, the iteration ends, and the final inter-frame camera pose motion parameters R and t are obtained. As an example, and not a limitation, one example is that the preset convergence condition can be that the difference between the results of two iterations (or the deviation change between the results of two iterations) is less than or equal to a preset fourth threshold; another example is that the preset convergence condition can be that the number of iterations reaches a preset fifth threshold.

[0171] In a possible implementation B, the method 200 further includes: estimating the pose motion parameters of the visual sensor based on the second set of feature point pairs: initial value R0 of the rotation matrix and initial value t0 of the translation vector; and constructing a reprojection error based on the first set of weights and the second set of feature point pairs, the reprojection error satisfying: (For ease of description, this formula is referred to as Formula 5), ​​where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p i Let i represent the feature points in the feature point pair in the set of the second feature point pairs corresponding to the i-th reflection map, where 1≤i≤n; This represents the pose motion parameters R of the visual sensor at the target time. i and t i Regarding p iThe reprojected point, where the target time is the time when the original image of the i-th reflection map was acquired; W i This represents the first weight set corresponding to the i-th reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0172] For example, after obtaining the second set of feature point pairs, the agent can estimate the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor based on the second set of feature point pairs. A detailed description can be found in the relevant description in Implementation A above; for brevity, it will not be repeated here.

[0173] Understandably, after matching inter-frame feature point pairs, the problem can be transformed into a reprojection problem to solve for the pose of the visual sensor and the coordinates of the feature world points. More specifically, the agent can construct the reprojection error based on the first set of weights and the second set of feature point pairs.

[0174] Furthermore, the agent can substitute the initial values ​​R0 and t0 of the pose motion parameters from the visual sensor into Formula 5 above for optimization iteration (the optimization method can be the LM method). When the preset convergence condition is met, the result is considered converged, the iteration ends, and the final inter-frame camera pose motion parameters R and t are obtained. For a detailed description of the preset convergence condition, please refer to the relevant description in Implementation Method A above. For the sake of brevity, it will not be repeated here.

[0175] It is important to understand that the difference between implementation A and implementation B lies in the fact that in the reprojection error formula (i.e., formula four) of implementation A, p j This represents the j-th feature point of the first reflection map in the set of second feature point pairs corresponding to the first reflection map, that is, a certain feature point of the first reflection map in the set of second feature point pairs corresponding to the first reflection map; in the reprojection error formula (i.e., formula five) of the above implementation method B, p i It represents the feature points in the feature point pairs in the second feature point pair set corresponding to the i-th reflection map, that is, all feature points of the first reflection map in the second feature point pair set corresponding to a certain first reflection map.

[0176] As an example and not a limitation, the agent may be equipped with the image processing device shown in FIG1, so that the agent can calculate the pose motion parameters R and t of the visual sensor on the agent based on the first weight set and the second feature point set based on the fusion module 103.

[0177] Understandably, the above method, which involves extracting features from a reflection image and then optimizing and processing the feature points, can be called the feature point method.

[0178] In one possible implementation, the method 200 may further include: processing the N reflection images based on optical flow tracing to obtain the pose motion parameters R and t of the visual sensor.

[0179] For example, the agent can estimate the pixel motion (u, v) of the pixel within a preset image window in a short time Δt based on the brightness invariance assumption (that is, assuming that the brightness value of the pixel at the same spatial point is fixed in each image). The pixel position before and after the motion can be used as the feature point pair tracked in time Δt. Based on the feature point pair tracked in time Δt, the pose motion parameters R and t of the visual sensor in time Δt can be estimated.

[0180] In one possible implementation, the method 200 may further include: processing the N reflection images based on a direct method to obtain the pose motion parameters R and t of the visual sensor.

[0181] For example, the agent can calculate the position of the pixel in the image in the next frame by adjusting the coarsely estimated pose motion parameters of the inter-frame visual sensor for each pixel based on the N reflection maps. When the convergence condition is met (e.g., the brightness difference at the corresponding pixel is less than the preset sixth threshold), it is considered that the pose motion parameters R and t of the inter-frame visual sensor at this time meet the requirements, that is, the pose motion parameters R and t of the visual sensor are obtained.

[0182] As an example and not a limitation, the agent may be equipped with the image processing device shown in Figure 1, so that the agent can process the N reflection images based on the fusion module 103 to obtain the pose motion parameters R and t of the visual sensor on the agent.

[0183] In one possible implementation, method 200 can be applied to the calibration of external parameters of a vision sensor.

[0184] For example, in an application scenario involving the calibration of the external parameters of a visual sensor, the agent can obtain the pose motion parameters R and t of the visual sensor on the agent based on the method 200 described above. Furthermore, the agent can also calculate the external parameters of the visual sensor based on the obtained pose motion parameters R and t.

[0185] For example, we can define the spatial transformation relationship from the agent to the vision sensor as X, the inter-frame vision sensor pose motion as A, and the inter-frame agent pose motion as B. The following relationship can exist between A, B, and X: AX = XB (i.e., the hand-eye calibration formula). A and B are known conditions, so X can be solved, which means the external parameters of the vision sensor can be calculated.

[0186] Figure 7 is a schematic hand-eye calibration diagram for external parameter calibration of a vision sensor provided in an embodiment of this application.

[0187] As shown in Figure 7, the visual sensor is fixed to the intelligent agent and can be regarded as a rigid body. and All can be represented as spatial position transformation relationships (including rotation and translation) from the agent to the visual sensor; C i and C j V represents the state of the visual sensor at time i and time j, respectively. i and V j These represent the states of the agent at time i and time j, respectively. and Let i and j represent the motion of the visual sensor and the motion of the agent between frames i and j, respectively. and The following relationships can exist between them: The spatial position transformation relationship from the known agent to the visual sensor satisfies but This can be converted to AX = XB, and by solving this equation, the spatial position transformation relationship between the agent and the visual sensor can be obtained. and That is, the external parameters X of the vision sensor can be obtained.

[0188] It is understandable that time i can be interpreted as the time represented by the timestamp of the original image corresponding to the nth reflection image, and time j can be interpreted as the time represented by the timestamp of the original image corresponding to the (n+1)th reflection image.

[0189] Understandably, the external parameter calibration of vision sensors (such as cameras) can be divided into two main categories: offline calibration and online calibration. Offline calibration can be designed with targets containing rich artificial features to achieve high-precision external parameter calibration, and can also reduce environmental impact by constructing calibration chambers. However, in practical applications, offline calibration increases the number of times users need to return to the store for regular maintenance, but many users do not have time for regular maintenance. Reducing the number of user returns is a need for most users. In this case, online calibration, which is not limited by location, better meets this need. However, based on current online calibration methods, they are often limited by environmental and weather conditions. For example, under low light conditions, the image quality and the number of feature points acquired by light-sensitive vision sensors will decrease, thus affecting the accuracy of the calibration results. Furthermore, based on current known online calibration methods, when there is a large angular deviation in the external parameters of the vision sensor (e.g., a collision causing the vision sensor to rotate or translate), the external parameter change correction function of the vision sensor is prone to errors or failure, and even the external parameter change monitoring function may fail, thus creating some safety risks.

[0190] Online calibration of vision sensors based on the image processing scheme provided in this application can overcome scene limitations under non-ideal lighting conditions (e.g., nighttime), enabling rapid and high-precision online calibration of vision sensors. This avoids safety risks caused by large angular deviations in the extrinsic parameters of the vision sensors. In the field of intelligent driving, it can also improve the safety of vehicles performing intelligent driving at night. Furthermore, the vehicle production line already possesses the function of online calibration of nighttime vision sensors, eliminating the need to enter a calibration room for calibration, and there are no requirements regarding open road lighting conditions.

[0191] In one possible implementation, method 200 can be applied to visual odometry.

[0192] For example, in a visual odometry scenario, the agent can obtain the pose and motion parameters R and t of the visual sensor on the agent based on the method 200 described above. Furthermore, based on the obtained pose and motion parameters R and t of the visual sensor, the agent can calculate the spatial position (e.g., three-dimensional spatial position) of each feature point through triangulation constraints, and accumulate the data frame by frame to obtain a map (generally described as a point cloud). This map can be used in the localization and reconstruction domain. That is, given the known intrinsic parameters of the visual sensor and the motion of the agent containing the visual sensor, the problem can be transformed into a BA problem through triangulation constraints.

[0193] Figure 8 is a schematic diagram of triangulation constraints.

[0194] As an example, and not a limitation, as shown in Figure 8, k can represent the k-th frame image, and k+1 can represent the k+1-th frame image; i can represent the i-th pair of feature points in every pair of images; O k O can represent the origin of the camera coordinate system corresponding to the k-th frame image. k+1 It can represent the origin of the camera coordinate system corresponding to the (k+1)th frame image; p i k p can represent the pixel of the feature point in the i-th pair of feature points in the k-th frame of the image. i k+1 It can represent the pixel of the feature point of the (k+1)th frame image to which the i-th pair of feature points belongs, that is, p i k and p i k+1 This can be understood as the projection point; s i k The depth s can represent the depth of the feature point in the i-th pair of feature points in the k-th frame image to the actual spatial point P. i k+1 This can represent the depth of the feature point in the (k+1)th frame of the image to the actual spatial point P, belonging to the i-th pair of feature points; additionally, not shown in Figure 8, x i k x can represent the feature point of the k-th frame image in the i-th pair of feature points in three-dimensional space. i k+1 It can represent the feature points of the (k+1)th frame image in the i-th pair of feature points in the three-dimensional space.

[0195] Based on the triangulation constraints shown in Figure 8, the following formula can be constructed: s i k+1 ·x i k+1 =s i k ·R·x i k +t (for ease of description, this formula will be referred to as Formula Six), multiply both sides of Formula Six by x. i (k+1)^, we can get: s i k+1 ·x i (k+1)^ ·x i k+1 =s i k ·x i (k+1) ^ ·R·x ik +x i (k+1)^ ·t, which can be further derived to: s i k ·x i (k+1)^ ·R·x i k +x i (k+1)^ ·t=0 (for ease of description, this formula is referred to as Formula 7). Based on Formula 6 and Formula 7, as well as multiple pairs of feature points, solving the overdetermined system of equations yields s. i k and s i k+1 .

[0196] Furthermore, based on s i k and x i k P can be constructed O k =s i k ·x i k (For ease of description, this formula will be referred to as Formula Eight), P O k It can represent that a point P in space lies in O. k The three-dimensional coordinates below; without loss of generality, P O k+1 =s i k+1 ·x i k+1 (For ease of description, this formula will be referred to as Formula Nine), P O k+1 It can represent that a point P in space lies in O. k+1 The three-dimensional coordinates below.

[0197] Furthermore, based on Formula 8 and Formula 9, we can obtain x. i k =K -1 ·p i k (For ease of description, this formula is referred to as Formula Ten) and x i k+1 =K -1 ·p i k+1 (For ease of description, this formula will be referred to as Formula Eleven), and P O k+1 =R·P O k+t (for ease of description, this formula will be referred to as Formula Twelve), and then according to O k and O k+1 By relating it to the world coordinate system, spatial point P can be obtained, and the set of multiple spatial points P can then be a map. In Equations 10 and 11, K can represent the intrinsic parameters of the visual sensor.

[0198] Furthermore, it is understandable that the above method 200 can be applied to the field of localization and reconstruction. In other words, the above method 200 has good universality and transferability, and can be applied to many scenarios and fields.

[0199] To facilitate better understanding, the image processing method provided in the embodiments of this application will be briefly explained again below with reference to Figure 9.

[0200] Figure 9 is another schematic flowchart of the image processing method provided in the embodiments of this application.

[0201] Steps 901 to 907 shown in Figure 9 can be executed by an intelligent agent, or the method can be executed by a component (such as a chip, chip system, etc.) configured in the intelligent agent, or it can be implemented by a logic module or software capable of realizing all or part of the functions of the intelligent agent, or it can be executed by an image processing device. This application does not limit this. The following uses the execution of the method by an intelligent agent as an example to briefly explain each step shown in Figure 9.

[0202] In step 901, the vision sensor acquires the raw image.

[0203] For example, the agent can acquire the raw image based on any visual sensor deployed on the agent. A detailed description can be found in the relevant explanation of step 210 in method 200 above, and for the sake of brevity, it will not be repeated here.

[0204] In step 902, the original image is processed to obtain a brightness map and a reflection map.

[0205] For example, the agent can process the raw image acquired by the vision sensor. For instance, it can perform brightness estimation on the raw image to obtain a brightness map, and then separate the brightness map from the raw image to obtain a reflection map. A detailed description can be found in the relevant description in method 200 above, and for the sake of brevity, it will not be repeated here.

[0206] In step 903, the reflectance image is denoised and its details are enhanced.

[0207] For example, after obtaining the reflection map, the agent can optimize the reflection map. For instance, it can perform morphological operations on the first reflection map or remove noise through filtering. It can also use methods such as histogram equalization to enhance local contrast. For a detailed description, please refer to the relevant description in Method 200 above. For the sake of brevity, it will not be repeated here.

[0208] In step 904, a first weight set and a second weight set are obtained based on the brightness map.

[0209] For example, the agent can obtain a first weight set of the brightness map based on the brightness map. The first weight set includes the weight of the pixel brightness corresponding to each pixel point in the brightness map. For a detailed description, please refer to the relevant description in method 200 above. For the sake of brevity, it will not be repeated here.

[0210] The agent can also obtain a second weight set based on the difference in pixel brightness corresponding to each pair of feature points in the brightness map and the first feature point pair set. For details regarding the first feature point pair set, please refer to method 200 above or step 906 below; for brevity, it will not be repeated here. It is understood that the agent can calculate the first feature point pair set, and then obtain the second weight set based on the difference in pixel brightness corresponding to each pair of feature points in the brightness map and the first feature point pair set.

[0211] In step 905, feature extraction is performed on the reflection map to obtain a first set of feature points, and the first set of feature points is optimized to obtain a second set of feature points.

[0212] For example, the agent can extract features from the reflection map and optimize and filter the extracted features to obtain a second set of feature points. A detailed description can be found in the relevant explanation in Method 200 above; for brevity, it will not be repeated here.

[0213] In step 906, feature point pair matching is performed on the nth reflection map and the (n+1)th reflection map to obtain a first feature point pair set, and the first feature point pair set is optimized to obtain a second feature point pair set.

[0214] For example, the agent can perform feature point pair matching on the nth reflection map and the (n+1)th reflection map to obtain a first set of feature point pairs. As an example and not a limitation, the agent can use brute-force feature matching or fast approximate nearest neighbor matching to match the feature points in the second feature set of the nth reflection map among the N reflection maps mentioned in method 200 with the feature points in the second feature point set of the (n+1)th reflection map. A detailed description can be found in the relevant explanation in method 200 above; for brevity, it will not be repeated here.

[0215] After obtaining the first set of feature point pairs, the agent can optimize and filter the first set of feature point pairs based on the difference in pixel brightness corresponding to each pair of feature points in the first set to obtain the second set of weights. For a detailed description, please refer to the relevant explanation in Method 200 above; for brevity, it will not be repeated here.

[0216] Furthermore, after obtaining the second set of weights, the agent can optimize and filter the first set of feature point pairs based on the second set of weights to obtain the second set of feature point pairs. For a detailed description, please refer to the relevant explanation in Method 200 above; for the sake of brevity, it will not be repeated here.

[0217] In step 907, the reprojection error is constructed, and the pose motion parameters of the visual sensor are calculated.

[0218] For example, the agent can construct a reprojection error based on the first weight set and the second feature point pair set, and calculate the pose motion parameters of the visual sensor based on the reprojection error. A detailed description can be found in the relevant description in method 200 above, and for the sake of brevity, it will not be repeated here.

[0219] It is understandable that the steps shown in Figure 9 can be applied to the calibration of the external parameters of a vision sensor. After calculating the pose and motion parameters of the vision sensor, the agent can also calculate the external parameters of the vision sensor based on these parameters. A detailed description can be found in the relevant explanation in Method 200 above; for brevity, it will not be repeated here.

[0220] For example, in the field of intelligent driving, after the vehicle (an example of an intelligent agent) is powered on, the above step 901 can be performed. That is, the visual sensor can collect image data (i.e., the original image) in real time, and can also transmit the collected data to the vehicle computing platform for processing. That is, the vehicle can perform the above steps 902 to 907 based on the vehicle computing platform.

[0221] It is also understandable that the steps shown in Figure 9 can be applied to visual odometry. After calculating the pose and motion parameters of the visual sensor, the agent can also calculate the spatial position (e.g., three-dimensional spatial position) of each feature point based on the obtained pose and motion parameters of the visual sensor through triangulation constraints, and accumulate the data frame by frame to obtain a map (which is generally described by a point cloud). This map can be used in the field of localization and reconstruction. For a detailed description, please refer to the relevant description in Method 200 above. For the sake of brevity, it will not be repeated here.

[0222] It should be noted that the arrows in Figure 9 only indicate the direction of data flow and do not limit the order in which the steps are executed.

[0223] Based on the above technical solutions, firstly, directly processing the reflectance map, which reflects the intrinsic properties of objects in the environment (e.g., feature-based methods, optical flow tracing methods, or direct methods), can reduce the influence of non-ideal lighting conditions and make the calculated pose and motion parameters of the visual sensor more accurate. Specifically, feature extraction from the reflectance map effectively increases the number of feature points. Furthermore, while the feature point method discards some image information, using a first weight set based on the brightness map to optimize and filter feature points effectively utilizes image brightness information and improves the accuracy of feature point extraction. Secondly, based on the brightness map and the first feature point pair set, an inter-frame pixel brightness consistency weight function is constructed, increasing the brightness relationship between matching frames (i.e., the nth reflectance map and the (n+1)th reflectance map), establishing a brightness-based correspondence or constraint. Optimizing the first feature point pair set based on the inter-frame pixel brightness consistency weight function can improve the accuracy of feature point matching. Finally, constructing a reprojection error based on the first weight set effectively utilizes brightness information and adds brightness-based constraints, improving robustness.

[0224] The image processing method provided by this application has been described in detail above with reference to the accompanying drawings. The image processing apparatus provided by the embodiments of this application will be described in detail below with reference to Figures 10 and 11.

[0225] It should be understood that the image processing apparatus shown in Figures 10 and 11 can be used to implement the functions of the intelligent agent in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In the embodiments of this application, the image processing apparatus can be the intelligent agent in any of the method embodiments shown in Figure 2 or Figure 9, or it can be a component configured in the intelligent agent (such as a chip, chip system, processor, etc.), or it can be a logic module or software capable of implementing some or all of the functions of the intelligent agent.

[0226] Figure 10 is a schematic block diagram of an image processing apparatus provided in an embodiment of this application.

[0227] As shown in Figure 10, the image processing device 1000 includes an acquisition module 1010 and a processing module 1020. This image processing device 1000 can be used to implement the functions of the intelligent agent in any of the method embodiments shown in Figure 2 or Figure 9.

[0228] For example, when the image processing device 1000 is used to implement the function of the intelligent agent in the method embodiment shown in FIG2, the acquisition module 1010 can be used to acquire the original image of the physical environment where the visual sensor is located based on the visual sensor; the processing module 1020 can be used to obtain a reflection map based on the original image, and the reflection map is used to determine the pose motion parameters of the visual sensor.

[0229] Optionally, the acquisition module 1010 can be used to acquire M original images based on the visual sensor, where M is an integer greater than or equal to 2; and the processing module 1020 can be used to obtain N reflection maps based on the M original images, where N is an integer greater than or equal to 2 and less than or equal to M.

[0230] Optionally, the processing module 1020 can be used to perform brightness estimation on the first image among the M original images to obtain a brightness map of the first image, wherein the first image is a keyframe image among the M original images, and the M original images include the N keyframe images; and to separate the brightness map from the first image to obtain a reflection map of the first image.

[0231] Optionally, the processing module 1020 can be used to perform brightness estimation on the second image among the M original images to obtain a brightness map of the second image, wherein the second image is any one of the M original images; separate the brightness map from the second image to obtain a reflection map of the second image; and determine N reflection maps from the M reflection maps.

[0232] Optionally, the processing module 1020 can also be used to extract features from the first reflection map of the N reflection maps to obtain a first set of feature points, wherein the first reflection map is any one of the N reflection maps.

[0233] Optionally, the processing module 1020 can also be used to obtain a first weight set of the first brightness map based on the first brightness map. The first weight set includes the weight of the pixel brightness corresponding to each pixel point in the first brightness map. The first brightness map is any one of the N brightness maps used to obtain the N reflection maps.

[0234] Optionally, in the first weight set, the weight corresponding to the first pixel is 1, and the weight corresponding to the second pixel is 0. The first pixel is a pixel whose pixel brightness is greater than or equal to the first threshold and less than or equal to the second threshold, and the second pixel is a pixel whose pixel brightness is less than the first threshold or greater than the second threshold.

[0235] Optionally, the processing module 1020 can also be used to optimize and filter the feature points in the first feature point set based on the first weight set to obtain a second feature point set. The first feature point set is the feature point set obtained after feature extraction based on the first reflection map, and the first reflection map is any one of the N reflection maps.

[0236] Optionally, the processing module 1020 can also be used to match the feature points in the second feature set of the nth reflection map with the feature points in the second feature point set of the (n+1)th reflection map to obtain a first feature point pair set, where n is an integer greater than or equal to 1 and less than or equal to N-1; based on the difference in pixel brightness corresponding to each pair of feature points in the first feature point pair set, a second weight set is obtained, which includes the weight corresponding to each pair of feature points in the first feature point pair set; based on the second weight set, the feature points in the first feature point pair set are optimized and filtered to obtain a second feature point pair set.

[0237] Optionally, in the second weight set, the weight corresponding to the first feature point pair is 1, and the weight corresponding to the second feature point pair is 0. The first feature point pair is a feature point pair in the first feature point pair set whose pixel brightness difference between the two feature points is less than or equal to a third threshold, and the second feature point pair is a feature point pair in the first feature point pair set whose pixel brightness difference between the two feature points is greater than a third threshold.

[0238] Optionally, the processing module 1020 can also be used to estimate the pose motion parameters of the visual sensor based on the second set of feature point pairs: the initial value R0 of the rotation matrix and the initial value t0 of the translation vector; and to construct a reprojection error based on the first set of weights and the second set of feature point pairs, wherein the reprojection error satisfies: Where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p j Let j represent the j-th feature point of the first reflection map in the set of second feature point pairs corresponding to the first reflection map, where 1≤j≤k, and k indicates that the set of second feature point pairs corresponding to the first reflection map includes k feature point pairs; This represents the pose motion parameters R of the visual sensor at the target time. j and t j Regarding p j The reprojected point, where the target time is the time when the original image of the first reflection map was acquired; W k This represents the weight of the k-th feature point pair in the set of second feature point pairs corresponding to the first reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0239] Optionally, the processing module 1020 can also be used to estimate the pose motion parameters of the visual sensor based on the second set of feature point pairs: the initial value R0 of the rotation matrix and the initial value t0 of the translation vector; and to construct a reprojection error based on the first set of weights and the second set of feature point pairs, wherein the reprojection error satisfies: Where R represents the rotation matrix of the visual sensor's motion; t represents the translation vector of the visual sensor's motion; p i Let i represent the feature points in the feature point pair in the set of the second feature point pairs corresponding to the i-th reflection map, where 1≤i≤n; This represents the pose motion parameters R of the visual sensor at the target time. i and t i Regarding p i The reprojected point, where the target time is the time when the original image of the i-th reflection map was acquired; W i This represents the first weight set corresponding to the i-th reflection map; ||·|| 2 The second normal form is represented; based on the initial values ​​R0 and t0 of the pose motion parameters of the vision sensor, and the reprojection error, the pose motion parameters R and t of the vision sensor are calculated.

[0240] Optionally, the processing module 1020 can also be used to process the reflection image based on the optical flow tracking method to obtain the pose motion parameters R and t of the visual sensor.

[0241] Optionally, the processing module 1020 can also be used to process the reflection image based on the direct method to obtain the pose motion parameters R and t of the visual sensor.

[0242] Optionally, this method is applied to the calibration of the external parameters of the vision sensor.

[0243] Optionally, this method is applied to visual odometry.

[0244] For a more detailed description of each of the above modules, please refer directly to the relevant descriptions in the method embodiment shown in Figure 2, which will not be repeated here.

[0245] It should be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0246] Figure 11 is another schematic block diagram of the image processing apparatus provided in the embodiments of this application.

[0247] The image processing apparatus 1100 can be a chip system, or it can be a device configured with a chip system to implement the methods described in the above-described method embodiments. In the embodiments of this application, the chip system can be composed of chips, or it can include chips and other discrete devices.

[0248] As shown in FIG11, the image processing device 1100 may include a processor 1110, which can be used to execute computer programs or instructions in memory to implement the steps performed by the agent in any of the method embodiments shown in FIG2 or FIG9.

[0249] Optionally, the image processing apparatus 1100 further includes a communication interface 1120. The communication interface 1120 can be used to communicate with other devices via a transmission medium, thereby enabling the image processing apparatus 1100 to communicate with other devices. The communication interface 1120 may be, for example, a transceiver, interface, bus, circuit, or a device capable of transmitting and receiving functions. The processor 1110 can use the communication interface 1120 to input and output data, and to implement the method described in any embodiment corresponding to FIG. 2 or FIG. 9. Specifically, the image processing apparatus 1100 can be used to implement the functions of the image processing apparatus in the above-described method embodiments.

[0250] Optionally, the image processing apparatus 1100 further includes at least one memory 1130 for storing program instructions and / or data. The memory 1130 is coupled to the processor 1110. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 1110 may operate in conjunction with the memory 1130. The processor 1110 may execute program instructions stored in the memory 1130.

[0251] In this application, the memory 1130 can be integrated into the processor 1110, or the processor 1110 and the memory 1130 can be set up separately. This application does not limit this.

[0252] It should be understood that the coupling in the embodiments of this application is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information interaction between devices, units, or modules. The processor 1110 may operate in conjunction with the memory 1130. The embodiments of this application do not limit the specific connection medium between the processor 1110, communication interface 1120, and memory 1130. In Figure 11, the processor 1110, communication interface 1120, and memory 1130 are connected via a bus 1140. The bus 1140 is represented by a thick line in Figure 11. The connection methods between other components are only illustrative and not intended to be limiting. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 11, but this does not indicate that there is only one bus or one type of bus.

[0253] This application also provides an intelligent agent, which includes the image processing device as described above.

[0254] This application also provides an intelligent agent, which includes modules that can implement the methods described in any of the embodiments shown in FIG2 or FIG9.

[0255] Optionally, in the embodiments of this application, the intelligent agent may include vehicles, robots, or drones, etc.

[0256] This application also provides a chip system, which includes at least one processor for supporting the implementation of the functions involved in any possible implementation of the embodiments shown in FIG2 or FIG9, such as receiving or processing data and / or instruction information involved in the above methods.

[0257] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0258] The chip system can consist of chips or include chips and other discrete components.

[0259] This application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run, can implement the steps performed by the intelligent agent in any embodiment of the method described in Figure 2 or Figure 9.

[0260] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it can implement the steps performed by the intelligent agent in any of the embodiments shown in FIG2 or FIG9.

[0261] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0262] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0263] The terms "unit," "module," etc., used in this specification can be used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. In the embodiments of this application, "unit" and "module" have the same meaning and can be used interchangeably.

[0264] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0265] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0267] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0268] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the technology, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0269] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method includes: The original image of the physical environment in which the visual sensor is located is acquired based on the visual sensor; A reflection map is obtained based on the original image, and the reflection map is used to determine the pose and motion parameters of the visual sensor.

2. The method of claim 1, wherein, The acquisition of the original image of the physical environment where the visual sensor is located based on the visual sensor includes: Based on the visual sensor, M original images are acquired, where M is an integer greater than or equal to 2; and, The process of obtaining the reflection map based on the original image includes: Based on the M original images, N reflection images are obtained, where N is an integer greater than or equal to 2 and less than or equal to M.

3. The method of claim 2, wherein, The process of obtaining N reflection images based on the M original images includes: Brightness estimation is performed on the first image among the M original images to obtain the brightness map of the first image. The first image is a keyframe image among the M original images, and the M original images include the N keyframe images. The brightness map is separated from the first image to obtain the reflection map of the first image.

4. The method of claim 2, wherein, The process of obtaining N reflection images based on the M original images includes: Brightness estimation is performed on the second image among the M original images to obtain a brightness map of the second image, where the second image is any one of the M original images; The brightness map is separated from the second image to obtain the reflection map of the second image; N reflection maps are determined from M reflection maps.

5. The method of any one of claims 2 to 4, wherein, The method further includes: Feature extraction is performed on the first reflection map among the N reflection maps to obtain a first set of feature points, wherein the first reflection map is any one of the N reflection maps.

6. The method of any one of claims 3 to 5, wherein, The method further includes: Based on the first brightness map, a first weight set of the first brightness map is obtained. The first weight set includes the weight of the pixel brightness corresponding to each pixel point in the first brightness map. The first brightness map is any one of the N brightness maps used to obtain the N reflection maps.

7. The method of claim 6, wherein, In the first set of weights, the weight of the first pixel is 1, and the weight of the second pixel is 0. The first pixel is a pixel whose brightness is greater than or equal to the first threshold and less than or equal to the second threshold, and the second pixel is a pixel whose brightness is less than the first threshold or greater than the second threshold.

8. The method of claim 6 or 7, wherein, The method further includes: Based on the first weight set, the feature points in the first feature point set are optimized and filtered to obtain the second feature point set. The first feature point set is the feature point set obtained after feature extraction based on the first reflection map. The first reflection map is any one of the N reflection maps.

9. The method of claim 8, wherein, The method further includes: The feature points in the second feature set of the nth reflection map of the N reflection maps are matched with the feature points in the second feature point set of the (n+1)th reflection map to obtain a first feature point pair set, where n is an integer greater than or equal to 1 and less than or equal to N-1. Based on the difference in pixel brightness corresponding to each pair of feature points in the first feature point pair set, a second weight set is obtained, which includes the weight corresponding to each pair of feature points in the first feature point pair set. Based on the second weight set, the feature points in the first feature point pair set are optimized and filtered to obtain the second feature point pair set.

10. The method of claim 9, wherein, In the second weight set, the weight corresponding to the first feature point pair is 1, and the weight corresponding to the second feature point pair is 0. The first feature point pair is a feature point pair in the first feature point pair set whose pixel brightness difference between two feature points is less than or equal to a third threshold. The second feature point pair is a feature point pair in the first feature point pair set whose pixel brightness difference between two feature points is greater than a third threshold.

11. The method of claim 9 or 10, wherein, The method further includes: Based on the second set of feature point pairs, the pose motion parameters of the visual sensor are estimated: the initial value of the rotation matrix R0 and the initial value of the translation vector t0. Based on the first weight set and the second feature point pair set, a reprojection error is constructed, wherein the reprojection error satisfies: wherein R represents a rotation matrix of the visual sensor motion; t represents a translation vector of the visual sensor motion; p j represents the jth feature point of the first reflection map in the second set of feature point pairs corresponding to the first reflection map, 1≤j≤k, k represents that there are k feature point pairs in the second set of feature point pairs corresponding to the first reflection map; R represents a pose motion parameter R of the visual sensor at a target time j and t j to the p j projected point after re-projection, the target time being a time at which an original image of the first reflection map is acquired; W k represents a weight value corresponding to the kth feature point pair in the second feature point pair set corresponding to the first reflection map; ‖·‖ 2 represents a two-norm; Based on the initial values ​​R0 and t0 of the pose motion parameters of the visual sensor, and the reprojection error, the pose motion parameters R and t of the visual sensor are calculated.

12. The method of claim 9 or 10, wherein, The method further includes: Based on the second set of feature point pairs, the pose motion parameters of the visual sensor are estimated: the initial value of the rotation matrix R0 and the initial value of the translation vector t0. Based on the first weight set and the second feature point pair set, a reprojection error is constructed, wherein the reprojection error satisfies: wherein R represents a rotation matrix of the visual sensor motion; t represents a translation vector of the visual sensor motion; p i represents a feature point in the feature point pair in the second set of feature point pairs corresponding to the i-th reflection map, 1≤i≤n; represents a pose motion parameter R of the visual sensor at a target time i and t i to the p i the projection point after re-projection, the target time being a time at which an original image of the ith reflection map is acquired; W i represents the first weight set corresponding to the ith reflection map; ‖·‖ 2 represents a two-norm; Based on the initial values ​​R0 and t0 of the pose motion parameters of the visual sensor, and the reprojection error, the pose motion parameters R and t of the visual sensor are calculated.

13. The method of claim 1, wherein, The method further includes: Based on the optical flow tracking method, the reflection image is processed to obtain the pose motion parameters R and t of the visual sensor.

14. The method of claim 1, wherein, The method further includes: Based on the direct method, the reflection image is processed to obtain the pose motion parameters R and t of the visual sensor.

15. The method of any one of claims 1 to 14, wherein, The method is applied to the calibration of the external parameters of the vision sensor.

16. The method of any one of claims 1 to 14, wherein, The method is applied to visual odometry.

17. An image processing apparatus characterized by comprising: The apparatus includes a module for performing the method as described in any one of claims 1 to 11.

18. An image processing apparatus characterized by comprising: Including processor and memory, among which, The memory is used to store programs; The processor is used to invoke the program so that the device performs the method as described in any one of claims 1 to 16.

19. An agent, comprising: The intelligent agent is a module used to perform the method as described in any one of claims 1 to 16.

20. An agent, comprising: The intelligent agent includes the image processing apparatus as described in claim 17 or 18.

21. The agent of claim 19 or 20, wherein, The intelligent agent includes vehicles, robots, or drones.

22. A computer readable storage medium having stored thereon a program, characterized in that, When the program is executed, the method as described in any one of claims 1 to 16 is performed.

23. A computer program product, characterised in that, Includes a computer program that, when run, causes the method as described in any one of claims 1 to 16 to be performed.

Images