Device and method for generating a visual representation of an environment

Abstract

Aspects concern a method for generating a visual representation of an environment, the method comprising receiving, for each device pose of a plurality of device poses within the environment, a lidar point cloud generated from lidar data and multiple images, wherein the lidar data and the multiple images are recorded by a device having the device pose in the environment, and an estimate of the device pose and optimizing an objective function over model parameters of a visual representation model modelling an environment shown by the received images and over, for each device pose, device pose compensations of the device pose, wherein, one or more learning rates are used for the device pose compensations which depend on the sensitivity of the lidar point cloud with respect to the device pose compensations.

Description

DEVICE AND METHOD FOR GENERATING A VISUAL REPRESENTATION OF AN ENVIRONMENTTECHNICAL FIELD

[0001] Various aspects of this disclosure relate to devices and methods for generating a visual representation of an environment.BACKGROUND

[0002] 3D Reconstruction from multi-view images (i.e. generating a visual representation of a (3D) environment (i.e. a 3D scene)) is a fundamental problem in computer vision. Multiview stereo methods often employ complex multi-stage pipelines that include feature matching, depth estimation, point clouds fusion, and ultimately, surface reconstruction from aggregated point clouds. In contrast, neural implicit methods, such as NeRF (Neural Radiance Fields), significantly simplify the pipeline by optimizing an implicit surface representation through volume rendering. Notable advancements have been made through the adoption of more expressive scene representations, advanced training strategies, and the integration of monocular priors. Despite these advances, these methods are mostly limited in reconstructing foreground objects and are computationally expensive to optimize. Further, to achieve high visual quality, they require accurately estimated camera poses, which is time-consuming (up to 10+hours for about 1000 images). Accordingly, approaches for generating a visual representation of an environment with high visual quality based on imprecise camera pose estimates (resulting in noisy point clouds) are desirable.SUMMARY

[0003] Various embodiments concern a method for generating a visual representation of an environment, the method comprising receiving, for each device pose of a plurality of device poses (for different time frames, i.e. sensor data acquisition times, wherein a device pose includes a device position and a device orientation) within the environment, a lidar point cloud generated from lidar data and multiple images, wherein the lidar data and the multiple images are recorded by a device having the device pose in the environment, and an estimate of the device pose and optimizing an objective function over model parameters of a visualrepresentation model modelling an environment shown by the received images and over, for each device pose, device pose compensations of the device pose, wherein, one or more learning rates are used for the device pose compensations which depend on the sensitivity of the lidar point cloud with respect to the device pose compensations.

[0004] According to one embodiment, an adaptive learning rate (according to an adaptive learning rate strategy) which depends on the sensitivity of the lidar point cloud (projected to screen space) with respect to the device pose compensations is set per group of device pose compensations of a plurality of groups of compensations values, wherein the plurality of groups of device pose compensations comprises a group of device orientation compensations (for correction of orientation estimates of the device) and a group of device translation compensations (for correction of translation estimates of the device).

[0005] According to one embodiment, for each device pose, each image is recorded by a (respective) camera having a (respective) camera pose, wherein the method comprises receiving estimates of the camera poses and wherein the optimizing further includes optimizing over camera pose compensations of the camera poses (i.e. corrections of camera pose estimates (to correct them to the correct camera poses)). The camera pose may here be a camera pose in a world frame, i.e. an “absolute” camera pose or a camera pose in a device frame (i.e. with respect to device origin), i.e. a camera-to-device pose, i.e. an “absolute” camera pose.

[0006] According to one embodiment, one or more further learning rates are used for the camera pose compensations which depend on the sensitivity of the lidar point cloud (projected to screen space) with respect to the camera pose compensations.

[0007] According to one embodiment, a further learning rate which depends on the sensitivity of the lidar point cloud (projected to screen space) with respect to the camera pose compensations is set per group of camera pose compensations of a plurality of groups of further compensations values, wherein the plurality of groups of camera pose compensations comprises a group of camera orientation compensations (for correction of orientations estimates of the cameras) and a group of camera translation compensations (for collection of the translation estimates of the camera).

[0008] According to one embodiment, the camera poses are relative camera poses to the device pose (i.e. are camera-to-device poses).

[0009] According to one embodiment, for each device pose and each camera, an absolute camera pose (i.e. camera pose with respect to a world frame) is determined for the optimizingfrom the estimate of the device pose (which is a pose in the world frame), corrected by the device pose compensation for the device pose, and an estimate of the relative camera pose of the camera, corrected by the camera pose compensation for the relative camera pose.

[0010] According to one embodiment, the camera pose compensations are shared for all device poses (i.e. the same camera pose compensations are used for all device poses, but the camera pose compensations for different cameras may be different).

[0011] According to one embodiment, the objective function includes a reconstruction loss of the multiple images from the visual representation model.

[0012] According to one embodiment, for each device pose, the estimate of the device pose is corrected by the device pose compensation for the device pose for the optimizing by multiplying (according to a matrix multiplication) the estimate of the device pose, written as element of the special Euclidian group in three dimensions (i.e. SE(3)) from the right by the compensation for the device pose, written as element of the special Euclidian group in three dimensions.

[0013] According to one embodiment, the optimizing includes optimizing the objective function over camera intrinsics of the cameras (in addition to (at least) the model parameters and the device pose compensations, possibly the camera pose compensations as well).

[0014] According to one embodiment, the objective function depends on multiple parameters including the device pose compensations and wherein the optimizing includes for each of at least some of the parameters a penalty in the objective function for a deviation of the parameter from an initial value of the parameter (the stronger the more the parameter deviates from the initial value) wherein the penalty decreases (for at least some values of the deviation and for example does not increase for all values of the deviation) in course of the optimizing (i.e. the amount with which deviations are punished becomes lower) in later iterations than in early iterations of the optimization process).

[0015] According to one embodiment, the penalty includes an inverse bell-shape loss for the deviation of the parameter from its initial value. For the inverse bell-shape loss (i.e. U-shape loss) a function like a log barrier can be used. According to various embodiments, the penalty decreases in the course of the optimization (i.e. in later iterations of the optimization process).

[0016] According to one embodiment, the objective function includes, for each pair of images of multiple pairs of images, an epipolar loss between the images of the pair, whereineach pair includes two images with a high co-visibility (i.e. the scenes that the two images observe overlap, but with different camera position and / or view angles). This can be achieved by two images recorded at the same device pose but at different camera poses or two images recorded at different device poses but with the same camera.

[0017] According to one embodiment, the objective function includes, for each pair of images of multiple pairs of images, a reprojection loss between the images of the pair, wherein each pair includes two images with high co-visibilities, such as those recorded at the same device pose but at different camera poses or two images recorded at different device poses but with the same camera.

[0018] For determining the epipolar and / or the reprojection loss (as described in the two preceding paragraphs), the method for example includes determining a pixel-to-pixel correspondence matching, so that the reprojection and / or epipolar line error can be established. For determining the epipolar loss and / or the reprojection loss, each pair of images includes two images covering the same scene with different view angles, with high co-visibility. Images from the same camera with adjacent time frames is an example of this case.

[0019] According to one embodiment, a data processing device is provided (e.g. comprising one or more sensors, a memory and a processing unit) configured to perform the method of any one of the embodiments described above.

[0020] According to one embodiment, a computer program element is provided comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of the embodiments described above.

[0021] According to one embodiment, a computer-readable medium is provided comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of the embodiments described above.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The invention will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 shows a robot.- FIG. 2 illustrates a camera pose optimization using different methods.- FIG. 3 shows an illustration of a log-barrier method.- FIG. 4 illustrates a key-point matching between two images.- FIG. 5 shows a flow diagram illustrating a method for performing localization of a robot device.- FIG. 6 shows a data processing device according to an embodiment.DETAILED DESCRIPTION

[0023] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0024] Embodiments described in the context of one of the devices or methods are analogously valid for the other devices or methods. Similarly, embodiments described in the context of a device are analogously valid for a vehicle or a method, and vice-versa.

[0025] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0026] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0027] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0028] In the following, embodiments will be described in detail.

[0029] A visual representation of an environment can, for example, be generated from sensor data gathered by a robot. However, it should be noted that this is only an example and it can also be gathered by a human user having a backpack comprising sensors (i.e. a sensor rig).

[0030] FIG. 1 shows a robot 100.

[0031] The robot 100 is a mobile device. In the example of FIG. 1, it is a quadruped robot having four legs 101 (e.g. a robotic dog) for walking on ground 102 and has a lidar sensor 103 (or multiple lidar sensors) to observe its environment (i.e. its surroundings), in particular the ground 102 and objects 104 (typically obstacles). According to various embodiments, it is assumed that the robot 100 further comprises multiple cameras 105 which allow the robot 100 to acquire color images (e.g. RGB images) 106 of the robot’s environment.

[0032] Via the acquired sensor data, the robot 100 observes surface points of objects in the robot’ s environment. These surface points may be represented as a point cloud, i.e. a processing device 110 of the robot may generate a point cloud 107 from the sensor data delivered by the lidar sensor 103 which comprises points for 3D locations indicating surface points of the objects 104 found by the lidar sensors 103. Using the color images 106 acquired by the cameras 105, the processing device 110 may further assign colors to the points in the point cloud, i.e. the processing device 110 may fuse the lidar point cloud 107 with the color image data 106 to acquire a “colored” point cloud 108 representing its environment. The robot may do this for each of multiple poses it has in the environment (at various data acquisition times), i.e. it traverses the environment and, for each of multiple poses it has in the environment (i.e. position and orientation in the environment), it generates a colored point cloud 108 (i.e. a lidar point cloud 107 along with color camera images 106). Using a SLAM (Simultaneous Localization and Mapping) method, the robot’s processing device 110 may, based on the gathered sensor data (a lidar- point cloud 107 and color camera images 106) estimate the poses in the environment, i.e. it may associate each set of sensor data (a lidar point cloud with associated color camera images) with a pose estimate. For this, the robot 100 includes an inertial measurement unit (IMU) 109.

[0033] The sensor data gathered in the above manner (along with estimations of the poses in which the robot 100 has gathered the sensor data) allows generating a visual representation of the environment. Generating a visual representation of an environment or, in other words, visually reconstructing a scene (ideally in a photorealistic manner) has a wide range of applications. These include visualizing architectural concepts, animating movie characters, and generating AR / VR (augmented reality or virtual reality) content for the gaming and film industries. Additionally, it enhances robotic perception (in particular localization) and can for example be used for autonomous driving.

[0034] Among various approaches for generating visual representations of an environment, 3D Gaussian splatting (3DGS) is a powerful approach. According to 3DGS, a scene is modelled using a set of cloud points, each enhanced with attributes such as view-dependent color, size, scale, orientation, and opacity. In 3DGS, each cloud point becomes a transparent, viewdependent, colored Gaussian ellipsoid floating in 3D space. For representing a scene, an optimization process is carried out which aims to find the correct set of Gaussian points with appropriate position, color, scale, orientation, and transparency. Rendering of such a representation is performed through efficient rasterization, projecting (or splatting) 3D Gaussian ellipsoids onto 2D screen space. The 3DGS approach achieves comparable novelview synthesis quality to other approaches but with much faster convergence (less than 30 minutes) and real-time rendering (> 100 fps).

[0035] Due to its explicit point-cloud representation, integrating 3DGS into robotics applications is an effective approach for simultaneous localization and mapping (SLAM). In particular, the data gathered for SLAM (images, camera poses, and dense colored point clouds from LiDAR fusion) offer robust geometric priors for 3DGS.

[0036] However, the synchronization between the cameras 105 (which may be strictly synchronized to each other), the IMU 109, and the LiDAR sensor 103 is often loose, resulting in imprecise (i.e. “noisy”) pose estimates for the captured color (e.g. RGB) images and dense but noisy point clouds. Directly learning a 3D representation from such data leads to blurry reconstructions and degraded geometry.

[0037] In view of the above, according to various embodiments, a method for generating a visual representation of an environment (or, in other words, a scene) from noisy camera pose estimates and noisy LiDAR point clouds using a SLAM setup with multiple cameras is provided. Specifically, according to various embodiments, a constrained optimization algorithm is provided that simultaneously reconstructs the scene and refines both intrinsic and extrinsic camera parameters, eliminating the need for a time-consuming structure from motion (SfM) preprocessing step. According to one embodiment, the method involves decomposing a camera pose into a sequence of camera-to-(device-)center transformations (i.e. a relative poses of the cameras in the device) and (device-)center-to-world transformations (i.e. a pose of the device in the environment, wherein the device is for example a robot 100) and optimizing each parameter group (i.e. those related to the device pose and those related to the camera poses within the device, as well as, according to various embodiments, their intrinsics) withmeaningful constraints. Decomposing camera poses in this manner (in relative camera poses and a device pose) significantly reduces the number of parameters to refine, as a camera rig (comprising an arrangement of multiple cameras 105) typically captures multiple images simultaneously (and thus the cameras share a device pose). According to various embodiments, the method comprises a differentiable framework to jointly refine (i.e. optimize) the camera poses decomposed in that manner, camera intrinsics (shared by the camera poses, i.e. by the image acquisition times), and scene reconstruction parameters (i.e. the parameters of the visual representation, i.e. in particular 3DGS parameters) using a dense point cloud fused with LiDAR.

[0038] Such a method may still struggle in complex scenes due to two main issues: the inherent ambiguity in perspective projection (where intrinsic parameters and camera poses describe relative and nonlinear relationships that can have multiple feasible solutions) and the persistent overparameterization of camera poses (e.g. adjusting one camera's orientation with respect to the device center is equivalent to altering the device center for all acquisition times). To address these challenges, according to various embodiments, the optimization is based on a sensitivity per parameter group of multiple parameter groups (e.g. sensitive parameters should adjust more slowly). This is achieved by utilizing the Jacobian matrix of parameters, which provides an approximate Hessian to precondition the optimization. Additionally, according to various embodiments, with coarsely-estimated poses and estimates of camera intrinsics (of the given SLAM setup), an inverse bell-shape loss (i.e. penalty), e.g. a log-barrier, is used to ensure that critical parameters remain within predefined feasible regions (e.g. focal length should not change by more than 5%), gradually relaxing these constraints during the optimization (as the visual reconstruction of the scene matures).

[0039] Further, according to various embodiments, in view of SfM techniques where keypoint matches determine relative poses, a similar constraint is applied as strong regularization in the image space. This is achieved by minimizing both the soft epipolar error and reprojection error computed from two camera poses to optimize these key-point matches, which may be obtained from a vision foundation model. These two geometric constraints effectively guide the model towards convergence.

[0040] In the following, various embodiments are described in detail, starting with a brief explanation of 3D Gaussian Splatting (3DGS).

[0041] 3DGS represents the scene using 3D Gaussian eclipses, whose density function can be determined by the center Z E R'Jand covariance matrix E G

[0042] The means of these 3D Gaussians are initialized using an acquired point cloud, either from sparse point clouds derived from SfM or a dense point cloud from LiDAR fusion (i.e. a colored point cloud 108 as described above). To enable photorealistic rendering, each Gaussian eclipse is augmented with additional attributes: spherical harmonics (SH) representing color C where k denotes its degrees of freedom; a rotation vector r G K1defining its T r- tn>3orientation, a scale vector «> t IK including a stretch distance along different axes; and an opacity scalar O' G ]R- Scantherefore be computed from S and 7*asthe covariance matrix describes the scales and orientation of an ellipsoid configured by 5 =the rotation matrix R,where q2R(.) maps a quaternion vector to a 3 x 3 rotation matrix.

[0043] Equipped with such attributes, one can efficiently render an RGB image from the set of Gaussian pointsaview-point (world-to-camera) matrixand a camera intrinsic matrix K GProjecting a 3D eclipse into camera coordinates results in another 2D eclipse whose mean and covariance can be expressed bywhere J is the Jacobian matrix for the respective projection, and [•] [n,mJ denotes the slicing operation for a matrix.

[0044] For each pixel in screen space (i.e. the space to which the visual representation is rendered), the color and opacity of all the Gaussian eclipses are computed using equation (1) and the final rendered color is calculated as the alpha-blending of all ordered points (ordered based on depth) that overlap on this pixel, expressed as:

[0045] Benefiting from a fully-differentiable rasterization which allows a gradient backward computation with respect to various attributes, a photorealistic loss between ground truth image (acquired by a camera 105) and the rendered image can be used for the optimization so that the reconstruction can be effectuated with learnable sets of Gaussian's attributes (which from the parameters of the visual representation).

[0046] A possible approach for optimizing camera poses to reconstruct 3DGS scenes for SLAM and re-localization tasks is to process the backward path with respect to the camera poses. This allows camera poses to adjust parameters within a 6 -degree- of- freedom (6-DoF), enhancing the photometric accuracy of the rendered images. To avoid the necessity for knowledge about accurate intrinsic parameters, according to various embodiments, these parameters are optimized during the optimization process, i.e. the optimization process includes refinement of (initial guesses of) camera intrinsics.

[0047] Camera intrinsics include the two focal lengths fxand fy, as well as the location of the principal point ex and cy. These parameters control how 3D geometry relates to 2D screen space (wv space), while an inaccurate intrinsic can have an important impact, especially on the rendered details. The backward of the rasterization can be expressed as:

[0048] According to the chain rule, the first terms of each of the above two equations are the partial derivatives from the loss to uv variables representing the screen coordinates of Gaussian eclipses, and they are computed beforehand via the differentiable rasterizer. The second terms are derivatives of uv with respect to intrinsic parameters and have analytical solutions expressed as:_ where Z / Cam is the Gaussian mean in camera space, withdenoting its x, y and z components.

[0049] According to various embodiment, the objective function of the optimization process includes, as a primary loss (also referred to as reconstruction loss), an LI norm pixel-wise loss between ground truth and rendered images, and a structure- similarity loss:is the pixel- wise photometric loss, whilestructural-similarity loss.

[0050] However, naively optimizing an objective function including only this loss to jointly refine camera poses, intrinsics and noisy -initialized Gaussian eclipses may lead to suboptimals. Approaches to address this are described in the following.

[0051] Given a set of coarsely estimated camera pose {풫_i}|^N_{i=1} ∈ SE(3) with their respective RGB images j / j ( Ewhere H and W represent the height and width of the captured images, while i denotes the image / pose index (1 < i < N) among the total N images. These poses provided are inaccurate due to two complementary facts. First, in a conventional setting in the acquisition of SLAM, the orientation and position of the device (e.g. robot 100) i is typically calculated from complex derivation with the embedded inertial measurement unit (IMU) 109, which, itself, may result in a degenerate solution and noisy estimation. In addition, RGB images are captured in a coarsely synchronous manner with device pose acquisition, where the image pose p, is determined by finding the closest device pose and cascading it with the extrinsics of camera to device. This design ignores the unavoidable time-frame offset (often up to 50 ms) between them, which further amplifies the discrepancy between the estimated camera poses and true ones. Therefore, according to various embodiments, the method provided aims to jointly compensate for the noisy set of camera poses, while accurately reconstructing the 3D scene.

[0052] As described with reference to FIG. 1, several (e.g.) RGB cameras are on board the device (e.g. robot 100) and are synchronized to capture images at the same time. This is often and particularly useful for capturing a scene with wide coverage and to reduce the inspection time. However, this makes direct camera pose optimization intractable due to large number ofparameters to refine. To address this, according to various embodiments, each camera pose is modelled in a decomposed manner: factorizing the image camera pose to a cascade of camera-to-device and device-to-world poses, which can be expressed as 7^where is the camera-to- world pose captured by j* camera and by Ithtime frame, P̂^{t_1} denotes the device-to-world pose at acquisition time t, whilerepresents the jthcamera-to-device extrinsic. This design breaks the independence assumption of each image and allows one to reduce the parameter amount significantly: given 10,000 images taken by a 4-camera rig device, independently refining these extrinsics requires a total degree of freedom of 6x 10000 = 60000, while using the above decomposition only the modeling 4 shared extrinsics (1 < j < 4) and 2500 independent LiDAR poses 'Pl(1 < t < 2500) is required, with a total degree of freedom of 6x 2500+6x 4 = 15024. In addition, parameters shared across cameras and time frames lower the optimization challenge and stabilize the optimization when jointly refining the camera poses and the accurate 3D scene reconstruction.

[0053] As mentioned above, the SLAM provides inaccurate camera pose estimates. These pose estimates are refined in the course of the optimization process by determining offsets for them (for each acquisition time; these, when applied to the estimates, ideally result in the correct poses for each acquisition time). In other words, offsets are determined to compensate the possible errors that P^' andcarry, as follows:where 0^, f Gare learnable tensors with each consisting of a small rotation rot? rot® and translation, trans € R'* to compensate the LiDAR device pose at time frame t and j* camera-to-device error, respectively. / (•) and g(») are two functions defining how exactly the noisy poses are refined by these small deltas. The choice of compensation functions can be non-trivial for effective pose refinement. One natural choice of the compensation is to left-multiply the error matrix with the original pose.However, this leads to extremely unstable optimization, which can be demonstrated by showing the trajectory of the optimized camera location. Assume that f (•) is defined as:= [ΔR|Δt] ∈ SE(3) is the rigid-body transformation representation of φ⃗t. ΔR and Δt, defined as the upper-left part and third column of Φt, are the 3 x 3 rotation offset matrix and translation offset vectors. This forces the camera location to rotate with respect to the origin defined in the world origin, which is often far away from initial camera value), as suggested by the derivation:In equation (6),and ^originarctherefined and original camera locations in world coordinates. It can be observed that such a compensation results in unstable optimization behaviors.

[0054] Instead,which right-multiplies the error matrix with the original pose, provides better properties, as the new device center is defined as= R_d2w Δt + t_d2w. The difference between the two schemes of equation (6) and equation (7) is illustrated in FIG. 2, wherein the latter is used according to various embodiments.

[0055] FIG. 2 illustrates a camera pose optimization using different methods. The left diagram 201 illustrates an optimization of camera poses by rotation around the world origin according to equation (6). The right diagram 202 illustrates an optimization of camera poses based on the initial camera origin. It can be seen that the approach of the second diagram 202 has better optimization property and robustness

[0056] As mentioned above, the objective function of the optimization includes, as a primary loss, the photometric loss ℒ_pixel and the loss of structurally-based similarity sim. The former is a pixelwise 1 -loss and focuses on photorealistic fidelity, while the latter emphasizes perceptual quality. They are given as follows

[0057] However, directly optimizing an objective function including only this primary loss (i.e. only a loss according to equation (3)) may easily get stuck in a suboptimal point. This can be attributed to several major factors. First, the inherent ambiguity of the projection matrix (intrinsics and camera poses only describe relative relationships and can have multiple feasible solutions). Second, even when a camera pose decomposition strategy is utilized as described above, the resulting camera representation is still over-parameterized. For example, modifying the orientation of one camera-to-device pose is equivalent to adjusting the orientation of all LiDAR centers.

[0058] The Levenberg- Marquardt (LM) algorithm is a popular method to be employed to solve a general nonlinear optimization problem and has been used in the relevant task of camera calibration. This algorithm, which is used for the optimization according to one embodiment, solves a complex optimization by preconditioning the parameters with high-order curvature information, which is estimated by resorting to gaussian newton method. In contrast, default optimization optimizers often used in the deep learning frameworks estimate this information indirectly via computing running variance of gradients over time.

[0059] According to various embodiments, the optimization is constrained based on both the sensitivity of each parameter (e.g. sensitive parameter should move slower), and the given coarsely-estimated poses and intrinsics. In other words, according to various embodiments, a sensitivity-based pre-conditioning is performed as described in the following.

[0060] In the optimization, several sets of camera intrinsic and extrinsic parameters are learned simultaneously and each set of parameters has different impact on the rendering quality. For instance, as described above, the optimization searches for small deltas φ⃗rot and φ⃗trans for compensation of errors with regard to the estimation of the orientation and translation of the camera poses. A tiny refinement of these parameters (1%) can lead to different behaviors: orientation is more sensible and provides large pixel changes in the rendered image, compared to incrementing translation with the same amount. Therefore, for a more robust learning algorithm the camera orientation learning rate is adequately lowered due to its high sensitivity (of the objective function value with respect to it). More concretely, given the dense point cloud 풢 from the SLAM, the sensitivity of φ⃗rot, φ⃗trans, ρ⃗rot, andPtrans is defined and quantified by computing their respective sensitivities with respect to the UV coordinate of the rendered point cloud.where Pro j is the projection function given the current camera-to-world matrixc2w-When initializing φ⃗rot, φ⃗trans, ρ⃗rot, and ρ⃗trans as zero tensors, the sensitivity matrix can be efficiently computed by solving the Jacobean matrix of Proj(·, ·, ·, ·|풢, 풫_c2w):

[0061] This Jacobean matrix represents how small changes in each input element contributes to the output and can be efficiently computed from a deep learning framework, such as TensorFlow and PyTorch. The overall Jacobian matrix given a multiple view matrix is the average of the individual J averaged on different camera poses and average on each element of the same group. According to various embodiments, the learning rate of each parameter group (i.e. the groups d’rot, Straus, Pmi, and rans) is weighted based on the diagonal value of the matrix (풥⊤풥)^{-1 / 2}, which is the inverse square root of the Hessian matrix. This approach adaptively adjusts the learning rate of each parameter set based on the inverse of their respective sensitivities.

[0062] As mentioned above, according to various embodiments, at least some of the parameters (over the objective function is optimized) are constrained to a feasible region, i.e. at least some of the parameters are confined within an estimated viable region, e.g. using a Log-barrier method. This can be efficiently applied to parameters for which SLAM provides an approximate initial value. Constraints are defined ensuring that the refined value for such a parameter remains proximate to its initial estimation and does not deviate significantly during the optimization process. Through empirical observations, an error margin of each parameter can be determined and a log-barrier constraint can be constructed to ensure that crucial parameters adhere strictly to their designated feasible boundaries. For instance, the focal length should not vary by more than 3%.

[0063] Let parameter x be a parameter which should be constrained within a viable region.According to various embodiments, m inequality constraints< 0 (1 < i < tn) are defined for the parameter x. For example, for a U-shape like constraint of a parameter, as for example implemented by a log -barrier, a left constraint and a right constraint are used. The logbarrier method expresses the constraints under the form of negative log form, aswhere T is a temperature term which increases from a small value to a very large one. This establishes a gradually varying penalty barrier, as depicted in FIG. 3.

[0064] FIG. 3 shows an illustration of a log-barrier method. Lower and upper bounds are defined upfront based on SLAM initial estimation. The log-barrier method imposes a strong penalty if the current value goes far away from the initial one at the start of the transformation (first curve 301), while it becomes more relaxed within the bounds in course of the optimization (second curve 302) until the end of optimization (third curve 303). The penalty corresponds to a regularization term with gradient

[0065] As illustrated in FIG. 3, it establishes a symmetric penalty function centered at the initial value, ensuring that the learnable parameter remains in close-proximity to the initial value. Furthermore, the penalty gradient escalates significantly as the learned parameter nears the predefined boundaries (the gradient term —becomes large), thereby preventing it from venturing into infeasible regions. The temperature T is increased as the optimization progresses and the penalty thus diminishes, allowing the parameter to stabilize between the two boundaries. Thus, two bounds can be empirically set and the optimization can be guided towards a plausible solution. Such constraints may be set for any parameter. For example, it is only applied to the intrinsics and the pose optimizations. However, it may also be applied to one or more of the visual representation model parameters, i.e. one or more of the 3DGS parameters like color, ellipsoid position, ellipsoid scale, ellipsoid orientation etc.

[0066] Besides incorporating constraints from the optimization perspective, according to one embodiment, the optimization is done by enforcing geometry-related constraints. The traditional SfM algorithm relies on key- point matches between multiple co-visible images. In Colmap, this is implemented by using a scale -invariant feature transform (SIFT) and time-consuming matching. According to various embodiments, a deep learning model is used which directly outputs semi-dense (up to several hundreds) key point matches {. X j, j -n } for adjacent image frames i and i + n, with X.j,€ R^xM number of matches for the image pair, as well as n being a small integral 1 < n < 3 to ensure a large co-visibility between images.

[0067] FIG. 4 illustrates a key-point matching between two images.

[0068] In the (simple) example of figure 4, there are two images 401, 402. The images 401, 402 may have been recorded in the same time frame (i.e. at the same acquisition time) by different cameras or at subsequent time frames. Accordingly, they have been recorded from two different (absolute) camera poses. Accordingly, objects 403, 404 are shown from different view points in the two images 401, 402. A key point matching, as illustrated by the dashed lines, provides an association of key points of the objects 403, 404 between where there are shown in the first image 401 and where they are shown in the second images 402.

[0069] The matched key points may be utilized by two different yet complementary strategies which address different aspects of the scene geometry and are described in the following.

[0070] As a first geometry constraint, a soft epipolar constraint may be used. This constraint tries to enforce the learned the relative camera poses to respect the epipolar geometry constraint, where the matched pixels should lie in the epipolar plane. This constraint is implemented by estimating the Sampson distance (the distance of each pixel to the estimated epipolar line) from these matched pixel pairs, and the estimated fundamental matrix F,from the relative poses of the adjacent camera poses. This can be expressed as the following loss which is, according to one embodiment, included in the objective function:where Frelis the estimated fundamental matrix computed from relative camera poses Pj j and respective intrinsics Kj and Kj, which is computed from:

[0071] By imposing this constraint, both the camera intrinsics and the poses are learned by minimizing the Sampson distances. However, as the epipolar constraint does not take the depth information of the reconstruction into account and intrinsically has projective ambiguities, the following second geometry constraint addressing this issue may be used.

[0072] As a second geometry constraint, a re-projection Error Regularization is applied. In Structure from Motion (SfM) software, the regularization of reprojection error is often referred to as “Bundle Adjustment”. This technique aims at back-projecting and triangulating a single pixel from multiple images, with the reprojection error being the discrepancy between the re-projected pixels for the 3D point across different images. The reprojection error can be expressed as the following loss which is, according to one embodiment, included into the objective function:matched points tendered depths camera poses intrinsicswhere d⃗i and d⃗_{i+n} ∈ ℝ^{M×1} are the rendered depths for the matched points in the i^th and the i+n^th image. Minimizing the loss of equation (15) leads to minimizing the errors by considering the rendered distances, thus constraining the geometry of the scene complementary to the epipolar error loss of (13).

[0073] In summary, according to various embodiments, a method is provided as illustrated in FIG. 5.

[0074] FIG. 5 shows a flow diagram illustrating a method for generating a visual representation of an environment.

[0075] In 501, for each device pose of a plurality of device poses within the environment, a lidar point cloud generated from lidar data and multiple images (color, typically RGB) isreceived, wherein the lidar data and the multiple images are recorded by a device (comprising a lidar sensor and multiple cameras) having the device pose in the environment, and an estimate of the device pose (that is typically not perfectly correct, i.e. “noisy”)

[0076] In 502, an objective function is optimized over model parameters of a visual representation model (e.g. parameters of a Gaussian splatting) modelling an environment shown by the received images (i.e. allowing visual reconstruction of the environment as recorded by the images) and over, for each device pose, device pose compensations of the device pose (i.e. compensations of the noise in the device pose estimate, i.e. corrections of the device pose estimate (to correct it to the correct device pose)), wherein (during the optimizing, i.e. the optimization process), one or more learning rates are used for the device pose compensations which depend on the sensitivity of the lidar point cloud with respect to the device pose compensations.

[0077] The visual representation model given by the model parameters resulting from the optimization are then used as visual representation of the environment. It should be noted that while in the above examples, the sensor data is gathered by a robot, a rig of sensors may also be carried around by a human user. Nevertheless, a SLAM method may then be used.

[0078] According to various embodiments, in other words, parameters of a visual representation model are found by optimizing an objective function over the parameters jointly with corrections for the device pose estimates wherein the learning rates of (at least some of) the device pose compensations are set depending on the sensitivity of the lidar point cloud with respect to the device pose compensations.

[0079] According to various embodiments, for each of a plurality of sensor data acquisition times,- a lidar point cloud generated from lidar data recorded by a lidar sensor at the sensor data acquisition time having a device pose at the sensor data acquisition time - multiple images (color, typically RGB) recorded by multiple cameras at the sensor data acquisition time, each having a respective camera pose at the sensor data acquisition time and- an estimate of the device pose and estimates of the camera poses.are received and an objective function is optimized (at least) over parameters of a visual representation model and compensations of the device poses and camera poses.

[0080] The method of FIG. 5 is for example carried out by a data processing device as illustrated in FIG. 6.

[0081] FIG. 6 shows a data processing device 600 according to an embodiment.

[0082] The data processing device 600 includes a sensor 601 (attached to a robot, e.g. configured to sensor data such as lidar data and images). The data processing device 600 further includes a processing unit 602 and a memory 603. The memory 603 may be used to store, for example, the sensor data. The robot control system 600 is configured to perform the method of FIG. 5. The robot is generally a robot device which may include any mobile device (in particular mobile devices whose movement may be actively controlled) like vehicles etc.

[0083] The methods described herein may be performed and the various processing or computation units and the devices and computing entities described herein may be implemented by one or more circuits. In an embodiment, a "circuit" may be understood as any kind of a logic implementing entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor. A "circuit" may also be software being implemented or executed by a processor, e.g. any kind of computer program, e.g. a computer program using a virtual machine code. Any other kind of implementation of the respective functions which are described herein may also be understood as a "circuit" in accordance with an alternative embodiment.

[0084] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A method for generating a visual representation of an environment, the method comprising:receiving, for each device pose of a plurality of device poses within the environment, a lidar point cloud generated from lidar data and multiple images, wherein the lidar data and the multiple images are recorded by a device having the device pose in the environment, and an estimate of the device pose;optimizing an objective function over model parameters of a visual representation model modelling an environment shown by the received images and over, for each device pose, device pose compensations of the device pose, wherein, one or more learning rates are used for the device pose compensations which depend on the sensitivity of the lidar point cloud with respect to the device pose compensations.

2. The method of claim 1, wherein an adaptive learning rate which depends on the sensitivity of the lidar point cloud with respect to the device pose compensations is set per group of device pose compensations of a plurality of groups of compensations values, wherein the plurality of groups of device pose compensations comprises a group of device orientation compensations (for correction of orientation estimates of the device) and a group of device translation compensations (for correction of translation estimates of the device).

3. The method of claim 1 or 2, wherein, for each device pose, each image is recorded by a (respective) camera having a (respective) camera pose, wherein the method comprises receiving estimates of the camera poses and wherein the optimizing further includes optimizing over camera pose compensations of the camera poses (i.e. corrections of camera pose estimates (to correct them to the correct camera poses)).

4. The method of claim 3, wherein one or more further learning rates are used for the camera pose compensations which depend on the sensitivity of the lidar point cloud with respect to the camera pose compensations.

5. The method of claim 4, wherein a further learning rate which depends on the sensitivity of the lidar point cloud with respect to the camera pose compensations is set per group of camera pose compensations of a plurality of groups of further compensations values, wherein the plurality of groups of camera pose compensations comprises a group of camera orientation compensations (for correction of orientations estimates of the cameras) and a group of camera translation compensations (for correction of the translation estimates of the camera).

6. The method of claim 3 to 5, wherein the camera poses are relative camera poses to the device pose.

7. The method of claim 6, wherein for the optimizing, for each device pose and each camera, an absolute camera pose (i.e. camera pose with respect to a world frame) is determined from the estimate of the device pose (which is a pose in the world frame), collected by the device pose compensation for the device pose, and an estimate of the relative camera pose of the camera, corrected by the camera pose compensation for the relative camera pose.

8. The method of any one of claims 3 to 7, wherein the camera pose compensations are shared for all device poses (i.e. the same camera pose compensations are used for all device poses, but the camera pose compensations for different cameras may be different).

9. The method of any one of claims 1 to 8, wherein the objective function includes a reconstruction loss of the multiple images from the visual representation model.

10. The method of any one of claims 1 to 9, wherein for the optimizing, for each device pose, the estimate of the device pose is corrected by the device pose compensation for the device pose by multiplying (according to a matrix multiplication) the estimate of the device pose, written as element of the special Euclidian group in three dimensions (i.e.from the right by the compensation for the device pose, written as element of the special Euclidian group in three dimensions.

11. The method of any one of claims 1 to 10, wherein the optimizing includes optimizing the objective function over camera intrinsics of the cameras (in addition to (at least) the model parameters and the device pose compensations).

12. The method of any one of claims 1 to 11, wherein the objective function depends on multiple parameters including the device pose compensations and wherein the optimizing includes for each of at least some of the parameters a penalty in the objective function for a deviation of the parameter from an initial value of the parameter (the stronger the more the parameter deviates from the initial value) wherein the penalty decreases (for at least some values of the deviation and for example does not increase for all values of the deviation) in course of the optimizing (i.e. the amount with which deviations are punished becomes lower) in later iterations than in early iterations of the optimization process).

13. The method of claim 12, wherein the penalty includes an inverse bell-shape loss for the deviation of the parameter from its initial value.

14. The method of any one of claims 1 to 13, wherein the objective function includes, for each pair of images of multiple pairs of images, an epipolar loss between the images of the pair, wherein each pair includes two images recorded at the same device pose but at different camera poses or two images recorded at difference device poses but with the same camera.

15. The method of any one of claims 1 to 14, wherein the objective function includes, for each pair of images of multiple pairs of images, a reprojection loss between the images of the pair, wherein each pair includes two images recorded at the same device pose but at different camera poses or two images recorded at difference device poses but with the same camera.

16. A data processing device configured to perform a method of any one of claims 1 to 15.

17. A computer program element comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15.

18. A computer-readable medium comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15.

Images