Method and system for enabling resilient lane-level navigation in harsh weather conditions

Abstract

A method for enabling a resilient lane-level navigation in harsh weather conditions is provided. The method includes collecting a dataset associated with real-world driving conditions during harsh weather conditions using artificial intelligence (AI). The dataset includes at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata. The method also includes setting a generator network and a discriminator network to create a plurality of images resembling real driving scenes under the harsh weather conditions. The method also includes generating synthetic data by employing a pre-trained AI model by varying one or more weather conditions fed into the AI model. The method also includes incorporating the generated synthetic data into a training dataset for training perception models used in autonomous driving systems. The method also includes optimizing a navigation solution using trained perception models, weather information, and sensor data for enabling the resilient lane-level navigation.

Description

METHOD AND SYSTEM FOR ENABLING A RESILIENT LANE-LEVEL NAVIGATION IN HARSH WEATHER CONDITIONS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the provisional patent application titled “METHOD AND SYSTEM FOR ENABLING A RESILIENT LANE-LEVEL NAVIGATION IN HARSH WEATHER CONDITIONS”, with application number 63 / 722,164, filed in the United States Patent and Trademark Office on November 19, 2024. The specification of the above referenced patent application is incorporated herein by reference in its entirety.BACKGROUNDTechnical Field

[0002] The present invention generally relates to the field of navigation solutions. The present invention more particularly relates to a method and system for enabling a resilient lanelevel navigation in harsh weather conditions.Description of the Related Art

[0003] Typically, the fusion of Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) technologies is a critical aspect of modern navigation solutions, particularly in contexts requiring high reliability and accuracy, such as autonomous vehicles, unmanned aerial vehicles (UAVs), and mobile mapping systems. In this fused system, GNSS provides positioning and timing information globally based on satellites. It offers broad coverage but can vary in accuracy and be susceptible to signal degradation from atmospheric conditions, multipath reflections, and signal blockage in dense urban areas or natural obstructions. INS consists of accelerometers and gyroscopes that measure linear acceleration and angular rates, respectively. This system can calculate position, orientation, and velocity independently of external references by integrating sensor outputs over time. INS is highly reliable in the short term but suffers from drift errors that accumulate rapidly, making long-term accuracy a challenge. Besides the GNSS, Real-Time Kinematic (RTK) positioning combined with Precise Point Positioning (PPP) GNSS (Global Navigation Satellite System) and INS (Inertial Navigation System) fusion is becoming an essential technology. Integrating these technologies involves handling complex data and ensuring the synchronization of different sensors. This is critical for real-time applications like autonomous driving, where delays or errors in data processing can lead to accidents.

[0004] Generally, perception-based technologies combined with GNSS and INS offers arobust navigation solution, particularly useful in autonomous vehicles, robotics, and various mapping applications. Here, perception-based technologies can be considered as any type of data collection sensors, including light detection and ranging (LiDAR), Radar, and visual-based sensors. The main idea of perception-based simultaneous localization and mapping (SLAM) is to estimate the relative odometry using geometric algorithms (e.g., visual feature extraction and tracking algorithms such as SIFT and SURF, point clouds registration algorithms such as internet cache protocol and non-destructive testing (ICP and NDT) algorithms) and fusing with motion sensors (e.g., odometer, GNSS / INS solutions, pure INS solution).

[0005] However, the limitation of SLAM-aided GNSS / INS navigation solution are discussed. Firstly, the performance of SLAM is limited when the received image, radar, or LiDAR with unexpected noisy. Secondly, the strategy of sensor fusion is not adaptive to all the scenarios, particularly in adverse weather conditions. Further, the dynamic objects, such as moving vehicles and trucks bring errors in the SLAM system since the data association using moving objects is comprised of non-rigid transformation information. Also, the GNSS signals are blocked or with multi-paths noisy wherever there are tall buildings and unexpected physical obstruction. Therefore, the performance of GNSS signal is affected due to the blockage of multipaths, signal obstruction and loss. These errors will influence the safety and limit the applications of autonomous driving and advanced driver-assistance systems (ADAS). Also, low performance in harsh weather of SLAM solutions is another issue. Here, like LiDAR, Radar, and camera, the performance perception-based sensors are affected. The accuracy of the data association will be affected due to poor visibility conditions. Harsh weather will affect the navigation performance, for instance the accuracy of the GNSS signal will be affected due to signal degradation, multipath effects, and ionospheric disturbance. The RTK performance will be affected in the harsh weather due to the conditionally unstable calibration parameters. In addition, weather conditions can introduce physical noise and errors in sensor readings. For example, water droplets or snowflakes can cause false detections or obscure important features in LiDAR point clouds, Radar readings, or camera images, leading to inaccuracies in the data used for navigation.

[0006] The above-mentioned shortcomings, disadvantages, and problems are addressed herein, and will be understood by reading and studying the following specification.OBJECT OF THE TECHNOLOGY

[0007] A primary object of the embodiments of the present technology is to provide a method for enabling a resilient lane-level navigation in harsh weather conditions.

[0008] Yet another object of the embodiments of the present technology provides a multi-phase sensor fusion strategy using weather information.

[0009] Yet another object of the embodiments of the present technology provides accurate positioning solutions for advanced driver-assistance systems (ADAS), other smart vehicle applications, or mobile mapping-related applications.

[0010] Yet another object of the embodiments of the present technology enhances perception data by enhancing a resolution for improving clarity in low-visibility conditions, accentuating road signs, lane markings, and obstacles.

[0011] These and other objectives and advantages of the embodiments of the present technology will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings.SUMMARY

[0012] This summary is provided to introduce a selection of concepts in a simplified form that are further disclosed in the detailed description. This summary is not intended to determine the scope of the claimed subject matter.

[0013] The embodiments herein address the above-recited needs for a system and a method for enabling resilient lane-level navigation in harsh weather conditions.

[0014] In one aspect a method for enabling resilient lane-level navigation in harsh weather conditions is provided. The method includes collecting a dataset associated with one or more real-world driving conditions across harsh weather conditions using artificial intelligence (Al). The dataset includes at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata. The method also includes setting at least one of a generator network and a discriminator network, to generate a plurality of synthetic images resembling real driving scenes under the harsh weather conditions. The method also includes generating synthetic data by employing a pre-trained Al model by varying one or more weather condition parameters input into the Al model. The method also includes incorporating the generated synthetic data into a training dataset for perception models used in autonomous driving systems for training the perception models. The method also includes optimizing a navigation solution using the trained perception models, weather information, and sensor data for enabling the resilient lane-level navigation.

[0015] According to an embodiment, the method further includes uploading the trained perception models in to at least one of a cloud server or in a roadside unit.

[0016] According to an embodiment, a ground-truth weather condition is labeled for a portion of the dataset and the labeled portion of dataset is standardized to ensure consistency, by performing at least one of adjusting a scale of sensor readings or normalizing image pixel values.

[0017] According to an embodiment, the weather information is received from at least one of a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

[0018] According to an embodiment, the generator network generates synthetic images simulating driving scenes under a plurality of weather conditions and the discriminator network distinguishes between the real images and generated images.

[0019] According to an embodiment, a method for enabling a resilient lane-level navigation further includes enhancing perception data by enhancing image resolution for improving visibility in low-visibility conditions, accentuating road features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions, by reducing noise in images caused by low light levels or precipitation and by incorporating the enhanced perception data into a factor graph optimization to improve the positioning solution.According to an embodiment, a method for enabling a resilient lane-level navigation further includes enhancing sensor fusion strategy using a deep-learning-based algorithm, by synchronizing multimodal sensor data including images, Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, and visual odometry using timestamp alignment, by timestamping all data inputs to align accurately, and normalizing and preprocessing the data to ensure consistency, by capturing one or more temporal dependencies in navigation data, where recurrent neural network (RNNs) and Long-Short Term Memory (LSTMs) are selected as a demonstration example as the deep learning network for integrating time-series data from GNSS / INS and odometry, by defining a loss function that accurately reflects the performance of the fusion model in predicting vehicle position and orientation and by allowing dynamic adjustment based on real-time data inputs.

[0020] According to another embodiment, a system for enabling resilient lane-level navigation in harsh weather conditions is provided. The system includes a processor to fetch and execute computer-readable instructions stored in the memory of the system. The system also includes a memory to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, the one or more computer-readable instructions includes one or more executable modules including, a collection module for collecting a dataset associated with one or more real-world driving conditions across the harsh weather conditionsusing artificial intelligence (Al), where the dataset includes at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata, a network module for setting at least one of a generator network and a discriminator network, to generate a plurality of synthetic images resembling real driving scenes under the harsh weather conditions, a data generation module for generating synthetic data by employing a pre-trained Al model by varying one or more weather condition parameters input into the Al model, a training module for incorporating the generated synthetic data into a training dataset for perception models used in autonomous driving systems for training the perception models and a navigation module for optimizing a navigation solution using the trained perception models, weather information, and sensor data for enabling the resilient lane-level navigation.

[0021] According to an embodiment, the navigation module uploads the trained perception models in to at least one of a cloud server or in a roadside unit.

[0022] According to an embodiment, the collection module is further configured to label ground-truth weather condition for a portion of the dataset and the labeled portion of dataset is standardized to ensure consistency, by performing at least one of adjusting a scale of sensor readings or normalizing image pixel values.

[0023] According to an embodiment, the navigation module receives weather information from at least one of a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

[0024] According to an embodiment, the network module comprises a generator network generates synthetic images simulating driving scenes under a plurality of weather conditions and the discriminator network distinguishes between the real images and generated images.

[0025] According to an embodiment, the navigation module is further configured for enhancing perception data by enhancing image resolution for improving visibility in low-visibility conditions, accentuating road features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions, by reducing noise in images caused by low light levels or precipitation and by incorporating the enhanced perception data into a factor graph optimization to improve the positioning solution.According to an embodiment, the navigation module is further configured for enhancing sensor fusion strategy using deep-learning-based algorithm, by synchronizing multimodal sensor data including images, Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, and visual odometry using timestamp alignment, by timestamping all data inputs to align accurately, and normalizingand pre-processing the data to ensure consistency, by capturing one or more temporal dependencies in navigation data, where recurrent neural network (RNNs) and Long-Short Term Memory (LSTMs) are selected as a demonstration example as the deep learning network for integrating time-series data from Global Navigation Satellite System (GNSS) / Inertial Navigation System (INS) and odometry, by defining a loss function that accurately reflects the performance of the fusion model in predicting vehicle position and orientation and by allowing dynamic adjustment based on real-time data inputs.

[0026] The purpose of the present technology is to provide highly resilient lane-level navigation solutions in all scenarios and harsh weathers, especially the urban canyons (e.g., areas with tall buildings), heavy rains, snowy and foggy weather conditions and also to improve the resiliency of the navigation solution by providing accurate positioning solutions for advanced driver-assistance systems (ADAS) and other smart vehicle applications. Additionally, weather information to enhance the navigation solution is involved in the present technology. The weather information provided by the present technology is not only for sensor fusion optimization but also for perception data enhancement. In an embedded module of the Navigation App (Google Map), the navigation app (or software) can create a weather button for this technology. When this function is needed, the present system receives weather information, enhances the image (or, point clouds), and provides a lane-level navigation solution. In present technology, internet of things (IoT) technology is employed for autonomous driving and navigation solutions. Also, a multi-phase sensor fusion strategy is presented in this technology. The weather information is involved in two stages. Feature extraction and data enhancement and navigation solution optimization, including the navigation mode selection and fusion strategy optimization. Also, a multi-phase sensor fusion strategy using weather information is presented in this technology. The present technology leverages a GAN network for the generation of datasets that serve two purposes, firstly quality under adverse weather conditions, such as fog or snow and secondly integrating weather factors into the positioning and navigation solution through joint optimization.

[0027] It is to be understood that the aspects and embodiments of the disclosure described above may be used in any combination with each other. Several of the aspects and embodiments may be combined to form a further embodiment of the disclosure.

[0028] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0029] These and other objects and advantages will become more apparent when reference is made to the following description and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The other objects, features and advantages will occur to those skilled in the art from the following description of the preferred embodiment and the accompanying drawings in which:

[0031] FIG. 1A depicts a block diagram of an example environment suitable for practicing the systems and methods described herein.

[0032] FIG. 1B illustrates an exemplary block diagram of the system for enabling a resilient lane-level navigation in harsh weather conditions, in accordance with an embodiment of the present technology.

[0033] FIG. 2 illustrates a flow diagram depicting a method for enabling a resilient lanelevel navigation in harsh weather conditions, in accordance with an embodiment.

[0034] FIG. 3 illustrates an exemplary computer system in which or with which embodiments of the present disclosure may be implemented.

[0035] Although the specific features of the embodiments herein are shown in some drawings and not in others. This is done for convenience only as each feature may be combined with any or all of the other features in accordance with the embodiments herein.DETAILED DESCRIPTION OF THE DRAWINGS

[0036] The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the amount of details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0037] It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.

[0038] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0039] The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0040] It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.

[0041] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood however, it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0042] The various embodiments of the present technology provide an efficient technique for enabling resilient lane-level navigation in harsh weather conditions. The present technology enhances the simultaneous localization and mapping (SLAM) algorithms. The foggy, rainy, and snowy weather affects the precept data, resulting in unexpected errors. In this case, the present technology aims to enhance the visual images and point clouds by using the simulated data with different kinds of weather scenarios. In the present technology, the simulated data is generated using generative artificial intelligence (Al) technologies termed as data enhancement. The strategy of sensor fusion is adaptive to different weather conditions. The internet of things (IoT) technology is involved and transferred to the image enhancement technology in real-time. For example, smart vehicles receive the foggy weather, and will mainly reply on the solution of Global Navigation Satellite System / Inertial Navigation System (GNSS / INS) and enhances the light detection and ranging (LiDAR) and visual images using the pre-trained data enhancementmodels.

[0043] FIG. 1A depicts a block diagram of an example environment 100 suitable for practicing the systems and methods described herein. It should be noted, however, that the environment 100 is just an example and is simplified embodiment provided for illustrative purposes and reasonable deviations of this embodiment are possible as will be evident to those skilled in the art. A global navigation satellite system (GNSS) 102 is a network of satellites broadcasting timing and orbital information used for navigation and positioning measurements. These satellites broadcast signals 103 that identify which satellite is transmitting and its time, orbit and status or health. There are four main constellations in orbit global positioning system (GPS), GLONASS, Galileo and BeiDou as well as two regional systems QZSS and IRNSS and each are managed by a different country. The control segment 105 is a network of master control, data uploading and monitoring stations located around the world. These stations receive a satellite’s signal and compare where the satellite says it is with orbit models showing where it should be. Operators at these stations can control the satellites position to correct or alter their orbital paths, for example if a satellite has drifted, or needs to be moved to avoid debris collision. This process, as well as monitoring a satellite’s health, ensures a baseline of accuracy in GNSS positioning. The user segment 106 includes the equipment that receives satellite signals and outputs a position based on the time and orbital location of at least four satellites. This segment includes the user’s antennas to identify and receive good-quality signals as well as high-precision receivers and positioning engines that process the signals and resolve potential timing errors. The space segment 108 describes the GNSS constellations orbiting between 20,000 to 37,000 kilometers above the earth. These satellites broadcast signals that identify which satellite is transmitting and its time, orbit and status or health.

[0044] FIG. 1B illustrates an exemplary block diagram of the system 110 for enabling a resilient lane-level navigation in harsh weather conditions, in accordance with an embodiment of the present technology. The system 110 includes a processor 112, a memory 114, an interface 116, a processing engine 118, an AI / ML engine 120, a collection module 122, a network module 124, a data generation module 126, a training module 128, a navigation module 130 and an IOT / cloud service132. The processors 112 and the memory 114 may be communicably coupled to the one or more other processors. The one or more processor(s) may be implemented as one or more microprocessors, microcomputers, microcontrollers, edge or fog microcontrollers, digital signal processors, central processing units, logic circuitries, and / or any devices that process data based on operational instructions. Among other capabilities, one or more processor(s) may be configured to fetch and execute computer-readable instructions stored in the memory 114 of thesystem 110. The processor 112 is configured to fetch and execute computer-readable instructions stored in the memory 114. The memory 114 may be configured to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, which may be fetched and executed to create or share data packets over a network service. The memory 114 may include any non-transitory storage device including, for example, volatile memory such as Random-Access Memory (RAM), or non-volatile memory such as Erasable Programmable Read-Only Memory (EPROM), flash memory, and the like. The memory 114 may be configured to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, fetched and executed to create or share data packets over a network service. The processing engine 118 is configured to collect a broad dataset of a real-world driving conditions across harsh weather conditions using artificial intelligence (Al) and the AI / ML engine 120. The collection module is further configured to label ground-truth weather condition for a portion of the dataset, and the labeled portion of dataset is standardized to ensure consistency, by performing at least one of: adjusting a scale of sensor readings or normalizing image pixel values. The navigation module receives weather information from at least one of: a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution. The AI / ML engine 120 updates the pre-trained model in at least one of a cloud server or in a roadside unit. The AI / ML engine 120 receives a weather information from at least one of a roadside unit, a weather website, or one or more data publishers. The collection module 122 collects a dataset associated with one or more real-world driving conditions across the harsh weather conditions using artificial intelligence (Al) where the dataset includes at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata. The network module 124 sets at least one of a generator network and a discriminator network to generate a plurality of synthetic images resembling real driving scenes under harsh weather conditions. The network module comprises a generator network that generates images similar to real driving scenes under a plurality of weather conditions and the discriminator network differentiates between the real images and generated images. The data generation module 126 generates synthetic data by employing a pre-trained Al model by varying one or more weather condition parameters input into the Al model. The training module 128 incorporates the generated synthetic data into a training dataset for perception models used in autonomous driving systems for training the perception models. The navigation module 130 optimizes a navigation solution using the trained perception models, weather information, and sensor data for enabling resilient lane-level navigation. The navigation module 130 uploads the trained perception models into at least one of: a cloud server or in a roadside unit. The navigation module 130 enhances perception data by enhancing image resolution for improving visibility in low-visibility conditions, accentuatingroad features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions, reducing noise in images caused by low light levels or precipitation, and incorporating the enhanced perception data into a factor graph optimization to improve the positioning solution. The navigation module 130 enhances sensor fusion strategy using deep-learning-based algorithm by ensuring data streams comprising of: images, Global Navigation Satellite System (GNSS) / Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, visual odometry are synchronized in time, timestamping all data inputs to align accurately, and normalizing and pre-processing the data to ensure consistency, capturing one or more temporal dependencies in navigation data, wherein recurrent neural network (RNNs) and Long-Short Term Memory (LSTMs) are selected as a demonstration example as the deep learning network for integrating time-series data from Global Navigation Satellite System (GNSS) / Inertial Navigation System (INS) and odometry, defining a loss function that accurately reflects the performance of the fusion model in predicting vehicle position and orientation, and allowing dynamic adjustment based on real-time data inputs. The lOT / cloud service 132 may include loT devices, roadside units, or cloud-based services.

[0045] According to an embodiment, during training and data enhancement stage, a first step is using a generative Al model for simulated data collection. In some embodiments, one or more generative Al models known in the art is used for the training and simulated data generation is used. Here, generative adversarial networks (GANs) and variational autoencoders (VAEs) are two prominent types of generative models used for data generation. GANs are particularly well-suited for generating realistic images, making them ideal for simulating different weather conditions in visual data. In some embodiments, the VAEs are also used, especially if a model that offers easier training and smoother interpolation is required. The present technology employs several other known generative Al solutions for the simulated data generation and especially multiple-modality -related solutions. Traditional algorithms are also used as optional solutions for dataset generation. For example, visual image with white noise can be added to the training dataset for data enhancement. In other words, traditional image de-noisy algorithms can be added as optional solutions. Algorithms that adjust the intensity values were integrated based on the detected weather conditions or used filtering techniques to reduce noise in the point cloud data. A subsequent step in data generation is to collect a broad dataset of real-world driving conditions across a variety of weather scenarios. The dataset may include images, sensor readings (like LiDAR and radar data), and corresponding metadata (like weather conditions, time of day, and GPS coordinates). The ground-truth weather condition for each piece of data is labeled / given. The data is standardized to ensure consistency, which might involve adjusting the scale of sensor readings or normalizing image pixel values. For model training, two networks, a generator, anda discriminator is set up. The generator leams to create images that look like real driving scenes under various weather conditions, while the discriminator leams to differentiate between real and generated images. To specifically generate data under different weather conditions, a conditional GAN (cGAN) is used, where the condition could be the type of weather. This condition is used as input alongside the noise vector to the generator. For implementation using VAE, a VAE is trained where the encoder compresses the input data into a latent space and the decoder reconstructs the input data from this latent space and conditions the VAE on the weather type to help generate specific weather-based scenarios. During simulated data generation, a trained model is used to generate data by varying the weather condition parameters input into the model. For example, images of a sunny day are generated, then the conditions are modified to simulate rain, snow, fog, etc. In addition, it is crucial to ensure that the synthetic data is realistic and varied. The metrics such as frechet inception distance (FID) are used to compare the distribution of generated data to real data. The generated synthetic data are incorporated into the training dataset for the perception models used in autonomous driving systems. This can be particularly useful for some weather conditions and might be under-represented in real-world datasets. Training with a wider range of conditions can improve the ability of the model to handle real-world variability. With the increasing benchmark and data collection, the performance of the autonomous system in real-world tests to refine the generative models were used. For instance, if the vehicle struggles in foggy conditions, more fog-related images are generated or improve their realism. This is termed as perception data enhancement. Additionally, perception information is added as optional data enhancement solutions. For example, for the visual image, different illumination will highly affect the performance and accuracy of the feature extraction and data association. During the navigation stage, the pre-trained model is uploaded in the cloud server or is uploaded in the roadside unit. The weather information is received from roadside unit, or from weather website, or from other data publishers. The navigation solution is optimized using the weather information. The navigation solution is optimized using the received weather information and the multiple sensors and by estimating a position during normal weather conditions and by fusing the perception-based SLAM outputs and the GNSS / INS solutions, using tightly coupled navigation solutions. If LiDAR, monocular camera, and GNSS / INS is available, then the sensor fusion solution is summarized.

[0046] According to an embodiment, the end-to-end navigation optimization solution can be used for the optimization and LiDAR / Vision GNSS / INS navigation solution. It includes image / point cloud enhancement using weather information to remove the weather effects (foggy, or snowy), it also includes end-to-end network training for the pose estimation using visual / LiDAR streaming data. According to another embodiment, an end-to-end navigation solution for the poseestimation using visual / LiDAR streaming data using a designed loss function is employed. In addition, the error of data enhancement will be considered as one of the loss functions, which is jointly optimized in the end-to-end navigation network. Weather information, pre-trained weather effects data enhancement model, and ground-truth trajectory were used as an input for the training and an estimated trajectory is the output. Here, the map-aided navigation solution is used, which includes map aided LiDAR-based GNSS / INS navigation solution, map aided visual-based GNSS / INS navigation solution, and map map-aided LiDAR / Visual-based GNSS / INS navigation solution.

[0047] According to an embodiment, an extended Kalman Filter (EKF) is implemented to fuse the data from the positioning solution from GNSS and the INS mechanization from the 6-aix inertial measurement unit (IMU) via a loosely coupled sensor fusion strategy. Here, other filter algorithms such as particle filter, and Kalman filter can be used for the GNSS / INS fusion using a loosely coupled sensor fusion and the extended Kalman filter can be considered as an example. The INS mechanization estimates the relative motion state using IMU measurements including angular rates and forces collected using a gyroscope, and an accelerometer, respectively. The GNSS positioning is converted into the East-North-Up (ENU) frame using a pre-defined initial point. A brief introduction of EKF is provided as follows. The state x of the navigation system and variables can be formulated using position r, velocity v, and the IMU parameters including bias and scales. The state function can be formulated as following equation (1):χk= [XkW, vkW, ba,kB, bg,kB]T(1) Where XkW= [xkW, ykW, zkW] is the positioning of the GNSS receiver in the ENU frame at a timestamp k, and vkW= [vkW, vkW, vkW] demonstrates the velocity, respectively. ba,kBand bg,kBdenote the bias of the accelerometer and gyroscope.The state function between relative transformation can be formulated as equation (2):xk= f(xk-1, uk) + wk-1(2) where f and wkrepresent the relative state estimation function and white Gaussian errors, and can be further represented as equation (3):■ *fcW-i + <i,x• At ■ yW 1 + ^-l,y • At ^-1 + <i,z• At17W, W.A / .vk-l,r,x+ ak-l,x11117W, W A fuk-l,r,yak-l,ylSL17W, W.Afr,.vk-l,r,z+ afc-l,z ^Lf(Xk-r, Uk) =Bba,k-l,x ba,k-l,yuhaB,k-l,zuhgB,k-l,xuhgB,k-l,y LuhgB,k-l,z J(3) The measurement model of the EKF can be represented as equation (4):ZK= h(Xk) + ek(4) where ZK= (xkGNSS, ykGNSS, zkGNSS)Tis the positioning solution received from the GNSS receiver, while ekis the white noise of the measurements, which can be described with a covariance matrix. Here, tightly-coupled of GNSS and INS or other filter-based algorithms can be used for the global positioning.After estimating the global positioning based on the fusion of GNSS and INS, another phase of sensor fusion is implemented to fuse other odometers and constraints. Here, the factor graph optimization (FGO) is presented as an implementation example. The major advantages of factor graph are discussed as two aspects. Firstly, the graph structure can efficiently achieve the optimal iteratively, avoiding the sub-optimal solutions based on one-time calculation. Secondly, the graph structure can fuse heterogeneous data with different frequencies. The structure of factor graph optimization (FGO) defines the variable and the factor.The system state can be represented as equation (5):x = [RT, pT,vT,bT]T(5) where R, p and v is the orientation, position, and velocity of the system state, while b is the IMU bias.The objective of factor graph optimization is to minimize the error function, which is conducted using the states and the observations. The error function for tightly coupled system can be demonstrated as following equation (6):TBW*= argmin ∑k=0,1,...,K(||ekodom||2Σ+ ||ekGNSS / INS||2Σ+ ||ekGlobal||2Σ+ ||ekIMU||2Σ) '(6) where types of factors are involved in constructing the factor graph, including the following error function: ekGlobalglobal constraints, ekGNSS / INSGNSS global positioning, ekodomodometry estimated using LiDAR odometer, Radar odometer, or visual odometer, and ekIMUIMU pre-integration. Here the global constraints can be the loop closure for the global optimization, or other global constraints, such as semantic information or control points. Or the global constraints can be other perception information which can be considered as observation measurements, for example, semantic information.For IMU pre-integration, the raw IMU measurements can be represented as below:ât= at+ ba+ RtWgW+ na(7) ω̂t= ωt+ bω+ nω(8) where âtand ω̂tare the raw measurements of a gyroscope and an accelerometer, while ba, bw, na, and nωrepresent the biases and additive noises, respectively. RtWis the rotation matrix of the body frame. The IMU pre-integration can be represented using the following equation:=ff “b«t)df2' / “'te[tfc,tfc+1] Pbkk+1= fRV (at -b«t)dtf 1 Ybk+1= - a(a>t- bwyrfkdt(9)where thePbk+1’an<^ Ybk+1are lbepre-integration items, while:0 — (i)Z(l)y= [-L"Jtx"], Hx= [ roz0 -mx]— d)‘ 01....n-ωyωx0(10) As for the IMU pre-integration factor, the representation of IMU factor can be represented as following:0 -RtW[ât- ba]×-RtW0 0(H) As for other error factors, the map matching constraints from LiDAR can be represented as following equation (12):1 \ „2 Il / MU ||21TW \ f T'lMU llefc IlyIMU1B,k) fB’k / IlyIMULk(12) where ekodomis the error function, Σkodomis the information matrix of the error function, while the operation ⊖ is the minus operation. Here, the ekodomdemonstrates the LiDAR odometer, radar odometer, or visual odometer if they are available, as following equation (13):||ekodom||2Σ= ||(TB,k-1W -1TB,kW) ⊖ (TB,k-1Odom -1TB,kodom)||2Σ(13)As for global constraints ekGlobal, the factor is added to the factor graph when a constrain is available. Loop closure is one of the global constraints. Besides the loop closure, semantic information and control point (e.g., landmarks with known coordinates) can be considered as global constraints as well.For example, a loop closure constraint is added to the FGO as following equation (14):||ekloop||2Σ= ||(TB,iW -1TB,jW) ⊖ TL,iWTi,jloop||2Σ(14) Or when the semantic of point clouds estimated via deep learning network or the semantic landmarks is available, a map registration constraint can be added to FGO as optional constraint,||esemantic(xk, sk)||2Σ= (f(xk, sk) - zsemantic)TΣsemantic-1(f(xk, sk) - zsemantic)(15) Where zsemanticis the observed semantic measurement, Σsemanticis the covariance representing the uncertainty in the semantic observation. f(xk, sk) represents the measurements function that relates the states and the semantic features.Or the control point (cp) is available, an optional constraint can be added to FGO, as following,||ecp(xk)||2Σ= (xk- pk)TΣcp-1(xk- pk)(16)

[0048] During harsh weather such as foggy, rainy, and snowy data, when the weather information is available, relevant weather data is identified that impacts sensor performance, such as fog, snow, or rainy, which might come from onboard sensors, internet data, or dedicated weather services. The weather information is transferred to the image (or point clouds) groundtruth information (e.g., heavy rain, heavy fog, road with ice, heavy snowy, and the like). There are two steps for the enhancement, data enhancement, and the sensor fusion strategy optimization. The perception data and the resolution for better clarity are both enhanced, especially in low-visibility conditions like fog or heavy rain. The present system accentuates road signs, lane markings, and obstacles that may be obscured due to poor weather conditions and reduces noise in images (point clouds) caused by low light levels or precipitation. Then, the enhancement performance is added into the factor graph optimization to further enhance the positioning solution. The equation is optimized as following equation (17):TIV > argmin \? ' ( I II „enhanced-odom ||2. || GNSS / INS ||2\ ^k+ ||ekIMU||2Σ+ ||ekGlobal||2Σ+ ||ekweather||2Σ)(17) where ekenhanced-odomrepresents the error function estimated via the enhanced data aided by the pre-trained models.

[0037] As the LiDAR, radar, and visual data are enhanced using weather information, The covariance S(W7t) is dynamically adjusted based on weather conditions to reflect measurement reliability and is given by equation (18):ll^weather (-^fc> ^t) ll ^yweather ( vwwt4) (vw e ath e r fc ) TAveather(^t)) weatherC^t) (^weatherC-'-fc) AveatherOw))(18) Where h and z represent the expected data measurements from sensors and the observed measurements from sensors.

[0049] Where Wtis weather state node in the graph representing weather conditions at time t. The factor graph is extended to include weather factors alongside visual-inertial and loop closure factors. A solver (e.g., GTSAM or Ceres) is used to perform optimization, considering all constraints and state estimates are adjusted to minimize the overall graph error. The graph is continuously updated as and when new data is received, for refining state estimates based on current weather conditions.

[0050] Beside the FGO algorithm, the sensor fusion strategy is enhanced using other filter-based algorithm and graph-based algorithms.

[0051] Beside the FGO algorithm, the sensor fusion strategy is enhanced using deep-learning-based algorithm. All data streams (GNSS / INS and perception-based data) are synchronized in time. This might involve timestamping all data inputs to align them accurately. The data is normalized and preprocessed to ensure consistency. This includes scaling the sensor readings, standardizing image inputs, and converting all data into a format suitable for neural network processing and as data pre-processing. As a demonstration example, RNNs and LSTMs is selected as the deep learning network, which are ideal for integrating time-series data from GNSS / INS and odometry, capturing temporal dependencies in the navigation data. Meanwhile, multimodal transformers other custom fusion layers are open to fuse the GNSS / INS and perception-based odometry. The loss function is defined, that accurately reflects the performance of the fusion model in predicting vehicle position and orientation. In some embodiments optimization algorithms such as Adam or RMSprop for effective learning is used. The system is allowed to dynamically adjust based on real-time data inputs. For instance, if GNSS data becomes unreliable (e.g., in tunnels or urban canyons), the system should rely more heavily on INS, LiDAR odometry, and visual odometry.

[0052] FIG. 2 illustrates a flow diagram depicting a method 200 for enabling resilient lane-level navigation in harsh weather conditions. At step 202, a dataset associated with one or more real-world driving conditions across harsh weather conditions is collected using artificial intelligence (Al). The dataset includes at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata. At step 204, at least one of a generator network and adiscriminator network is configured to generate a plurality of synthetic images resembling real driving scenes under harsh weather conditions. At step 206, synthetic data is generated by employing a pre-trained Al model by varying one or more weather condition parameters input into the Al model. At step 208, the generated synthetic data is incorporated into a training dataset for perception models used in autonomous driving systems for training the perception models. At step 210, a navigation solution is optimized based on the trained perception models, weather information, and sensor data for enabling the resilient lane-level navigation.

[0053] According to an embodiment, the method further includes uploading the trained perception models in to at least one of a cloud server or in a roadside unit.

[0054] According to an embodiment, a ground-truth weather condition is labeled for a portion of the dataset and the labeled portion of the dataset is standardized to ensure consistency, by performing at least one of adjusting a scale of sensor readings or normalizing image pixel values.

[0055] According to an embodiment, the weather information is received from at least one of a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

[0038] According to an embodiment, the generator network generates synthetic images simulating driving scenes under a plurality of weather conditions and the discriminator network distinguishes between the real images and generated images.

[0056] According to an embodiment, the method for enabling a resilient lane-level navigation further includes enhancing perception data by enhancing image resolution for improving visibility in low-visibility conditions, accentuating road features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions, by reducing noise in images caused by low light levels or precipitation and by incorporating the enhanced perception data into a factor graph optimization to improve the positioning solution.According to an embodiment, the method for enabling a resilient lane-level navigation further includes enhancing sensor fusion strategy using deep-learning-based algorithm, by synchronizing multimodal sensor data including images, Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, and visual odometry using timestamp alignment, by timestamping all data inputs to align accurately, and normalizing and preprocessing the data to ensure consistency, by capturing one or more temporal dependencies in navigation data, where recurrent neural network (RNNs) and long-short term memory (LSTMs) are selected as a demonstration example as the deep learning network forintegrating time-series data from GNSS / INS and odometry, by defining a loss function that accurately reflects the performance of the fusion model in predicting vehicle position and orientation and by allowing dynamic adjustment based on real-time data inputs.

[0057] According to an embodiment, the weather information is received from at least one of a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

[0058] According to an embodiment, the generator network generates synthetic images simulating driving scenes under a plurality of weather conditions and the discriminator network distinguishes between the real images and generated images.

[0039] FIG. 3 illustrates an exemplary computer system 300 in which or with which embodiments of the present disclosure may be implemented. The computer system 300 may include an external storage device 310, a bus 320, a main memory 330, a read-only memory 340, a mass storage device 350, a communication port(s) 360, and a processor 370. A person skilled in the art will appreciate that the computer system 300 may include more than one processor and communication ports. The processor 370 may include various modules associated with embodiments of the present disclosure. The communication port(s) 360 may be any of an RS -232 port for use with a modem-based dialup connection, a 10 / 100 Ethernet port, a Gigabit or 10 Gigabit port using copper or fiber, a serial port, a parallel port, or other existing or future ports. The communication ports(s) 360 may be chosen depending on a network, such as a Local Area Network (LAN), Wide Area Network (WAN), or any network to which the computer system 300 connects.

[0059] In an embodiment, the main memory 330 may be Random Access Memory (RAM), or any other dynamic storage device commonly known in the art. The read-only memory 340 may be any static storage device(s) e.g., but not limited to, a Programmable Read Only Memory (PROM) chip for storing static information e.g., start-up or basic input / output system (BIOS) instructions for the processor 370. The mass storage device 350 may be any current or future mass storage solution, which can be used to store information and / or instructions. Exemplary mass storage solutions include, but are not limited to, parallel advanced technology attachment (PATA) or serial advanced technology attachment (SATA) hard disk drives or solid-state drives (internal or external, e.g., having Universal Serial Bus (USB) and / or Firewire interfaces).

[0060] In an embodiment, the bus 320 may communicatively couple the processor(s) 370 with the other memory, storage, and communication blocks. The bus 320 may be, e.g. a PeripheralComponent Interconnect PCI) / PCI Extended (PCI-X) bus, Small Computer System Interface (SCSI), USB, or the like, for connecting expansion cards, drives, and other subsystems as well as other buses, such a front side bus (FSB), which connects the processor 370 to the computer system 300.

[0061] In another embodiment, operator, and administrative interfaces, e.g., a display, keyboard, and cursor control device may also be coupled to the bus 320 to support direct operator interaction with the computer system 300. Other operator and administrative interfaces can be provided through network connections connected through the communication port(s) 360. Components described above are meant only to exemplify various possibilities. In no way should the aforementioned exemplary computer system 300 limit the scope of the present disclosure.

[0062] The embodiments herein can include both hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. Furthermore, the embodiments herein can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0063] The system, method, computer program product, and propagated signal described in this application may, of course, be embodied in hardware; e.g., within or coupled to a Central Processing Unit (" CPU"), microprocessor, microcontroller, System on Chip (" SOC"), or any other programmable device. Additionally, the system, method, computer program product, and propagated signal may be embodied in software (e.g., computer readable code, program code, instructions and / or data disposed in any form, such as source, object or machine language) disposed, for example, in a computer usable (e.g., readable) medium configured to store the software. Such software enables the function, fabrication, modeling, simulation, description and / or testing of the apparatus and processes described herein.

[0064] Such software can be disposed in any known computer usable medium including semiconductor, magnetic disk, optical disc (e.g., CD-ROM, DVD-ROM, and the like) and as a computer data signal embodied in a computer usable (e.g., readable) transmission medium (e.g., carrier wave or any other medium including digital, optical, or analog-based medium). As such, the software can be transmitted over communication networks including the Internet and intranets. A system, method, computer program product, and propagated signal embodied insoftware may be included in a semiconductor intellectual property core (e.g., embodied in HDL) and transformed to hardware in the production of integrated circuits. Additionally, a system, method, computer program product, and propagated signal as described herein may be embodied as a combination of hardware and software.

[0065] A "computer-readable medium" for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, system, or device. The computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.

[0066] A "processor" or "process" includes any human, hardware and / or software system, mechanism or component that processes data, signals or other information. A processor can include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location or have temporal limitations. For example, a processor can perform its functions in "real time," "offline," in a "batch mode," etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems.

[0067] The present invention provides highly resilient lane-level navigation solutions in all scenarios and harsh weathers, especially the urban canyons (e.g., areas with tall buildings) and heavy rains, snowy and foggy and also to improve the resiliency of the navigation solution, providing accurate positioning solutions for advanced driver-assistance systems (ADAS) and other smart vehicle applications. The weather information to enhance the navigation solution is involved in the present technology. The weather information is not only for the sensor fusion optimization but also for the perception data enhancement. In an embedded module of Navigation App (Google Map), the navigation app (or software) can create a weather button for this technology. When this function is needed, the APP (or software) receives weather information, enhances the image (or, point clouds), and then provides lane-level navigation solution. In this technology, internet of things (IoT) technology is involved for the autonomous driving and navigation solution. Also, a multi-phase sensor fusion strategy is presented in this technology. The weather information is involved in two stages. Feature extraction and data enhancement and navigation solution optimization, including the navigation mode selection and fusion strategy optimization. A multi-phase sensor fusion strategy using weather information is presented in this technology and weather information is involved in the navigation system. The present system leverages a GAN network for the generation of datasets that serve two purposes: (a) Enhancingdata quality under adverse weather conditions, such as fog or snow and (b) Integrating weather factors into the positioning and navigation solution through joint optimization.

[0068] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such as specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modifications. However, all such modifications are deemed to be within the scope of the claims. The scope of the embodiments will be ascertained by the claims to be submitted at the time of filing a complete specification.

Claims

CLAIMSWhat is claimed is:

1. A method for enabling resilient lane-level navigation in harsh weather conditions, the method comprising;collecting, by a collection module, a dataset associated with one or more real-world driving conditions during harsh weather conditions using artificial intelligence (Al), wherein the dataset comprises at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata;configuring, by a network module, at least one of a generator network and a discriminator network to generate a plurality of synthetic images resembling real driving scenes under harsh weather conditions;generating, by a data generation module, synthetic data using a pre-trained Al model by varying one or more weather condition parameters input into the Al model; incorporating, by a training module, the generated synthetic data into a training dataset for training one or more perception models used in autonomous driving systems; andoptimizing, by a navigation module, a navigation solution based on the trained perception models, weather information, and sensor data to enable resilient lane-level navigation.

2. The method of claim 1, further comprising uploading the trained perception models to at least one of a cloud server or a roadside unit.

3. The method of claim 1, wherein a ground-truth weather condition is labeled for a portion of the dataset, and the labeled portion of the dataset is standardized by performing at least one of: adjusting a scale of sensor readings or normalizing image pixel values.

4. The method of claim 1, wherein the weather information is received from at least one of: a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

5. The method of claim 1, wherein the generator network generates synthetic images simulating driving scenes under a plurality of weather conditions, and the discriminator network distinguishes between real and generated images.

6. The method of claim 1, further comprising enhancing perception data by:enhancing image resolution to improve visibility in low-visibility conditions; accentuating road features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions;reducing image noise caused by low-light or precipitation; andincorporating the enhanced perception data into a factor graph optimization for improved positioning solution.

7. The method of claim 1, further comprising enhancing a sensor fusion strategy using a deep-learning-based algorithm, the enhancement comprising:synchronizing multimodal sensor data including images, Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, and visual odometry using timestamp alignment;normalizing and preprocessing the data to ensure consistency;capturing temporal dependencies in the navigation data using at least one of a Recurrent Neural Network (RNNs) or a LongShort-Term Memory (LSTM) model;defining a loss function representing performance in position and orientation prediction; anddynamically adjusting the fusion model based on real-time sensor input.

8. A system for enabling resilient lane-level navigation in harsh weather conditions, the system comprising;a processor to fetch and execute computer-readable instructions stored in the memory of the system;a memory storing computer-readable instructions executable by the processor; and the instructions comprising:a collection module configured to collect a dataset associated with one or more real- world driving conditions during harsh weather conditions using artificial intelligence (Al), wherein the dataset comprises at least one of a plurality of images, a plurality of sensor readings, and corresponding metadata;a network module configured to set at least one of a generator network and a discriminator network to generate a plurality of synthetic images resembling real driving scenes under harsh weather conditions;a data generation module configured to generate synthetic data using a pre-trained Al model by varying one or more weather condition parameters input into the Al model;a training module configured to incorporate the generated synthetic data into a training dataset for training one or more perception models used in autonomous driving systems; anda navigation module configured to optimize a navigation solution based on the trained perception models, weather information, and sensor data to enable resilient lanelevel navigation.

9. The system of claim 8, wherein the navigation module is further configured to upload the trained perception models to at least one of a cloud server or a roadside unit.

10. The system of claim 8, wherein the collection module is further configured to label groundtruth weather condition for a portion of the dataset and standardize the labeled portion of the dataset by performing at least one of: adjusting a scale of sensor readings or normalizing image pixel values.

11. The system of claim 8, wherein the navigation module receives weather information from at least one of: a roadside unit, a weather website, or one or more data publishers to optimize the navigation solution.

12. The system of claim 8, wherein the network module comprises a generator network to generate synthetic images simulating driving scenes under a plurality of weather conditions and the discriminator network for distinguishing between real and generated images.

13. The system of claim 8, wherein the navigation module is further configured to enhance perception data by:enhancing image resolution to improve visibility in low-visibility conditions; accentuating road features including road signs, lane markings, and obstacles that may be obscured due to poor weather conditions;reducing image noise caused by low-light or precipitation; andincorporating the enhanced perception data into a factor graph optimization for improved positioning solution.

14. The system of claim 8, wherein the navigation module is further configured to enhance a sensor fusion strategy using a deep-learning-based algorithm, by:synchronizing multimodal sensor data including images, Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Light Detection and Ranging (LiDAR), Radar, and visual odometry using timestamp alignment;normalizing and pre-processing the data to ensure consistency;capturing temporal dependencies in the navigation data using at least one of a Recurrent Neural Network (RNNs) or a Long Short-Term Memory (LSTM) model;defining a loss function representing performance in position and orientation prediction; anddynamically adjusting the fusion model based on real-time sensor input.

Images