Relocalization using inertial measurement unit
Machine learning models process IMU data to improve localization accuracy and reduce computational complexity, addressing the challenges of noisy IMU data in SLAM systems, resulting in efficient and accurate relocalization.
Patent Information
- Application Number
- PCT/US2026/026401
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-05-01
- Publication Date
- 2026-11-12
AI Technical Summary
Existing localization systems face challenges in accurately and efficiently relocalizing devices due to the noisy and drifting nature of inertial measurement unit (IMU) data, leading to increased error accumulation, latency, and high computational complexity in simultaneous localization and mapping (SLAM) processes.
Utilizing machine learning models to process IMU data, reducing noise and drift, and integrating it with image data to improve localization accuracy and reduce computational complexity by limiting search regions in the mapping process.
Enhances localization accuracy, reduces error accumulation, and decreases power consumption by leveraging IMU data with machine learning to provide efficient and faster relocalization.
Smart Images

Figure US2026026401_12112026_PF_FP_ABST
Abstract
Description
PATENTQualcomm Ref. No. 2504046WO1SYSTEMS AND METHODS FOR RELOCALIZATION USING INERTIAL MEASUREMENT UNITCROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims the priority oflndia Patent Application No. 202541044767, filed May 8, 2025, which is incorporated by reference herein in its entirety.FIELD
[0002] This application is related to localization. More specifically, this application relates to systems and methods for pose determination and / or relocalization based on sensor data from an inertial measurement unit and / or image sensor(s).BACKGROUND
[0003] Many devices include one or more cameras. For example, a smartphone or tablet includes a front facing camera to capture selfie images and a rear facing camera to capture an image of a scene (such as a landscape or other scenes of interest to a device user). A camera can capture images using an image sensor of the camera, which can include an array of photodetectors. Some devices can analyze image data captured by an image sensor to detect an object within the image data. Sometimes, cameras can be used to capture images of scenes that include one or more people. Inertial measurement units (IMUs) include accelerometers and / or gyroscopes, and can be used to detect motion of a system.BRIEF SUMMARY
[0004] Localization systems and techniques are described. In some examples, a localization system processes a plurality of inertial measurement unit (IMU) measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time. A plurality of timestamps correspond to capture times of the plurality of IMU measurements by an IMU within a context time window. The prediction time is after the context time window. In some examples, in response to a comparison between a confidence associated with thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO2indication of the predicted pose and a relocalization threshold, the localization system dynamically initializes a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.
[0005] In one example, an apparatus is provided that includes one or more memories and one or more processors coupled to the one or more memories. The at least one processor is configured to: process a plurality of IMU measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU, and wherein the prediction time is after the context time window; and in response to a comparison between a confidence associated with the indication of the predicted pose and a relocalization threshold, dynamically initialize a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.
[0006] According to at least one example, a method is provided. The method includes: processing a plurality of IMU measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU, and wherein the prediction time is after the context time window; and in response to a comparison between a confidence associated with the indication of the predicted pose and a relocalization threshold, dynamically initializing a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.
[0007] In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: process a plurality of IMU measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU, and wherein the prediction time is after the context time window; and in response to a comparison between a confidence associated with the indication of the predicted pose and a relocalization threshold, dynamically initialize a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO0
[0008] In another example, an apparatus for imaging is provided. The apparatus includes: means for processing a plurality of IMU measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU, and wherein the prediction time is after the context time window; and in response to a comparison between a confidence associated with the indication of the predicted pose and a relocalization threshold, means for dynamically initializing a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.
[0009] In some aspects, the apparatus is part of, and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile telephone and / or mobile handset and / or so-called “smart phone” or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatuses described above can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensor).
[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.
[0011] The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO4BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Illustrative aspects of the present application are described in detail below with reference to the following drawing figures:
[0013] FIG. l is a block diagram illustrating an example architecture of an image capture and processing system, in accordance with some examples;
[0014] FIG. 2 is a graph diagram illustrating relative translation error increasing over time for traditional localization based on inertial measurement unit (IMU) data, in accordance with some examples;
[0015] FIG. 3 is a block diagram illustrating a timeline identifying timing of IMU measurements from an IMU and image frames from an image sensor relative to a context time window and a prediction time window, in accordance with some examples;
[0016] FIG. 4 is a block diagram illustrating an imaging system that processes IMU measurements and / or image frames using machine learning model(s) (ML model(s)) to generate an indication of a predicted pose associated with the prediction time window, in accordance with some examples;
[0017] FIG. 5 is a block diagram illustrating a simultaneous localization and mapping (SLAM) system that processes IMU measurements and / or image frames using machine learning model(s) (ML model(s)) to track features, manage a map of an environment that the system is in, and locate (or relocate) the system within the map, in accordance with some examples;
[0018] FIG. 6 is a block diagram illustrating an example of a neural network that can be used for imaging operations, in accordance with some examples;
[0019] FIG. 7 is a conceptual diagram illustrating matching of points between a first image and a second image, in accordance with some examples;
[0020] FIG. 8 is a graph diagram illustrating reduction in frequency of relative translation error in relocalization by a system with the ML engine relative to a system without the ML engine, in accordance with some examples;Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO5
[0021] FIG. 9Ais a perspective diagram illustrating a head-mounted display (HMD) that is used as part of an imaging system, in accordance with some examples;
[0022] FIG. 9B is a perspective diagram illustrating the head-mounted display (HMD) of FIG.9A being worn by a user, in accordance with some examples;
[0023] FIG. 10A is a perspective diagram illustrating a front surface of a mobile handset that includes front-facing cameras and that can be used as part of an imaging system, in accordance with some examples;
[0024] FIG. 10B is a perspective diagram illustrating a rear surface of a mobile handset that includes rear-facing cameras and that can be used as part of an imaging system, in accordance with some examples;
[0025] FIG. 11 is a perspective diagram illustrating a vehicle that includes various sensors and that can be used as part of an imaging system, in accordance with some examples;
[0026] FIG. 12Ais a perspective diagram illustrating an unmanned ground vehicle (UGV) that can be used as part of an imaging system, in accordance with some examples;
[0027] FIG. 12B is a perspective diagram illustrating an unmanned aerial vehicle (UAV) that can be used as part of an imaging system, in accordance with some examples;
[0028] FIG. 13 is a flow diagram illustrating a process for imaging, in accordance with some examples; and
[0029] FIG. 14 is a diagram illustrating an example of a computing system for implementing certain aspects described herein.DETAILED DESCRIPTION
[0030] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO6However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.
[0031] The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example aspects will provide those skilled in the art with an enabling description for implementing an example aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.
[0032] An inertial measurement unit (IMU) is a device that measures and reports a system's specific force, angular rate, the magnetic field surrounding the system, and / or pressure on the system, using a combination of accelerometers, gyroscopes, magnetometers, and / or pressure sensors. In some cases, measurements from an IMU can also be used to determine a speed or velocity of the system, a direction of movement of the system, a position of the system (e.g., latitude, longitude, and altitude or elevation), an orientation (e.g., pitch, yaw, and / or roll).
[0033] A camera is a device that receives light and captures image frames, such as still images or video frames, using an image sensor. The terms “image,” “image frame,” and “frame” are used interchangeably herein. Cameras can be configured with a variety of image capture and image processing settings. The different settings result in images with different appearances. Some camera settings are determined and applied before or during capture of one or more image frames, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain. For example, settings or parameters can be applied to an image sensor for capturing the one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as alterations to contrast, brightness, saturation, sharpness, levels, curves, or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) for processing the one or more image frames captured by the image sensor.
[0034] Localization systems and techniques are described. In some examples, a localization system processes a plurality of inertial measurement unit (IMU) measurements and an indication of a prediction time to generate an indication of a predicted pose associated with the prediction time. A plurality of timestamps correspond to capture times of the plurality of IMU measurementsPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO7by an IMU within a context time window. The prediction time is after the context time window. Tn some examples, in response to a comparison between a confidence associated with the indication of the predicted pose and a relocalization threshold, the localization system dynamically initializes a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map.
[0035] Various aspects of the application will be described with respect to the figures. FIG. 1 is a block diagram illustrating an architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components that are used to capture and process images of one or more scenes (e.g., an image of a scene 110). The image capture and processing system 100 can capture standalone images (or photographs) and / or can capture videos that include multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces a scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by an image sensor 130. In some examples, the scene 110 is a scene in an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For instance, the scene 110 can be a scene of one or both of the user’s eyes, and / or at least a portion of the user’s face.
[0036] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for instance, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms besides those that are illustrated, such as control mechanisms controlling analog gain, flash, HDR, depth of field, and / or other image capture properties.
[0037] The focus control mechanism 125B of the control mechanisms 120 can obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focusPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO8setting, the focus control mechanism 125B can move the lens 115 closer to the image sensor 130 or farther from the image sensor 130 by actuating a motor or servo, thereby adjusting focus. Tn some cases, additional lenses may be included in the system 100, such as one or more microlenses over each photodiode of the image sensor 130, which each bend the light received from the lens 115 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.
[0038] The exposure control mechanism 125A of the control mechanisms 120 can obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control a size of the aperture (e.g., aperture size or f / stop), a duration of time for which the aperture is open (e.g., exposure time or shutter speed), a sensitivity of the image sensor 130 (e.g., ISO speed or fdm speed), analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.
[0039] The zoom control mechanism 125C of the control mechanisms 120 can obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control a focal length of an assembly of lens elements (lens assembly) that includes the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which can be lens 115 in some cases) that receives the light from the scene 110 first, with the light then passing through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses of equal or similarPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO9focal length (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.
[0040] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures an amount of light that eventually corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and may thus measure light matching the color of the filter covering the photodiode. For instance, Bayer color filters include red color filters, blue color filters, and green color filters, with each pixel of the image generated based on red light data from at least one photodiode covered in a red color filter, blue light data from at least one photodiode covered in a blue color filter, and green light data from at least one photodiode covered in a green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also referred to as “emerald”) color filters instead of or in addition to red, blue, and / or green color filters. Some image sensors may lack color filters altogether, and may instead use different photodiodes throughout the pixel array (in some cases vertically stacked). The different photodiodes throughout the pixel array can have different spectral sensitivity curves, therefore responding to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.
[0041] In some cases, the image sensor 130 may alternately or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes, or portions of certain photodiodes, at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier to amplify the analog signals output by the photodiodes and / or an analog to digital converter (ADC) to convert the analog signals output by the photodiodes (and / or amplified by the analog gain amplifier) into digital signals. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may be included instead or additionally in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO10
[0042] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 1 4), one or more host processors (including host processor 152), and / or one or more of any other type of processor 1410 discussed with respect to the computing system 1400. The host processor 152 can be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip can also include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 can include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output port). In one illustrative example, the host processor 152 can communicate with the image sensor 130 using an I2C port, and the ISP 154 can communicate with the image sensor 130 using an MIPI port.
[0043] The image processor 150 may perform a number of tasks, such as de-mosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, receipt of inputs, managing outputs, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1420, read-only memory (ROM) 145 and / or 1425, a cache, a memory unit, another storage device, or some combination thereof.
[0044] Various input / output (VO) devices 160 may be connected to the image processor 150. The I / O devices 160 can include a display screen, a keyboard, a keypad, a touchscreen, a trackpad, a touch-sensitive surface, a printer, any other output devices 1435, any other input devices 1445, or some combination thereof. In some cases, a caption may be input into the image processingPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO11device 105B through a physical keyboard or keypad of the I / O devices 160, or through a virtual keyboard or keypad of a touchscreen of the I / O devices 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, over which the system 100 may receive data from the one or more peripheral device and / or transmit data to the one or more peripheral devices. The I / O devices 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, over which the system 100 may receive data from the one or more peripheral device and / or transmit data to the one or more peripheral devices. The peripheral devices may include any of the previously-discussed types of I / O devices 160 and may themselves be considered I / O devices 160 once they are coupled to the ports, jacks, wireless transceivers, or other wired and / or wireless connectors.
[0045] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105 A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 10 A and the image processing device 105B may be coupled together, for example via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105 A and the image processing device 105B may be disconnected from one another.
[0046] As shown in FIG. 1, a vertical dashed line divides the image capture and processing system 100 of FIG. 1 into two portions that represent the image capture device 105 A and the image processing device 105B, respectively. The image capture device 105 A includes the lens 115, control mechanisms 120, and the image sensor 130. The image processing device 105B includes the image processor 150 (including the ISP 154 and the host processor 152), the RAM 140, the ROM 145, and the VO devices 160. In some cases, certain components illustrated in the image capture device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image capture device 105 A.
[0047] The image capture and processing system 100 can include an electronic device, such as a mobile or stationary telephone handset (e.g., smartphone, cellular telephone, or the like), aPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO12desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 can include one or more wireless transceivers for wireless communications, such as cellular network communications, 1402.11 WiFi communications, wireless local area network (WLAN) communications, or some combination thereof. In some implementations, the image capture device 105 A and the image processing device 105B can be different devices. For instance, the image capture device 105Acan include a camera device and the image processing device 105B can include a computing device, such as a mobile handset, a desktop computer, or other computing device.
[0048] While the image capture and processing system 100 is shown to include certain components, one of ordinary skill will appreciate that the image capture and processing system 100 can include more components than those shown in FIG. 1. The components of the image capture and processing system 100 can include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The software and / or firmware can include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image capture and processing system 100.
[0049] FIG. 2 is a graph diagram 200 illustrating relative translation error 220 increasing over time 210 for traditional localization based on inertial measurement unit (IMU) data. The graph diagram 200 plots time 210 in milliseconds (ms) along a horizontal axis, and plots the relative translation error 220 in centimeters (cm) along a vertical axis.
[0050] In simultaneous localization and mapping (SLAM) and other mapping technologies, relocalization refers to a process of determining the pose (position and orientation) of the devicePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO13after it has lost the track of its location in an old map. In some examples, a system can use image data from image sensor(s) (e.g., of camera(s)) for SLAM, which may be referred to as visual simultaneous localization and mapping (VSLAM).
[0051] In some examples, once a SLAM system identifies that there is a tracking failure, the system starts to build a new map (e g., based on new image frames from the camera). The system then searches and matches the descriptors from the current frame with the features from the old map. This search and match process involves searching across all the descriptors of the old map. If multiple maps exist, then this search and matching process involves searching across all the descriptors of each map. In some examples, once a match is found, the system estimates the pose of the device with respect to the old map. This process can be time and resource consuming, and can increase relocalization time, as it involves searching across large portions of the map and performing extensive feature matching operations. This results in increased latency, higher processor utilization, and increased power consumption relative to systems that limit search regions or improve pose estimation accuracy.
[0052] In some examples, a system can use IMU data from an IMU for SLAM, either situationally (e.g., when cameras are disabled or image data from the cameras is not clear), in addition to image data, or instead of image data. For instance, a system can use IMU measurements captured by an IMU to determine the dead-reckoning pose and search for the descriptors in the limited region of the old map. However, the dead-reckoning pose drifts with time and degrades quickly due to noise and other non-linearities in IMU measurements. In particular, as shown in the graph diagram 200, the relative translation error 220 in traditional localization and / or relocalization that is based on IMU data increases exponentially over time 210. This is because IMU measurements are noisy. Systems and methods are disclosed for improving over other techniques for IMU-based localization and / or relocalization by using trained machine learning models to process IMU data to reduce noise and therefore reduce errors in IMU data, greatly improving accuracy of localization and / or relocalization based on IMU data. In some examples, the techniques described herein provide a technical improvement to localization systems by reducing error accumulation associated with inertial sensor drift, improving pose estimation accuracy over time, and reducing computational complexity associated with relocalization operations. In some examples, these improvements enable morePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO14efficient use of processor resources, reduced power consumption, and faster relocalization by limiting search space within a map. In some examples, these improvements represent enhancements to operation of sensor processing systems, machine learning-based estimation systems, and simultaneous localization and mapping (SLAM) systems.
[0053] FIG. 3 is a block diagram illustrating a timeline 300 identifying timing of IMU measurements 340 from an IMU and image frames 345 from an image sensor relative to a context time window 310 and a prediction time window 320. In particular, the timeline 300 tracks time 305 on a linear arrow from left to right. Numerous downward-facing arrows are shown along the left-to-right timeline 300, with each downward arrow representing either IMU measurements 340 or image frames 345 as indicated by the legend 330. In particular, the legend 330 indicates that the thinner, shorter downward-facing arrows represent IMU measurements 340 captured by the IMU(s) of a system, while the thicker, longer downward-facing arrows represent image frames 345 captured by image sensor(s) of camera(s) of the system.
[0054] Two distinct time windows are highlighted along the timeline 300. The time windows include a context time window 310 and a prediction time window 320. The prediction time window 320 refers to the specific time frame during which the systems disclosed herein predict the delta pose. IMU measurements 340 are available for this specific period. The context time window 310 refers to a specific time frame just before the prediction time window 320. Within the example illustrated in FIG. 3, a system has already received sensor data (e.g., the IMU measurements 340 and / or the image frames 345) captured by sensor(s) of the system (e.g., IMU(s) and / or image sensor(s)) during the context time window 310. In some examples, the system has not yet received at least a subset of the image frames 345 that are to be captured by image sensor(s) of the camera(s)) of the system during the prediction time window 320 (e.g., because the prediction time window 320, or at least a portion thereof, has not yet occurred). In some examples, the system has received some of the IMU measurements 340 that are to be captured by the IMU of the system during the prediction time window 320 (e.g., because some of the prediction time window 320 may have already occurred, and because the IMU captures the IMU measurements 340 at a faster rate than the image sensor(s) capture the image frames 345). In some examples, the system has not yet received at least a subset of the IMU measurements 340 that are to be captured by image sensor(s) (of the camera(s)) of the system during the prediction time windowPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO15320 (e.g., because the prediction time window 320, or at least a portion thereof, has not yet occurred). Thus, in some examples, sensor data (e.g., the TMU measurements 340 and / or the image frames 345) captured by sensor(s) of the system (e.g., IMU(s) and / or image sensor(s)) during the context time window 310 can be used to predict a pose of the system during the prediction time window 320, a pose delta (or pose difference) between a pose of the system at the beginning of the prediction time window 320 (e.g., which is also the end of the context time window 310) and an end of the prediction time window 320, a pose delta (or pose difference) between a current pose of the system (e.g., during or at the end of the context time window 310) and a pose of the system associated with the prediction time window 320 (e.g., during or at the end of the prediction time window 320), or a combination thereof. In some examples, the pose delta can include a relative transformation between poses corresponding to different times within the prediction time window. In some examples, the prediction is performed in real time prior to availability of image frames corresponding to the prediction time window, which allows for low-latency pose estimation. In some examples, the prediction time window spans a duration that includes multiple intervals between consecutive image frames, enabling prediction of pose over longer time horizons than image-based estimation alone.
[0055] As shown by the timeline 300, an IMU is generally able to capture IMU measurements 340 at a significantly faster rate than an image sensor is able to capture image frames 345. For instance, in some examples, an image sensor can capture the image frames 345 at 30 frames per second (or 30 Hz), while the IMU can capture the IMU measurements 340 at higher sampling rates, for instance in the range of 1.5 Hz (measurements per second) to 32000 Hz depending on the configuration and make. In an illustrative example, the image frames 345 can be captured (by the image sensor(s)) at 100 Hz and / or 100 frames per second (fps). Thus, even if the IMU measurements 340 are noisy, and IMU-based localization tends to develop significant error over time, a system generally has more granular IMU measurements 340 than image frames 345, and may have more up-to-date IMU measurements 340 than image frames 345. For instance, the IMU measurements 340 can get sub-sampled pose data from between image frames 345. Thus, use of the IMU measurements 340 can greatly contribute to accuracy of localization and / or relocalization, and / or to how up-to-date the localization and / or relocalization is, if the error can be reduced. In some examples, image sensor capture rate and IMU capture rate can vary based onPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO16configuration and model. In some examples, the IMU can capture the IMU measurements 340 at 1.5 Hz to 32 kilohertz (kHz). In some examples, the image sensor(s) of the camera(s) can capture image frames at 15 fps, 30 fps, 45 fps, 60 fps, 75 fps, 90 fps, 105 fps, or 120 fps. However, having the IMU capture at a faster rate than the image sensor(s) of the camera(s) generally uses less power and fewer computational resources than the opposite configuration, and the IMU measurements 340 can be used to determine sub-sampled pose data from between image frames 345 in any such configuration.
[0056] FIG. 4 is a block diagram illustrating an imaging system 400 that processes IMU measurements 405 and / or image frames 410 using machine learning model(s) (ML model(s) 425) to generate an indication 430 of a predicted pose 435 associated with the prediction time window 320. The ML model(s) 425 can be trained and / or updated using an ML engine 420. In some examples, the ML engine 420 can pre-process and / or organize inputs to the ML model(s) 425 before the ML model(s) 425 process the inputs.
[0057] The inputs received by the ML model(s) 425 can include IMU measurements 405 (e.g., of the IMU measurements 340) captured by one or more IMU(s) of a system during the context time window 310 and / or during the prediction time window 320 (to the extent IMU measurements 405 captured during the prediction time window 320 have been received). In some examples, the inputs received by the ML model(s) 425 include timestamps corresponding to capture of the sensor data (e.g., IMU measurements 405, images, videos, and / or other sensor data) by sensor(s) (e.g., the IMU(s), camera(s), image sensor(s), and / or other sensor(s)) during the context time window 310 and / or during the prediction time window 320. In some examples, the inputs received by the ML model(s) 425 include image frames 410 (e.g., of the image frames 345) captured by one or more image sensor(s) (e.g., image sensor 130) of a system during the context time window 310 and / or during the prediction time window 320 (to the extent IMU measurements 405 captured during the prediction time window 320 have been received). In some examples, the inputs received by the ML model(s) 425 include timestamps (e.g., image timestamps) corresponding to capture of the image frames 410 by the image sensor(s) during the context time window 310 and / or during the prediction time window 320. The inputs received by the ML model(s) 425 include indication(s) of the prediction time window 320, and / or of a prediction time within the prediction time window 320. For instance, in some examples, the inputs received by the MLPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO17model(s) 425 include indication(s) of a beginning time of the prediction time window 320, an end time of the prediction time window 320, a middle time of the prediction time window 320, expected IMU capture times (of additional IMU measurements by the IMU) during the prediction time window 320, expected image capture times (of additional image frames captured by the image sensor(s)) during the prediction time window 320, or a combination thereof. In some examples, the ML model(s) 425 generate the predicted pose using IMU measurements without using image frames, including scenarios in which image sensors are occluded, unavailable, or provide low-quality image data. In some examples, the ML model(s) 425 generate the predicted pose using image frames without using IMU measurements, including scenarios in which the IMU is defective, unavailable, or providing low-quality IMU data.
[0058] The ML model (s) 425 process the input(s) described above to generate an indication 430 of the predicted pose 435 associated with the prediction time window 320. In some examples, the ML model(s) 425 include a direct pose regression model that maps IMU measurements to the predicted pose without intermediate integration. In some examples, the ML model(s) 425 include an IMU correction model that generates corrected IMU measurements by reducing noise, bias, or drift in the IMU measurements, and the corrected IMU measurements are subsequently used to determine the predicted pose. The indication 430 of the predicted pose 435 associated with the prediction time window 320 can include a pose of the imaging system 400 during the prediction time window 320, a pose of the imaging system 400 at a specific time during the prediction time window 320 (e.g., a beginning time of the prediction time window 320, an end time of the prediction time window 320, a middle time of the prediction time window 320, an expected IMU capture time of an additional IMU measurement by the IMU during the prediction time window 320, an expected image capture time of an additional image frame by the image sensor(s) during the prediction time window 320, another time within the prediction time window 320, or a combination thereof). In some examples, the corrected IMU measurements are used to determine a corrected pose of the device.
[0059] The indication 430 of the predicted pose 435 associated with the prediction time window 320 can include a pose delta (or pose difference) between a pose of the imaging system 400 at the beginning of the prediction time window 320 (e.g., which is also the end of the context time window 310) and an end of the prediction time window 320. The indication 430 of the predictedPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO18pose 435 associated with the prediction time window 320 can include a pose delta (or pose difference) between a current pose of the imaging system 400 (e.g., during or at the end of the context time window 310) and a pose of the imaging system 400 associated with the prediction time window 320 (e.g., during or at the end of the prediction time window 320). The indication 430 of the predicted pose 435 associated with the prediction time window 320 can include a pose delta (or pose difference) between a pose of the imaging system 400 at a first time within the prediction time window 320 and a pose of the imaging system 400 at a second time within the prediction time window 320. The indication 430 of the predicted pose 435 associated with the prediction time window 320 can include a pose delta (or pose difference) between a pose of the imaging system 400 at a first time outside of the prediction time window 320 (e.g., within the context time window 310) and a pose of the imaging system 400 at a second time within the prediction time window 320. In some examples, the imaging system 400 (e.g., the ML engine 420 and / or the ML model(s) 425) performs, estimates, approximates, and / or predicts results of one or more pose regression calculations. In some examples, the ML model(s) 425 can be referred to as direct pose regression networks.
[0060] In some examples, the imaging system 400 (e.g., the ML engine 420 and / or the ML model(s) 425) performs some additional processing of the output(s) of the ML model(s) 425 in order to generate the indication 430 of the predicted pose 435 associated with the prediction time window 320. In some examples, this additional processing can include integral calculation(s) 415. For instance, in some examples, the output(s) of the ML model(s) 425 can be an acceleration value. The integral calculation(s) 415 can include an integral taken over the acceleration value to obtain a velocity value or speed value. The integral calculation(s) 415 can include a double integral taken over the acceleration value to obtain pose value(s) (e.g., position and / or orientation). The integral calculation(s) 415 can include an integral taken over the velocity value or speed value to obtain pose value(s) (e.g., position and / or orientation). In some examples, the ML model(s) 425 directly output pose values without performing integration. In some examples, the ML model(s) 425 output intermediate motion parameters that are subsequently processed using integration. In some examples, the additional processing can include regression calculation(s), for instance to obtain a value for a pose as a continuous variable (e.g., position and / or orientation). In some examples, the imaging system 400 (e.g., the ML engine 420 and / orPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO19the ML model(s) 425) performs, estimates, approximates, and / or predicts results of the additional processing (e.g., the integral calculation(s) 415 and / or the regression calculation(s)).
[0061] In some examples, the imaging system 400 (e.g., the ML engine 420) can update the ML model(s) 425 over time (e.g., using an update 450) to improve an accuracy of the ML model(s) 425 over time. For instance, in some examples, the imaging system 400 (e.g., the ML engine 420) can compare an observed pose 440 during the prediction time window 320 to the indication 430 of the predicted pose 435 associated with the prediction time window 320 to identify a difference between the two. In some examples, the observed pose 440 is based on additional sensor data 445 (e.g., additional IMU measurements and / or additional image frames) captured by sensor(s) (e.g., IMU(s), image sensor(s)) during the prediction time window 320. In some examples, the imaging system 400 (e.g., the ML engine 420) can base the update 450 to the ML model(s) 425 on the comparison (e.g., the difference) between the observed pose 440 during the prediction time window 320 and the indication 430 of the predicted pose 435 associated with prediction time window 320. For instance, if the comparison indicates that the observed pose 440 during the prediction time window 320 matches the indication 430 of the predicted pose 435 associated with prediction time window 320, or that a difference between the two is less than (or equal to) a threshold, this can be interpreted as positive feedback. Based on such positive feedback, the update 450 to ML Model(s) 425 can strengthen and / or reinforce weights (and / or connections and / or hyperparameters) associated with generation of the indication 430 to encourage the ML model(s) 425 to generate similar output(s) given similar input(s). In this way, the update 450 can improve the ML model(s) 425 itself by improving the accuracy of the ML model(s) 425 in generating output(s) (e.g., the indication 430) that are similarly accurate given similar input(s) (e.g., the IMU measurements 405, the image frames 410). In some examples, updating the ML model(s) 425 improves operation of the ML model(s) 425 themselves by reducing prediction error associated with IMU measurements and improving robustness to sensor noise and bias, thereby improving subsequent pose predictions.
[0062] On the other hand, if the comparison indicates that a difference between the observed pose 440 during the prediction time window 320 and the indication 430 of the predicted pose 435 associated with the prediction time window 320 is greater than (or equal to) the threshold, this can be interpreted as negative feedback. Based on such negative feedback, the update 450 to MLPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO20Model(s) 425 can weaken and / or remove weights (and / or connections and / or hyperparameters) associated with generation of the indication 430 to discourage the ML model(s) 425 from generating similar output(s) given similar input(s). In this way, the update 450 can improve the ML model(s) 425 itself by improving the accuracy of the ML model(s) 425 in generating output(s) (e.g., the indication 430) that are more accurate given similar input(s) (e.g., the IMU measurements 405, the image frames 410). In some examples, for instance, the update 450 can improve the accuracy of the ML model(s) 425 in generating output(s) (e.g., the indication 430) by reducing pose error in the output(s).
[0063] The ML model(s) 425 can include, for instance, one or more neural network(s) (NN(s)), one or more convolutional NN(s) (CNN(s)), one or more time delay NN(s) (TDNN(s)), one or more deep network(s) (DN(s)), one or more autoencoder(s) (AE(s)), one or more variational autoencoder(s) (VAE(s)), one or more deep belief net(s) (DBN(s)), one or more recurrent NN(s) (RNN(s)), one or more generative adversarial network(s) (GAN(s)), one or more conditional GAN(s) (cGAN(s)), one or more feed-forward network(s), one or more direct pose regression networks, one or more deep learning networks, one or more network(s) having fully connected layers, one or more support vector machine(s) (SVM(s)), one or more random forest(s) (RF), one or more computer vision (CV) system(s), one or more autoregressive (AR) model(s), one or more Sequence-to- Sequence (Seq2Seq) model(s), one or more large language model(s) (LLM(s)), one or more deep learning system(s), one or more classifier(s), one or more transformer(s), or a combination thereof. In some examples, the ML model(s) 425 include a temporal model that processes sequences of IMU measurements. In some examples, the ML model(s) 425 include a model trained to estimate bias parameters associated with IMU sensors. In some examples, the ML model(s) 425 include a hybrid model that combines learned outputs with physics-based integration.
[0064] In some examples, the ML model(s) 425 can include a U-Network (U-Net) structure and / or architecture that includes a contracting path and an expansive path. If the ML model(s) 425 is a U-Net, the ML model(s) 425 may include, for instance, a combination of convolution, up-convolution, pooling and skip connections that allows the ML model(s) 425 to extract and capture complex features, while also keeping and reconstructing spatial information.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO21
[0065] Within FIG. 4, a graphic representing the ML model(s) 425 illustrates a set of circles connected to one another. Each of the circles can represent a node, a neuron, a perceptron, a layer, a portion thereof, or a combination thereof. The circles are arranged in columns. The leftmost column of white circles represent an input layer. The rightmost column of white circles represents an output layer. Two columns of shaded circled between the leftmost column of white circles and the rightmost column of white circles each represent hidden layers. An ML model can include more or fewer hidden layers than the two illustrated, but includes at least one hidden layer. In some examples, the layers and / or nodes represent interconnected fdters, and information associated with the filters is shared among the different layers with each layer retaining information as the information is processed. The lines between nodes can represent node-to-node interconnections along which information is shared. The lines between nodes can also represent weights (e.g., numeric weights) between nodes, which can be tuned, updated, added, and / or removed as the ML model(s) 425 are trained and / or updated. In some cases, certain nodes (e.g., nodes of a hidden layer) can transform the information of each input node by applying activation functions (e.g., filters) to this information, for instance applying convolutional functions, downscaling, upscaling, data transformation, and / or any other suitable functions.
[0066] In some examples, the ML model(s) 425 can include an ensemble of multiple ML models, and the ML engine 420 can curate and manage the ML model(s) 425 in the ensemble. The ensemble can include ML model(s) 425 that are different from one another to produce different respective outputs, which the ML engine 420 can average (e.g., mean, median, and / or mode) to identify the output(s). In some examples, the ML engine 420 can calculate the standard deviation of the respective outputs of the different ML model(s) 425 in the ensemble to identify a level of confidence in the output(s) (e.g., the indication 430 of the predicted pose 435). The confidence is derived based on variation among outputs of the ensemble, statistical dispersion, model uncertainty estimation, or combinations thereof. In some examples, the confidence is based on a predicted error value output by the ML model(s) 425. In some examples, the confidence is based on historical prediction accuracy. In some examples, the confidence is based on sensor quality metrics. In some examples, the standard deviation can have an inverse relationship with confidence. For instance, if the respective outputs of the different ML model(s) 425 are very different from one another (and thus have a high standard deviation above a threshold), thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO1confidence that the output(s) are accurate may be low (e.g., below a threshold). On the other hand, if the respective outputs of the different ML model(s) 425 are equal or very similar to one another (and thus have a low standard deviation below a threshold), the confidence that the output(s) are accurate may be high (e.g., above a threshold).
[0067] In some examples, different ML models(s) 425 in the ensemble can include different types of models. For instance, in some examples, an ensemble can include a NN and a SVM that are both trained to process the input(s) (e.g., the IMU measurements 405, the image frames 410) to generate at least a subset of the output(s). In some examples, the ensemble may include different ML model(s) 425 that are trained to process different inputs of the input(s) and / or to generate different outputs of the output(s). For instance, in some examples, a first model (or set of models) can process the IMU measurements 405, while a second model (or set of models) can process the image frames 410. In some examples, the ML engine 420 can choose specific ML model(s) 425 to be included in the ensemble because the chosen ML model(s) 425 are effective at accurately processing particular types of input(s), are effective at accurately generating particular types of output(s), are generally accurate, process input(s) quickly, generate output(s) quickly, are computationally efficient, have higher or lower degrees of uncertainty than other models in the ensemble, or a combination thereof.
[0068] In some examples, one or more of the ML model(s) 425 can be initialized with weights, connections, and / or hyperparameters that are selected randomly. This can be referred to as random initialization. These weights, connections, and / or hyperparameters are modified over time through training, for instance through initial training with training data and / or an update 450, but the random initialization can still influence the way the ML model(s) 425 process data, and thus can still cause different ML model(s) 425 (with different random initializations) to produce different output(s). Thus, in some examples, different ML model(s) 425 in an ensemble can have different random initializations.
[0069] As an ML model (of the ML model(s) 425) is trained (e.g., along the initial training with the training data, the update 450, and / or other modification(s)), different versions of the ML model at different stages of training can be referred to as checkpoints. In some examples, after each new update to a model (e.g., update 450) generates a new checkpoint for the model, the ML engine 420 tests the new checkpoint (e.g., against testing data and / or validation data where thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO23correct output(s) are known) to identify whether the new checkpoint improves over older checkpoints or not, and / or if the new checkpoint introduces new errors (e.g., false positive(s) and / or false negative(s)). This testing can be referred to as checkpoint benchmark scoring. In some examples, in checkpoint benchmark scoring, the ML engine 420 produces a benchmark score for one or more checkpoint(s) of one or more ML model(s) 425, and keeps the checkpoint(s) that have the best (e.g., highest or lowest) benchmark scores in the ensemble. In some examples, if a new checkpoint is worse than an older checkpoint, the ML engine 420 can revert to the older checkpoint. The benchmark score for a can represent a level of accuracy of the checkpoint and / or number of errors (e.g., false positive or false negative) by the checkpoint during the testing (e.g., against the testing data and / or the validation data). In some examples, an ensemble of the ML model(s) 425 can include multiple checkpoints of the same ML model.
[0070] In some examples, the ML model(s) 425 can be modified, either through the initial training, an update 450, or another modification to introduce randomness, variability, and / or uncertainty into an ensemble of the ML model(s) 425. In some examples, such modification(s) to the ML model(s) 425 can include dropout (e.g., Monte Carlo dropout), in which one or more weights or connections are selected at random and removed. In some examples, dropout can also be performed during inference, for instance to modify the output(s) generated by the ML model(s) 425. The term Bayesian Machine Learning (BML) can refer to random dropout, random initialization, and / or other randomization-based modifications to the ML model(s) 425. In some examples, the modification(s) to the ML model(s) 425 can include a hyperparameter search and / or adjustment of hyperparameters. The hyperparameter search can involve training and / or updating different ML models 425 with different values for hyperparameters and evaluating the relative performance of the ML models 425 (e.g., against testing data and / or validation data where the correct output(s) are known) to identify which of the ML models 425 performs best. Hyperparameters can include, for instance, temperature (e.g., influencing the level of creativity and / or randomness), top P (e.g., influencing the level of creativity and / or randomness), frequency penalty (e.g., to prevent repetitive language between one of the output(s) and another), presence penalty (e g., to encourage the ML model(s) 425 to introduce new data in the output(s)), other parameters or settings, or a combination thereof.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO24
[0071] FIG. 5 is a block diagram illustrating a simultaneous localization and mapping system (SLAM system 500) that processes IMU measurements 405 and / or image frames 410 using machine learning model(s) (ML model(s) 425) to track features, manage a map of an environment that the system is in, and locate (or relocate) the system within the map. The SLAM system 500 includes the ML engine 420 and the ML model(s) 425 of the imaging system 400. Like in the imaging system 400, the ML model(s) 425 processes input(s) (e.g., including the IMU measurements 405 and / or the image frames 410) to generate the indication 430 of the predicted pose 435 associated with the prediction time window 320.
[0072] The SLAM system 500 includes a tracker 505 and a mapper 510. The tracker 505 includes a pose and feature predictor 515, a feature tracker 520, and a pose and feature estimator 525. The mapper 510 includes a relocator 530, a map management engine 535, and a bundle adjustment engine 540. Within the context of the SLAM system 500, the output(s) of the ML model(s) 425 (e.g., the indication 430 of the predicted pose 435 associated with the prediction time window 320) is fed into one or more elements of the tracker 505 (e.g., into the pose and feature predictor 515, the feature tracker 520, and / or the pose and feature estimator 525), and / or into one or more elements of the mapper 510 (e.g., into the relocator 530, the map management engine 535, and / or the bundle adjustment engine 540).
[0073] The pose and feature predictor 515 predicts a pose of the SLAM system 500 (e.g., a next pose and / or a pose in a prediction time window 320) and expected feature positions based on previous states. In some examples, the pose and feature predictor 515 provides an initial guess to guide subsequent tracking and estimation. In some examples, this prediction includes, or is based on the result(s) of the ML model(s) 425 (e.g., the indication 430 of the predicted pose 435 associated with the prediction time window 320). In some examples, the prediction (of the calculation(s) 415) is based on motion models and / or inertial data to enhance robustness in fast motion scenarios. The pose and feature predictor 515 predicts a pose of the SLAM system 500 and, in some examples, predicted positions and / or orientations of features within the environment based on previous states and the output(s) of the ML model (s) 425.
[0074] In some examples, the feature tracker 520 detects and / or extracts feature(s) (e.g., corners, edges, blobs) from the image frames 410, recognizes and / or tracks the features acrossPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO25multiple image frames 410, and / or correlates the extracted features to the pose and / or feature predictions from the pose and feature predictor 515. In some examples, the pose and feature estimator 525 ensures continuity of features over time.
[0075] In some examples, the pose and feature estimator 525 refines the pose (of the SLAM system 500) and / or the 3D feature positions based on a combination of the tracked feature data (from the feature tracker 520) and the predicted pose and feature data (from the pose and feature predictor 515). In some examples, the pose and feature estimator 525 can thus update predictions of the pose and feature predictor 515 and improve accuracy beyond initial predictions of the ML model(s) 425 and / or the pose and feature predictor 515.
[0076] In some examples, the relocator 530 re-estimates the system's pose in the map when tracking is lost or uncertain, enabling recovery. In some examples, the relocator 530 uses existing map data, maintained by the map management engine 535, to find the best match for the current view. In some examples, the relocator 530 discards or disregards existing map data to generate a new map based on the features and / or pose data (e.g., from the IMU measurements 405, the image frames 410, the ML model(s) 425, the pose and feature predictor 515, the feature tracker 520, and / or the pose and feature estimator 525). In some examples, the relocator 530 then matches features (e.g., corners, edges, blobs) from the new map to features from the old (existing) map to help improve the new map, the old map, and / or the pose of the SLAM system 500. In some examples, the relocator 530 also performs loop closure detection to correct drift. In some examples, the relocator 530 replaces or modifies a previously-determined pose during relocalization and / or relocation. In some examples, the relocator 530 replaces or modifies a previously-determined map during relocalization and / or relocation.
[0077] In some examples, the map management engine 535 maintains and updates a map of the environment. In some examples, the map can be a global map of an entirety of the environment. In some examples, the map can be split, divided, and / or segmented (e.g., by the map management engine 535) into map segments or map slices, to reduce how much of the map is to be analyzed and / or updated at a given time. For instance, in some examples, the map management engine 535 can segment the map so that different rooms of a structure (e.g., a home, an office, a building) correspond to different map segments or map slices, so that indoor areas and outdoor areasPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO26correspond to different map segments or map slices, so that large rooms are segmented into different map segments or map slices, or a combination thereof. In some examples, the map management engine 535 stores feature positions and / or keyframes associated with the maps, the map segments, and / or the map slices. In some examples, the map management engine 535 manages which features to keep or discard. In some examples, map segmentation is based on spatial regions. In some examples, map segmentation is based on feature density. In some examples, map segmentation is based on semantic regions within an environment.
[0078] In some examples, the bundle adjustment engine 540 performs global optimization of poses and 3D feature positions across multiple frames, improving overall consistency. In some examples, the bundle adjustment engine 540 takes data from the map management engine 535 and adjusts both the map and pose estimates. In some examples, bundle adjustment runs in the background to reduce latency in the main SLAM loop.
[0079] In some examples, the SLAM system 500 is an extended reality (XR) device, such as a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or a combination thereof. In some examples, the SLAM system 500 tracks movement(s) in six-degrees-of-freedom (6DoF), such as three-dimensional position (e.g., X, Y, Z) and orientation (e.g., pitch, yaw, and roll). In some examples, the SLAM system 500 takes image frames 410 and IMU measurements 405 as inputs (to the ML model(s) 425) to initially estimate a pose. The estimated poses are stored in a history. Multiple estimated poses and 3D feature positions are retrieved from a history and are used to build and continuously update the map as and when a new pose and map arrive. To get a more accurate pose in this map, the system performs bundle adjustment and relocalization as needed.
[0080] In some examples, if the IMU measurements 405 from the IMU and / or the predicted pose indicate (e.g., based on an analysis by the ML model(s) 425 and / or other elements of the tracker 505) that the pose has drifted significantly (e.g., by more than a threshold), the SLAM system 500 can wait before running the relocator 530, and can instead run the relocator 530 when the pose is once again closer to being accurate (e.g., when the drift is once again below the threshold). If the IMU measurements and / or the predicted pose indicate a displacement of the device exceeding a threshold, the SLAM system 500 can suppress execution of the relocator 530,Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO27and can activate the relocator 530 when the displacement is below the threshold indicating a likelihood of a map connection. These operations can save power and computational resources by selectively running the relocator 530. In some examples, the SLAM system 500 uses the predicted pose generated by the ML model(s) 425 to control activation of the relocator 530. The SLAM system 500 suppresses execution of the relocator 530 if the predicted pose indicates that a current pose estimate is unlikely to match an existing map within a threshold condition, and activates the relocator 530 if the predicted pose indicates a likelihood of successful relocalization. This selective activation reduces computational load, reduces power consumption, and improves efficiency of processing resources in the SLAM system 500.
[0081] In some examples, in a setup with a larger map, the mapper 510 can split the map into smaller manageable components, referred to as map segments or map slices. The SLAM system 500 can use a high-quality pose prediction generated by the ML model(s) 425 (based on the IMU measurements 405) to dynamically identify and select the appropriate map component for relocation based on the improved pose. This approach reduces time to relocalization, power usage, computational resource usage, and heat generation (and thus need for cooling) for localization, and / or mapping. This approach also improves throughput and efficiency for localization, relocalization, and / or mapping.
[0082] In some examples, in a setup with a larger map, the SLAM system 500 can identify the regions for feature matching without the need for splitting the map into smaller components. The SLAM system 500 can use a high-quality pose prediction generated by the ML model(s) 425 (based on the IMU measurements 405) to determine (e.g., localize) where to search (e.g., look) for the matching feature in the map, in some cases including restricting feature matching to a subset of map regions corresponding to the predicted pose and / or excluding regions outside a spatial threshold of the predicted pose. This reduces a number of feature comparisons and accelerates relocalization. This approach reduces time, power usage, computational resource usage, and heat generation (and thus need for cooling) for localization, relocalization, and / or mapping. This approach also improves throughput and efficiency for localization, relocalization, and / or mapping.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO28
[0083] In some examples, the SLAM system 500 can use fixed relocalization thresholds to identify when to trigger relocalization using the relocator 530 (e.g., when the relocalization thresholds are crossed). For instance, if the ML model(s) 425 indicates that its confidence level associated with its output (e.g., of the indication 430 of the predicted pose 435 associated with the prediction time window 320) is below the relocalization threshold, the SLAM system 500 can initiate relocalization using the relocator 530. If the ML model(s) 425 indicates that its confidence level associated with its output (e.g., of the indication 430 of the predicted pose 435 associated with the prediction time window 320) is above the relocalization threshold, the SLAM system 500 can proceed with its existing map (without initiating relocalization using the relocator 530). In some examples, relocalization is triggered based on a combination of confidence, predicted pose drift, and feature availability. In some examples, relocalization is deferred until a predicted pose satisfies a spatial proximity condition relative to a map region. In some examples, fixed relocalization thresholds work well for high-feature images, but can fail in low-feature images, such as images where a large portion of the image is occupied by a wall, a cloudless sky, or another area with a low amount of visual features (e.g., less than a threshold number of corners, edges, and / or blobs). To improve the performance of the SLAM system 500, the threshold can be adapted and / or adjusted dynamically based on quantity of image features (e g., as determined by the feature tracker 520 and / or the ML model(s) 425), and / or based on uncertainty estimates (or confidence levels) generated bythe ML model(s) 425. If the quantity of image features is low (e.g., below a threshold), and / or the network's uncertainty estimate is low (e.g., below a threshold) (or confidence level is high, exceeding a threshold), a smaller relocalization threshold can be used. If the quantity of image features is high (e g., exceeding a threshold), and / or the network's uncertainty estimate is high (e.g., exceeding a threshold) (or confidence level is low, below a confidence threshold), a normal or heightened relocalization threshold can be used. This approach improves the accuracy of relocalization, and saves power and computational resources by avoiding running of relocalization unnecessarily. For instance, a SLAM system 500 with a fixed relocalization threshold might unnecessarily run relocalization simply because the SLAM system 500 is facing a wall, a cloudless sky, or another area with a low amount of visual features (e.g., less than a threshold number of corners, edges, and / or blobs), even though the pose and / or map of the SLAM system 500 are still accurate. A SLAM system 500 with a dynamic relocalizationPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO29threshold knows when it is facing a wall, a cloudless sky, or another area with a low amount of visual features (e.g., less than a threshold number of corners, edges, and / or blobs), can reduce its relocalization threshold accordingly, and can thus prevent itself from unnecessarily running relocalization when the pose and / or map of the SLAM system 500 are still accurate.
[0084] FIG. 6 is a block diagram illustrating an example of a neural network 600 that can be used for imaging operations. The neural network 600 can include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief net (DBN), a Recurrent Neural Network (RNN), a Generative Adversarial Network (GAN), an auto-regressive transformer model, and / or other type of neural network. The neural network 600 may be, and / or may include, an example of the ML model(s) 425, ML model(s) of the imaging system 400, the ML model(s) 925, or a combination thereof.
[0085] An input layer 610 of the neural network 600 includes input data. The input data of the input layer 610 can include images captured using the image capture and processing system 100, IMU measurements 340, image frames 345, IMU measurements 405, image frames 410, the first image 705, the second image 710, images from any of the cameras 930A-930D, images from any of the cameras 1030A-1030D, images from any of the cameras 1130A-1130F, images from any of the sensors 1140A-1140B, images from the camera 1230, any other IMU measurements or IMU data discussed herein, any other images discussed herein, any other sensor data discussed herein, depth data such as point clouds, other types of inputs discussed herein, or a combination thereof.
[0086] The neural network 600 includes multiple hidden layers 612, 612B, through 612N. The hidden layers 612, 612B, through 612N include “N” number of hidden layers, where “N” is an integer greater than or equal to one. The number of hidden layers can be made to include as many layers as needed for the given application. The neural network 600 further includes an output layer 614 that provides an output resulting from the processing performed by the hidden layers 612, 612B, through 612N.
[0087] In some examples, the output layer 614 can provide output data. The output data can include the indication 430 of the predicted pose 435 associated with the prediction time windowPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO30320, the predicted pose 435 itself, another predicted pose, a pose delta, other types of outputs discussed herein, or a combination thereof.
[0088] The neural network 600 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn a feature representative of the input data. Information associated with the filters is shared among the different layers and each layer retains information as information is processed. In some cases, the neural network 600 can include a feed-forward network, in which case there are no feedback connections where outputs of the network are fed back into itself. In some cases, the network 600 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading in input.
[0089] In some cases, information can be exchanged between the layers through node-to-node interconnections between the various layers. In some cases, the network can include a convolutional neural network, which may not link every node in one layer to every other node in the next layer. In networks where information is exchanged between layers, nodes of the input layer 610 can activate a set of nodes in the first hidden layer 612A. For example, as shown, each of the input nodes of the input layer 610 can be connected to each of the nodes of the first hidden layer 612A. The nodes of a hidden layer can transform the information of each input node by applying activation functions (e.g., filters) to this information. The information derived from the transformation can then be passed to and can activate the nodes of the next hidden layer 612B, which can perform their own designated functions. Example functions include convolutional functions, downscaling, upscaling, data transformation, and / or any other suitable functions. The output of the hidden layer 612B can then activate nodes of the next hidden layer, and so on. The output of the last hidden layer 612N can activate one or more nodes of the output layer 614, which provides a processed output image. In some cases, while nodes (e.g., node 616) in the neural network 600 are shown as having multiple output lines, a node has a single output and all lines shown as being output from a node represent the same output value.
[0090] In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network 600. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable numeric weight that can be tunedPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO31(e.g., based on a training dataset), allowing the neural network 600 to be adaptive to inputs and able to learn as more and more data is processed.
[0091] In some aspects, training of one or more of the machine learning systems or neural networks described herein can be performed using online training (e.g., in some case on-device training), offline training, and / or various combinations of online and offline training. In some cases, online may refer to time periods during which the input data (e.g., such as the input data discussed with respect to the input layer 610) is processed, for instance for generating output data (e.g., such as the input data discussed with respect to the output layer 614). In some examples, offline may refer to idle time periods or time periods during which input data is not being processed. Additionally, offline may be based on one or more time conditions (e.g., after a particular amount of time has expired, such as a day, a week, a month, etc.) and / or may be based on various other conditions such as network and / or server availability, etc., among various others. In some aspects, offline training of a machine learning model (e.g., a neural network model) can be performed by a first device (e.g., a server device) to generate a pre-trained model, and a second device can receive the trained model from the second device. In some cases, the second device (e.g., a mobile device, an XR device, a vehicle or system / component of the vehicle, or other device) can perform online (or on-device) training of the pre-trained model to further adapt or tune the parameters of the model.
[0092] The neural network 600 is pre-trained to process the features from the data in the input layer 610 using the different hidden layers 612, 1012B, through 1012N in order to provide the output through the output layer 614.
[0093] FIG. 7 is a conceptual diagram 700 illustrating matching of points between a first image 705 and a second image 710. A legend 730 identifies that circles in the first image 705 indicate matching points 740 in the first image 705, while plus symbols (+) in the second image 710 indicate matching points 745 in the second image 710. For instance, features corresponding to a keypad 750 in the first image 710 are matched to features corresponding to the keypad 750 in the second image 710. Features corresponding to a sensor 755 in the first image 710 are matched to features corresponding to the sensor 755 in the second image 710. Features corresponding to a fire alarm 760 in the first image 710 are matched to features corresponding to the fire alarm 760 in the second image 710.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO32
[0094] As noted above with respect to FIG. 5, in some examples, the SLAM system 500 can use fixed relocalization thresholds or dynamic relocalization thresholds to identify when to trigger relocalization using the relocator 530 (e.g., when the relocalization thresholds are crossed). For instance, if the ML model(s) 425 indicates that its confidence level associated with its output (e.g., of the indication 430 of the predicted pose 435 associated with the prediction time window 320) is below a relocalization threshold, the SLAM system 500 can initiate relocalization using the relocator 530.
[0095] In some examples, the SLAM system 500 triggers relocalization by the relocator 530 if the a total number feature matches (such as those illustrated in FIG. 7) exceeds the relocalization threshold. A fixed relocalization threshold works well for images of environments that include many visual features (e.g., edges, corners, blobs). However, significant portions of the first image 705 and the second image 710 are taken up by a (generally featureless) wall, meaning that the first image 705 and the second image 710 may not have very many visual features (e.g., edges, corners, blobs). For instance, as only three features are matched between the first image 705 and the second image 710 (e.g., the keypad 750, the sensor 755, and the fire alarm 760). If the threshold is greater than three, this may inadvertently cause the SLAM system 500 to cross the relocalization threshold and trigger relocalization, even if the pose and / or map are still accurate, simply because there are few features to track and / or match to begin with. To address this, the SLAM system 500 dynamically adapts and / or adjusts (e.g., reduces) the relocalization threshold based on the number of matchable features in the image(s). In some examples, the SLAM system 500 can use an uncertainty estimate from the ML model(s) 425 to determine the relocalization threshold, to be dynamic and / or adaptive.
[0096] FIG. 8 is a graph diagram 800 illustrating reduction in frequency 820 of relative translation error in relocalization by a system 840 with the ML engine 420 (with the ML model(s) 425) relative to a system 830 without the ML engine 420 (without the ML model(s) 425). A horizontal axis of the graph diagram 800 tracks relative translation error in millimeters (mm) 810 over the course of one second. A vertical axis of the graph diagram 800 tracks frequency 820 of relative translation error.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO33
[0097] The graph diagram 800 shows that the frequency of relative translation error is significantly lower in the system 840 with the ML engine 420 (with the ML model(s) 425) relative to the system 830 without the ML engine (without the ML model(s) 425). This shows that the ML engine 420 (e.g., the ML model(s) 425) provides a significant reduction in error (of approximately 50%), and therefore a significant improvement in accuracy, of pose estimation, localization, and / or relocalization.
[0098] In the illustrative example graphed in the graph diagram 800, the ML model(s) 425 (in the system 840) are direct pose regression networks that predict the pose delta (e.g., the indication 430 of the predicted pose 435) associated with the prediction time window 320, without the integral calculation(s) 415. In some examples, the ML model(s) 425 (in the system 840) can be deep learning networks and / or other types of neural networks.
[0099] In some examples, the imaging system 400, the SLAM system 500, and / or the system 840 implement denoising and debiasing techniques to reduce dead reckoning pose error. In some examples, the imaging system 400, the SLAM system 500, and / or the system 840 utilize IMU-based pose estimation to dynamically switch on the relocator, thereby reducing power consumption. In some examples, the imaging system 400, the SLAM system 500, and / or the system 840 improve relocation in low feature areas by reducing the relocation threshold in low feature areas. In some examples, the imaging system 400, the SLAM system 500, and / or the system 840 can employ a deep-learning-based IMU-Corrector model that improves pose quality compared to employing IMU data directly.
[0100] FIG. 9A is a perspective diagram 900 illustrating a head-mounted display (HMD) 910 that is used as part of an imaging system (e.g., image capture and processing system 100, imaging system 400, SLAM system 500). The HMD 910 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 910 includes a first camera 930A and a second camera 930B along a front portion of the HMD 910. The HMD 910 includes thethird camera 930C and the fourth camera 930D facing the eye(s) of the user as the eye(s) of the user face the display(s) 940. In some examples, the HMD 910 may only have a single camera with a single image sensor. In some examples, the HMD 910 may include one or more additional cameras inPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO34addition to the first camera 930A, the second camera 930B, third camera 930C, and the fourth camera 930D. In some examples, the HMD 910 may include one or more additional sensors in addition to the first camera 930A, the second camera 930B, third camera 930C, and the fourth camera 930D. In some examples, the first camera 930A, the second camera 930B, third camera 930C, and / or the fourth camera 930D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, another camera or sensor discussed herein, or a combination thereof.
[0101] The HMD 910 may include one or more displays 940 that are visible to a user 920 wearing the HMD 910 on the user 920’s head. In some examples, the HMD 910 may include one display 940 and two viewfinders. The two viewfinders can include a left viewfinder for the user 920’s left eye and a right viewfinder for the user 920’s right eye. The left viewfinder can be oriented so that the left eye of the user 920 sees a left side of the display. The right viewfinder can be oriented so that the right eye of the user 920 sees a right side of the display. In some examples, the HMD 910 may include two displays 940, including a left display that displays content to the user 920’s left eye and a right display that displays content to a user 920’s right eye. The one or more displays 940 of the HMD 910 can be digital “pass-through” displays or optical “see-through” displays.
[0102] The HMD 910 may include one or more earpieces 935, which may function as speakers and / or headphones that output audio to one or more ears of a user of the HMD 910. One earpiece 935 is illustrated in FIGs. 9A and 9B, but it should be understood that the HMD 910 can include two earpieces, with one earpiece for each ear (left ear and right ear) of the user. In some examples, the HMD 910 can also include one or more microphones (not pictured). In some examples, the audio output by the HMD 910 to the user through the one or more earpieces 935 may include, or be based on, audio recorded using the one or more microphones.
[0103] FIG. 9B is a perspective diagram 950 illustrating the head-mounted display (HMD) of FIG. 9A being worn by a user 920. The user 920 wears the HMD 910 on the user 920’s head over the user 920’s eyes. The HMD 910 can capture images with the first camera 930A and the second camera 930B. In some examples, the HMD 910 displays one or more output images toward the user 920’s eyes using the display(s) 940. In some examples, the output images can includePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO35processed image data (e.g., a 3D occupancy prediction map such as the 3D occupancy prediction map 455). The output images can be based on the images captured by the first camera 930A and the second camera 930B (e.g., the image sensor 130), for example with the processed image data (e.g., a 3D occupancy prediction map such as the 3D occupancy prediction map 455) overlaid. The output images may provide a stereoscopic view of the environment, in some cases with the processed content overlaid and / or with other modifications. For example, the HMD 910 can display a first display image to the user 920’s right eye, the first display image based on an image captured by the first camera 930A. The HMD 910 can display a second display image to the user 920’s left eye, the second display image based on an image captured by the second camera 930B. For instance, the HMD 910 may provide processed content in the display images overlaid over the images captured by the first camera 930A and the second camera 930B. The third camera 930C and the fourth camera 930D can capture images of the eyes of the user before, during, and / or after the user views the display images displayed by the display(s) 940. This way, the sensor data from the third camera 930C and / or the fourth camera 930D can capture reactions to the processed content by the user’s eyes (and / or other portions of the user). An earpiece 935 of the HMD 910 is illustrated in an ear of the user 920. The HMD 910 may be outputting audio to the user 920 through the earpiece 935 and / or through another earpiece (not pictured) of the HMD 910 that is in the other ear (not pictured) of the user 920.
[0104] FIG. 10A is a perspective diagram 1000 illustrating a front surface of a mobile handset 1010 that includes front-facing cameras and can be used as part of an imaging system (e.g., image capture and processing system 100, imaging system 400, SLAM system 500). The mobile handset 1010 may be, for example, a cellular telephone, a satellite phone, a portable gaming console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.
[0105] The front surface 1020 of the mobile handset 1010 includes a display 1040. The front surface 1020 of the mobile handset 1010 includes a first camera 1030A and a second camera 1030B. The first camera 1030A and the second camera 1030B can face the user, including the eye(s) of the user, while processed image data (e.g., associated with pose and / or mapping basedPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO36on output(s) of the ML model(s) 425 that process IMU measurements 405 and / or image frames 410) is displayed on the display 1040.
[0106] The first camera 1030A and the second camera 1030B are illustrated in a bezel around the display 1040 on the front surface 1020 of the mobile handset 1010. In some examples, the first camera 1030A and the second camera 1030B can be positioned in a notch or cutout that is cut out from the display 1040 on the front surface 1020 of the mobile handset 1010. In some examples, the first camera 1030A and the second camera 1030B can be under-display cameras that are positioned between the display 1040 and the rest of the mobile handset 1010, so that light passes through a portion of the display 1040 before reaching the first camera 1030A and the second camera 1030B. The first camera 1030 A and the second camera 1030B of the perspective diagram 1000 are front-facing cameras. The first camera 1030 A and the second camera 1030B face a direction perpendicular to a planar surface of the front surface 1020 of the mobile handset 1010. The first camera 1030 A and the second camera 1030B may be two of the one or more cameras of the mobile handset 1010. In some examples, the front surface 1020 of the mobile handset 1010 may only have a single camera.
[0107] In some examples, the display 1040 of the mobile handset 1010 displays one or more output images toward the user using the mobile handset 1010. In some examples, the output images can include the processed image data (e g., a 3D occupancy prediction map such as the 3D occupancy prediction map 455). The output images can be based on the images (e.g., captured by image sensor 130) captured by the first camera 1030A, the second camera 1030B, the third camera 1030C, and / or the fourth camera 1030D, for example with the processed image data (e.g., associated with pose and / or mapping based on output of the ML model(s) 425 that processes IMU measurements 405 and / or image frames 410) overlaid.
[0108] In some examples, the front surface 1020 of the mobile handset 1010 may include one or more additional cameras in addition to the first camera 1030A and the second camera 1030B. In some examples, the front surface 1020 of the mobile handset 1010 may include one or more additional sensors in addition to the first camera 1030A and the second camera 1030B. In some cases, the front surface 1020 of the mobile handset 1010 includes more than one display 1040. For example, the one or more displays 1040 can include one or more touchscreen displays.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO37
[0109] The mobile handset 1010 may include one or more speakers 1035 A and / or other audio output devices (e.g., earphones or headphones or connectors thereto), which can output audio to one or more ears of a user of the mobile handset 1010. One speaker 1035Ais illustrated in FIG. IOA, but it should be understood that the mobile handset 1010 can include more than one speaker and / or other audio device. In some examples, the mobile handset 1010 can also include one or more microphones (not pictured). In some examples, the audio output by the mobile handset 1010 to the user through the one or more speakers 1035A and / or other audio output devices may include, or be based on, audio recorded using the one or more microphones.
[0110] FIG. 10B is a perspective diagram 1050 illustrating a rear surface 1060 of a mobile handset that includes rear-facing cameras and that can be used as part of a sensor data processing system. The mobile handset 1010 includes a third camera 1030C and a fourth camera 1030D on the rear surface 1060 of the mobile handset 1010. The third camera 1030C and the fourth camera 1030D of the perspective diagram 1050 are rear-facing. The third camera 1030C and the fourth camera 1030D face a direction perpendicular to a planar surface of the rear surface 1060 of the mobile handset 1010.[OHl] The third camera 1030C and the fourth camera 1030D may be two of the one or more cameras of the mobile handset 1010. In some examples, the rear surface 1060 of the mobile handset 1010 may only have a single camera. In some examples, the rear surface 1060 of the mobile handset 1010 may include one or more additional cameras in addition to the third camera 1030C and the fourth camera 1030D. In some examples, the rear surface 1060 of the mobile handset 1010 may include one or more additional sensors in addition to the third camera 1030C and the fourth camera 1030D. In some examples, the first camera 1030A, the second camera 1030B, the third camera 1030C, and / or the fourth camera 1030D may be examples of the image capture and processing system 100, the image capture device 105 A, the image processing device 105B, another camera or sensor discussed herein, or a combination thereof.
[0112] The mobile handset 1010 may include one or more speakers 1035B and / or other audio output devices (e.g., earphones or headphones or connectors thereto), which can output audio to one or more ears of a user of the mobile handset 1010. One speaker 1035B is illustrated in FIG. IOB, but it should be understood that the mobile handset 1010 can include more than one speakerPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO38and / or other audio device. In some examples, the mobile handset 1010 can also include one or more microphones (not pictured). In some examples, the mobile handset 1010 can include one or more microphones along and / or adjacent to the rear surface 1060 of the mobile handset 1010. In some examples, the audio output by the mobile handset 1010 to the user through the one or more speakers 1O35B and / or other audio output devices may include, or be based on, audio recorded using the one or more microphones.[01131 The mobile handset 1010 may use the display 1040 on the front surface 1020 as a pass-through display. For instance, the display 1040 may display output images, such as processed image data (e.g., associated with pose and / or mapping based on output of the ML model(s) 425 that processes IMU measurements 405 and / or image frames 410). The output images can be based on the images (e.g., from the image sensor 130) captured by the third camera 1030C and / or the fourth camera 1030D, for example with the processed image data (e.g., associated with pose and / or mapping based on output of the ML model(s) 425 that processes IMU measurements 405 and / or image frames 410) overlaid. The first camera 1030A and / or the second camera 1030B can capture images of the user’s eyes (and / or other portions of the user) before, during, and / or after the display of the output images with the processed content on the display 1040. This way, the sensor data from the first camera 1030A and / or the second camera 1030B can capture reactions to the processed content by the user’s eyes (and / or other portions of the user).
[0114] FIG. 11 is a perspective diagram 1100 illustrating a vehicle 1110 that includes various sensors and that can be used as part of an imaging system (e.g., image capture and processing system 100, imaging system 400, SLAM system 500). The vehicle 1110 may be, for example, an automobile, a truck, a bus, a train, a ground-based vehicle, an airplane, a helicopter, an aircraft, an aerial vehicle, a boat, a submarine, a watercraft, an underwater vehicle, a hovercraft, or a combination thereof. In some examples, the vehicle 1110 may be manned. In some examples, the vehicle 1110 maybe at least partially controlled and / or used with sub-systems of the vehicle 1110, such as an Advanced Driver Assistance System (ADAS) of the vehicle 1110, In-Vehicle Infotainment (IVI) systems of the vehicle 1110, driving assistance systems of the vehicle 1110, a vehicle electronic control unit (ECU) 930 of the vehicle 1110, or a combination thereof.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO39
[0115] The vehicle 1110 includes a display 1120. The vehicle 1110 includes various sensors, all of which can be examples of the sensor(s) 205. The vehicle 1110 includes a first camera 1130 A and a second camera 1130B at the front, a third camera 1130C and a fourth camera 1130D at the rear, and a fifth camera 1130E and a sixth camera 1130F on the top. The vehicle 1110 includes a first microphone 1135A at the front, a second microphone 1135B at the rear, and a third microphone 1135C at the top. The vehicle 1110 includes a first sensor 1140A on one side (e.g., adjacent to one rear-view mirror) and a second sensor 1140B on another side (e.g., adjacent to another rear-view mirror). The first sensor 1140A and the second sensor 1140B may include cameras, microphones, RADAR sensors, LIDAR sensors, or any other types of sensor(s) 205 described herein. In some examples, the vehicle 1110 may include additional sensor(s) 205 in addition to the sensors illustrated in FIG. 11. In some examples, the vehicle 1110 may be missing some of the sensors that are illustrated in FIG. 11.
[0116] In some examples, the display 1120 of the vehicle 1110 displays one or more output images toward a user of the vehicle 1110 (e.g., a driver and / or one or more passengers of the vehicle 1110). In some examples, the output images can be associated with pose and / or mapping based on output of the ML model(s) 425 that processes IMU measurements 405 and / or image frames 410. The output images can be based on the images captured by the first camera 1130A, the second camera 1130B, the third camera 1130C, the fourth camera 1130D, the fifth camera 1130E, the sixth camera 1130F, the first sensor 1140A, and / or the second sensor 1140B, for example with the virtual content (e.g., associated with pose and / or mapping based on output of the ML model(s) 425 that processes IMU measurements 405 and / or image frames 410) overlaid.
[0117] FIG. 12Ais a perspective diagram 1200 illustrating an unmanned ground vehicle (UGV) 1210 that can be used as part of an imaging system (e.g., image capture and processing system 100, imaging system 400, SLAM system 500). The UGV 1210 illustrated in the perspective diagram 1200 of FIG. 12Amay be an example of an imaging system. The UGV 1210 includes a camera 1230 along a front surface of the UGV 1210. In some examples, the UGV 1210 may include one or more additional cameras in addition to the camera 1230. In some examples, the UGV 1210 may include one or more additional sensors in addition to the camera 1230. The UGV 1210 includes multiple wheels 1215 along a bottom surface of the UGV 1210. The wheels 1215 may act as a conveyance of the UGV 1210, and may be motorized using one or more motors thatPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO40may be actuated by a movement actuator of the UGV 1210. The movement actuator, the motors, and thus the wheels 1215, may be actuated to move the UGV 1210 along a path.[01181 FIG. 12B is a perspective diagram 1250 illustrating an unmanned aerial vehicle (UAV) 1220 that can be used as part of an imaging system (e.g., image capture and processing system 100, imaging system 400, SLAM system 500). The UAV 1220 illustrated in the perspective diagram 1250 of FIG. 12B may be an example of an imaging system. The UAV 1220 includes a camera 1230 along a front portion of a body of the UAV 1220. In some examples, the UAV 1220 may include one or more additional cameras in addition to the camera 1230. In some examples, the UAV 1220 may include one or more additional sensors in addition to the camera 1230. The UAV 1220 includes multiple propellers 1225 along the top of the UAV 1220. The propellers 1225 may be spaced apart from the body of the UAV 1220 by one or more appendages to prevent the propellers 1225 from snagging on circuitry on the body of the UAV 1220 and / or to prevent the propellers 1225 from occluding the view of the camera 1230. The propellers 1225 may act as a conveyance of the UAV 1220, and may be motorized using one or more motors that may be actuated by a movement actuator of the UAV 1220. The movement actuator, the motors, and thus the propellers 1225, may be actuated to move the UAV 1220 along a path.
[0119] Where the imaging system is a vehicle, such as the vehicle 1110, the UGV 1210, and / or UAV 1220, the imaging system can include a simultaneous localization and mapping (SLAM) engine (e.g., a visual SLAM (VSLAM) engine), a path or route planning engine, and / or a movement actuator. The SLAM engine can locate the vehicle in the environment and can map the environment, the path planning engine may generate a path or route along which the vehicle is to move, and the movement actuator can actuate wheels, propellers, legs, and / or other conveyance actuators to move the vehicle along the planned path or route. In some examples, the path planning engine may use a Dijkstra algorithm to plan the path. In some examples, the path planning engine may include stationary obstacle avoidance and / or moving obstacle avoidance in planning the path. In some examples, the path planning engine may include determinations as to how to best move the vehicle from a first pose to a second pose in planning the path. In some examples, the path planning engine may plan a path that is optimized to reach and observe every portion of a first region of an environment (e.g., a first set of one or more rooms in the environment) before movingPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO41on to a second region of the environment (e.g., the second set of one or more rooms of the environment) in planning the path. In some examples, the path planning engine may plan a path that is optimized to reach and observe a predetermined set of rooms in an environment (e.g., every room in the environment) as quickly as possible. In some examples, the path planning engine may plan a path that returns to a previously-observed room to observe a particular feature again to improve one or more map points corresponding the feature in the local map and / or global map. In some examples, the path planning engine may plan a path that returns to a previously-ob served room to observe a portion of the previously-observed room that lacks map points in the local map and / or global map to see if any features can be observed in that portion of the room. The movement actuator may actuate one or more motors to actuate a motorized conveyance (e.g., the wheels 1215 or the propellers 1225) to move the vehicle along the path planned by the path planning engine.
[0120] In some cases, the propellers 1225 of the UAV 1220, or another portion of a vehicle (e.g., an antenna), may partially occlude the view of one of the one or more cameras in some images captured by the one or more cameras. In some examples, this partial occlusion may be masked out of any images in which the partial occlusion appears, for example as in a masking operation.
[0121] FIG. 13 is a flow diagram illustrating a process 1300 for localization. The process 1300 may be performed by a localization system. In some examples, the localization system can include, for example, the image capture and processing system 100, the image capture device 105 A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, an IMU that captures the IMU measurements 340, an image sensor that captures the image frames 345, the imaging system 400, an IMU that captures the IMU measurements 405, an image sensor that captures the image frames 410, the ML engine 420, the ML model(s) 425, the SLAM system 500, the tracker 505, the mapper 510, the neural network 600, the system 830, the system 840, the HMD 910, the mobile handset 1010, the vehicle 1110, the UGV 1210, the UAV 1220, the computing system 1400, the processor 1410, an apparatus, a system, a non-transitory computer-readable medium coupled to a processor, or a combination thereof.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO42
[0122] At operation 1305, the localization system (or at least one subsystem thereof) is configured to, and can, process a plurality of IMU measurements (e.g., IMU measurements 340, IMU measurements 405) and an indication of a prediction time (e.g., within or on a boundary of the prediction time window 320) to generate an indication of a predicted pose associated with the prediction time (e.g., indication 430 of the predicted pose 435). In some examples, the predicted pose corresponds to a corrected pose of the device, where the corrected pose is based on IMU measurements that have been processed to reduce drift, noise, bias, or non-linearities associated with the IMU measurements. The indication of the prediction time can be referred to as a time indication. The indication of the predicted pose can be referred to as a pose indication. A context time window (e.g., capture window 310) corresponds to capture of the plurality of IMU measurements by an IMU. In some examples, the prediction time is after the context time window. In some aspects, all of the IMU measurements that are processed in operation 1305 are captured during the capture window. In some aspects, some of the IMU measurements that are processed in operation 1305 are captured during the capture window, and some of the IMU measurements that are processed in operation 1305 are captured during the prediction window. In some examples, the prediction time is during the context time window. In some examples, prior to operation 1305, the localization system (or at least one subsystem thereof) is configured to, and can, receive IMU data from one or more inertial sensors (e.g., one or more IMUs) of a device. The IMU data can include the plurality of IMU measurements.
[0123] Examples of the IMU measurements include the IMU measurements associated with the graph diagram 200, the IMU measurements 340, the IMU measurements 405, the IMU measurements associated with the graph diagram 800, IMU measurements from an IMU of the HMD 910, IMU measurements from an IMU of the mobile handset 1010, IMU measurements from an IMU (e.g., sensors 1140A-1140B) of the vehicle 1110, IMU measurements from an IMU of the UGV 1210, IMU measurements from an IMU of the UAV 1220, IMU measurements from an IMU of the computing system 1400, other IMU data, or a combination thereof. In some examples, the IMU measurements are captured using one or more sensors of the localization system (e.g., one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more gyrometers, one or more magnetometers, one or more pressure sensors, and / or one or more other sensors of the localization system), such as an IMU of the image capture and processingPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO43system 100, an IMU of the imaging system 400, an IMU of the SLAM system 500, an IMU of the HMD 910, an TMU of the mobile handset 1010, an IMU (e.g., sensors 1140A-1140B) of the vehicle 1110, an IMU of the UGV 1210, an IMU of the UAV 1220, an IMU of the computing system 1400, or a combination thereof.
[0124] In some aspects, processing the plurality of IMU measurements and the indication of the prediction time (as in operation 1305) includes processing the plurality of IMU measurements and the indication of the prediction time using a trained machine learning model (e.g., the ML model(s) 425, the neural network 600) that generates the indication of the predicted pose associated with the prediction time. In some aspects, the trained machine learning model is a neural network (e.g., the neural network 600). In some aspects, the trained machine learning model is a deep learning (DL) network, In some aspects, the trained machine learning model is trained (e.g., previously trained before the process 1300) to compensate for drift, noise, bias, and / or non-linearities in the IMU measurements to generate a corrected pose. The training can be based on a training dataset that includes examples of predicted pose data (e.g., some of which may include issues such as drift, noise, bias, and / or non-linearities) at a time (e.g., a future time) as well as observed pose data (e.g., detected pose data using IMU(s) and / or other sensor(s)) at that time (e.g., once that time is reached), so that the trained machine learning model can learn to correct for issues (e.g., drift, noise, bias, and / or non-linearities) based on differences (e.g., deltas) between the predicted pose data and the observed pose data.
[0125] In some aspects, the localization system (or at least one subsystem thereof) is configured to, and can, receive additional sensor data (e.g., IMU measurements 340, IMU measurements 405, image frames 345, image frames 410) captured during the prediction time (e.g., within or on a boundary of prediction time window 320) (e.g., by the IMU and / or the image sensor(s)). In some aspects, generating the indication of the predicted pose (as in operation 1305) is also based on the additional sensor data. In some aspects, the localization system (or at least one subsystem thereof) is configured to, and can, determine a pose (e.g., an actual pose, an observed pose, a measured pose, a determined pose) associated with the prediction time based on the additional sensor data. In some aspects, the localization system (or at least one subsystem thereof) is configured to, and can, update the trained machine learning model based on a comparison and / or a difference between the pose associated with the prediction time and the predicted pose associated with thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO44prediction time. This update to the trained machine learning model can be performed by further training the trained machine learning model based on the comparison and / or difference, by fine-tuning the trained machine learning model based on the comparison and / or difference, by adjusting parameters (e.g., hyperparameters) of the trained machine learning model based on the comparison and / or difference, or a combination thereof. This update to the trained machine learning model can improve an accuracy of the trained machine learning model.
[0126] In some aspects, the predicted pose is prediction of a pose of the system at the prediction time.
[0127] In some aspects, the indication of the prediction time identifies a prediction time window (e.g., prediction time window 320) from a beginning time to an end time. The prediction time is one of the beginning time, the end time, or a time in between the beginning time and the end time. In some aspects, the indication of the predicted pose includes a pose delta from a context pose (corresponding to a time within the context time window) to the predicted pose. In some aspects, the indication of the predicted pose includes a pose delta from the beginning time to the end time.
[0128] In some aspects, processing the plurality of IMU measurements and the indication of the prediction time (as in operation 1305) includes processing the plurality of IMU measurements (e.g., the IMU measurements 340, the IMU measurements 405), a plurality of IMU timestamps (e.g., corresponding to capture of the IMU measurements 340 and / or the IMU measurements 405), the indication of the prediction time (e.g., indicating the beginning, end, or middle of the prediction time window 320), a plurality of images (e.g., the image frames 345 and / or the image frames 410), a plurality of image timestamps (e.g., corresponding to capture of the image frames 345 and / or the image frames 410), or a combination thereof. At least a subset of the plurality of IMU timestamps correspond to capture of at least a subset of the plurality of IMU measurements by the IMU within the context time window. In some aspects, a second subset of the plurality of IMU timestamps correspond to capture of a second subset of the plurality of IMU measurements by the IMU within the prediction time window. At least a subset of the plurality of image timestamps correspond to capture of at least a subset of the plurality of images by an image sensor within the context time window. In some aspects, at least a second subset of the plurality of imagePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO45timestamps correspond to capture of at least a second subset of the plurality of images by an image sensor within the prediction time window.
[0129] In some aspects, processing the plurality of IMU measurements and the indication of the prediction time (as in operation 1305) includes calculating a double integral of an acceleration (e.g., through the integration calculations 415 and / or the ML model(s) 425) to determine the predicted pose associated with the prediction time.
[0130] In some aspects, processing the plurality of IMU measurements and the indication of the prediction time (as in operation 1305) includes calculating a regression associated with the predicted pose associated with the prediction time (e.g., see regression for delta pose referenced in FIG. 3).
[0131] In some examples, the localization system determines whether to activate or deactivate a relocator (e.g., relocator 530) based on a displacement associated with the predicted pose or corrected pose. The displacement can represent a magnitude of movement of the device relative to a previous pose or map location. In some examples, the relocator is activated if the displacement is below a threshold indicating a likelihood of a map connection, and the relocator is deactivated if the displacement exceeds the threshold indicating that relocalization is unlikely to succeed.
[0132] In some examples, in response to activating the relocator, the localization system predicts one or more feature positions within a map based on the predicted pose or corrected pose. The predicted feature positions are used to reduce a search space for feature matching during relocalization by limiting feature matching operations to regions of the map corresponding to the predicted pose.
[0133] At operation 1310, the localization system (or at least one subsystem thereof) is configured to, and can, perform a comparison between a confidence (associated with the indication of the predicted pose) and a relocalization threshold, If the confidence is less than (or, in some examples, equal to) the relocalization threshold, operation 1310 is followed by operation 1315. If the confidence is greater than (or, in some examples, equal to) the relocalization threshold, operation 1310 can end the process 1300, and / or return to operation 1305 to continue processing subsequent IMU measurements (e.g., in a continuous localization loop).Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO46
[0134] At operation 1315, the localization system (or at least one subsystem thereof) is configured to, and can, dynamically initialize a map of an environment based on the indication of the predicted pose to relocalize the predicted pose relative to the map. In some examples, dynamically initializing the map includes selecting a map component from a plurality of map components (e.g., map segments or map slices) based on the predicted pose or corrected pose. In some examples, operation 1315 can include locating the predicted pose within the map.
[0135] Examples of the plurality of images include images captured using the image capture and processing system 100, the image frames 345, the image frames 410, the first image 705, the second image 710, the images from any of the cameras 930A-930D, images from any of the cameras 1O3OA-1O3OD, images from any of the cameras 1130A-1130F, images from any of the sensors 1140A-1140B, images from the camera 1230, any other images discussed herein, images from any other camera or sensor discussed herein, or a combination thereof.
[0136] In some aspects, the plurality of images are captured using one or more image sensors of one or more cameras. In some examples, the localization system includes the one or more image sensors and / or the one or more cameras. Examples of the one or more image sensors and / or the one or more cameras include the image capture and processing system 100, the image sensor 130, the cameras 930A-930D, the cameras 1030A-1030D, the cameras 1130A-1130F, the sensors 1140A-1140B, the camera 1230, any other image sensors or cameras discussed herein, images from any other camera or sensor discussed herein, or a combination thereof.
[0137] In some aspects, dynamically initializing the map of the environment (as in operation 1315) includes modifying a previously-determined pose based on the predicted pose, for instance by adjusting the previously-determined pose, moving the previously-determined pose, or replacing the previously-determined pose with the predicted pose.
[0138] In some aspects, the localization system (or at least one subsystem thereof) is configured to, and can, modify the map of the environment based on the predicted pose within the map of the environment. For instance, if localization indicates that the predicted pose is outside of the map, then the localization system can update the map of the environment to encompass the predicted pose. If relocalization moves the pose of a device (that includes the IMU and / or the image sensors), and the sensor(s) of the device indicate that a wall or other object is a first distancePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO47from the device, but the map indicates that the wall or other object is a second distance from the device, then the map can be modified to make the wall or other object is actually at the first distance from the device.
[0139] In some aspects, dynamically initializing the predicted pose within the map of the environment (as in operation 1315) includes identifying that the predicted pose is within a particular map segment of a plurality of map segments that the map is divided into. The map segment can be referred to as a map slice or map portion. In some examples, the particular map segment can refer to a specific room (or set of rooms), or a portion thereof, while another map segment of the map can refer to a different room (or set of rooms) or portion thereof. In some examples, the particular map segment can refer to an indoor area, while another map segment in the map can refer to an outdoor area. In some examples, the particular map segment can refer to an outdoor area, while another map segment in the map can refer to an indoor area.
[0140] In some aspects, dynamically initializing the predicted pose within the map of the environment (as in operation 1315) includes relocalizing the predicted pose within the map (e.g., via mapper 510, relocator 530, map management engine 535, and / or bundle adjustment engine 540). In some aspects, relocalizing the predicted pose within the map (e.g., via mapper 510, relocator 530, map management engine 535, and / or bundle adjustment engine 540) is performed by the localization system (or at least one subsystem thereof) in response to the confidence being less than the relocalization threshold (as in operation 1310). The confidence is associated with the predicted pose. In some aspects, the relocalization threshold may be dynamic, and may be based on a quantity of visual features in image data associated with the environment (e.g., based on whether the quantity of visual features is above or below a visual feature quantity threshold), a level of uncertainty (or confidence) associated with features extracted from image data associated with the environment (e.g., based on whether the uncertainty or confidence is above or below an uncertainty threshold or confidence threshold), or a combination thereof. In some aspects, the relocalization threshold may instead be fixed.
[0141] In some aspects, the predicted pose includes a three-dimensional position and / or an orientation. The three-dimensional position can refer to X, Y, and Z coordinates, such as latitude, longitude, and / or altitude / elevation. The orientation can refer to a pitch, a yaw, and / or a roll.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO48
[0142] In some examples, the process 1300 provides technical improvements to operation of computing systems by reducing data movement and bandwidth usage within the system. For instance, by generating the corrected pose based on IMU measurements and selectively activating the relocator based on the corrected pose, the localization system can reduce a frequency of image frame retrieval, transmission, and processing. This can reduce bandwidth utilization between an image sensor, an image signal processor (ISP), system memory, and one or more processors. In some examples, limiting feature matching to map regions corresponding to the corrected pose reduces memory accesses associated with retrieving map data and reduces data transfer across internal buses or interconnects, thereby improving memory efficiency and reducing contention for shared resources. In some examples, these reductions in data movement improve throughput of the system by allowing processing resources to be allocated to additional tasks and by reducing latency associated with memory and data transfers.
[0143] In some examples, the process 1300 further improves operation and longevity of hardware components by reducing unnecessary activation of computational and mechanical subsystems. For instance, selectively activating the relocator reduces execution of computationally intensive feature matching operations on processors such as central processing units (CPUs), graphics processing units (GPUs), and digital signal processors (DSPs), thereby reducing thermal load and power consumption. In some examples, reduced reliance on imagebased relocalization decreases usage of camera sensors and associated control mechanisms (e.g., focus control mechanisms 125B and exposure control mechanisms 125 A), which can reduce wear and tear on moving components (e.g., actuators and the components moved by the actuators) and extend device lifetime. In examples involving robotic systems or vehicles (e.g., UGV 1210 or UAV 1220), improved pose estimation and reduced relocalization latency can reduce corrective movements and actuator usage, thereby reducing mechanical wear and improving motion efficiency. In some examples, these improvements enhance coordination across sensor pipelines, processing pipelines, and memory subsystems, resulting in increased overall system throughput, reduced latency, and more efficient hardware utilization.
[0144] In some examples, the processes described herein (e.g., the respective processes of FIGs.1, 4, 5, 6, 7, the process 1300 of FIG. 13, and / or other processes described herein) may be performed by a computing device or apparatus. In some examples, the processes described hereinPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO49can be performed by the image capture and processing system 100, the image capture device 105 A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, an IMU that captures the IMU measurements 340, an image sensor that captures the image frames 345, the imaging system 400, an IMU that captures the IMU measurements 405, an image sensor that captures the image frames 410, the ML engine 420, the ML model(s) 425, the SLAM system 500, the tracker 505, the mapper 510, the neural network 600, the system 830, the system 840, the HMD 910, the mobile handset 1010, the vehicle 1110, the UGV 1210, the UAV 1220, the imaging system that performs the process 1300, the computing system 1400, the processor 1410, an apparatus, a system, a non-transitory computer-readable medium coupled to a processor, or a combination thereof.
[0145] The computing device can include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a vehicle or computing device of a vehicle, a robotic device, a television, and / or any other computing device with the resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) that are configured to carry out the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive the data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other type of data.
[0146] The components of the computing device can be implemented in circuitry. For example, the components can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO50
[0147] The processes described herein are illustrated as logical flow diagrams, block diagrams, or conceptual diagrams, the operation of which represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the processes.
[0148] Additionally, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0149] FIG. 14 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, FIG. 14 illustrates an example of computing system 1400, which can be for example any computing device making up an internal computing system, a remote computing system, a camera, or any component thereof in which the components of the system are in communication with each other using connection 1405. Connection 1405 can be a physical connection using a bus, or a direct connection into processor 1410, such as in a chipset architecture. Connection 1405 can also be a virtual connection, networked connection, or logical connection.
[0150] In some aspects, computing system 1400 is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some aspects, one or more of the described system components represents manyPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO51such components each performing some or all of the function for which the component is described. Tn some aspects, the components can be physical or virtual devices.
[0151] Example system 1400 includes at least one processing unit (CPU or processor) 1410 and connection 1405 that couples various system components including system memory 1415, such as read-only memory (ROM) 1420 and random access memory (RAM) 1425 to processor 1410. Computing system 1400 can include a cache 1412 of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1410.
[0152] Processor 1410 can include any general purpose processor and a hardware service or software service, such as services 1432, 1434, and 1436 stored in storage device 1430, configured to control processor 1410 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 1410 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.
[0153] To enable user interaction, computing system 1400 includes an input device 1445, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system 1400 can also include output device 1435, which can be one or more of a number of output mechanisms. In some instances, multimodal systems can enable a user to provide multiple types of input / output to communicate with computing system 1400. Computing system 1400 can include communications interface 1440, which can generally govern and manage the user input and system output. The communication interface may perform or facilitate receipt and / or transmission wired or wireless communications using wired and / or wireless transceivers, including those making use of an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, a BLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE) wireless signal transfer, an IBEACON® wireless signal transfer, a radio-frequency identification (RFID) wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 1402.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, Visible LightPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO52Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communications interface 1440 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system 1400 based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
[0154] Storage device 1430 can be a non-volatile and / or non-transitory and / or computer-readable memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, a EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#),Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO53resistive random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.
[0155] The storage device 1430 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 1410, it causes the system to perform a function. In some aspects, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1410, connection 1405, output device 1435, etc., to carry out the function.
[0156] As used herein, the term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediua capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, or the like.
[0157] In some aspects, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
[0158] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood byPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO54one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.
[0159] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.
[0160] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO55
[0161] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.
[0162] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.
[0163] In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.
[0164] One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (“<”) and greater than or equal to (“>”) symbols, respectively, without departing from the scope of this description.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO56
[0165] Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.
[0166] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.
[0167] Claim language or other language reciting “at least one of’ a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of’ a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.
[0168] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
[0169] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication devicePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO57handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.
[0170] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC).Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO58
[0171] Illustrative aspects of the disclosure include:
[0172] Aspect 1. An apparatus for relocalization in a mapping system, the apparatus comprising: one or more memories; and one or more processors coupled to the one or more memories and configured to: process a plurality of inertial measurement unit (IMU) measurements and a time indication of a prediction time to generate a pose indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU; and in response to a comparison between a confidence associated with the pose indication of the predicted pose and a relocalization threshold, dynamically initialize a map of an environment based on the pose indication of the predicted pose to relocalize the predicted pose relative to the map.
[0173] Aspect 2. The apparatus of Aspect 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements and the time indication of the prediction time using a trained machine learning model that generates the pose indication of the predicted pose associated with the prediction time.
[0174] Aspect 3. The apparatus of Aspect 2, wherein the trained machine learning model is a neural network.
[0175] Aspect 4. The apparatus of any one of Aspects 2 to 3, wherein the at least one processor is configured to: receive additional sensor data captured during the prediction time; determine a pose associated with the prediction time based on the additional sensor data; and update the trained machine learning model based on a difference to improve an accuracy of the trained machine learning model, wherein the difference is between the pose associated with the prediction time and the predicted pose associated with the prediction time.
[0176] Aspect 5. The apparatus of any one of Aspects 1 to 4, wherein the predicted pose is prediction of a pose of the apparatus at the prediction time, and wherein the prediction time is after the context time window.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO59
[0177] Aspect 6. The apparatus of any one of Aspects 1 to 5, wherein the time indication of the prediction time identifies a prediction time window from a beginning time to an end time, wherein the prediction time is one of the beginning time or the end time.
[0178] Aspect 7. The apparatus of Aspect 6, wherein the pose indication of the predicted pose includes a pose delta from a context pose to the predicted pose, wherein the context pose corresponds to a time within the context time window.
[0179] Aspect 8. The apparatus of any one of Aspects 6 to 7, wherein the pose indication of the predicted pose includes a pose delta from the beginning time to the end time.
[0180] Aspect 9. The apparatus of any one of Aspects 1 to 8, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, the time indication of the prediction time, and a plurality of images, wherein at least a subset of the plurality of images are captured during the context time window using an image sensor that is coupled to the IMU.
[0181] Aspect 10. The apparatus of any one of Aspects 1 to 9, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, a plurality of IMU timestamps, the time indication of the prediction time, a plurality of images, and a plurality of image timestamps, wherein at least a subset of the plurality of IMU timestamps correspond to capture of at least a subset of the plurality of IMU measurements by the IMU within the context time window, and wherein at least a subset of the plurality of image timestamps correspond to capture of at least a subset of the plurality of images by an image sensor within the context time window.
[0182] Aspect 11. The apparatus of any one of Aspects 1 to 10, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, a plurality of timestamps, and the time indication of the prediction time, wherein the plurality of timestamps correspond to capture of the plurality of IMU measurements by the IMU within the context time window.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO60
[0183] Aspect 12. The apparatus of any one of Aspects 1 to 11, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to modify a previously-determined pose based on the predicted pose.
[0184] Aspect 13. The apparatus of any one of Aspects 1 to 12, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to calculate a double integral of an acceleration to determine the predicted pose associated with the prediction time.
[0185] Aspect 14. The apparatus of any one of Aspects 1 to 13, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to calculate a regression associated with the predicted pose associated with the prediction time.
[0186] Aspect 15. The apparatus of any one of Aspects 1 to 14, wherein the at least one processor is configured to: modify the map of the environment based on the predicted pose within the map of the environment.
[0187] Aspect 16. The apparatus of any one of Aspects 1 to 15, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to identify that the predicted pose is within a particular map segment of a plurality of map segments that the map is divided into.
[0188] Aspect 17. The apparatus of any one of Aspects 1 to 16, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to relocalize the predicted pose within the map in response to a confidence being less than a relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein the relocalization threshold is based on a quantity of visual features in image data associated with the environment.
[0189] Aspect 18. The apparatus of any one of Aspects 1 to 17, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to relocalize the predicted pose within the map in response to a confidence being less than a relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO61relocalization threshold is based on uncertainty associated with features extracted from image data associated with the environment.
[0190] Aspect 19. The apparatus of any one of Aspects 1 to 18, wherein the predicted pose includes a three-dimensional position and an orientation, and wherein the orientation includes at least one of a pitch, a yaw, or a roll.
[0191] Aspect 20. A method of relocalization in a mapping system, the method comprising: processing a plurality of inertial measurement unit (IMU) measurements and a time indication of a prediction time to generate a pose indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU; and in response to a comparison between a confidence associated with the pose indication of the predicted pose and a relocalization threshold, dynamically initializing a map of an environment based on the pose indication of the predicted pose to relocalize the predicted pose relative to the map.
[0192] Aspect 21. The method of Aspect 20, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes processing the plurality of IMU measurements and the time indication of the prediction time using a trained machine learning model that generates the pose indication of the predicted pose associated with the prediction time.
[0193] Aspect 22. The method of Aspect 21, wherein the trained machine learning model is a neural network.
[0194] Aspect 23. The method of any one of Aspects 21 to 22, further comprising: receiving additional sensor data captured during the prediction time; determining a pose associated with the prediction time based on the additional sensor data; and updating the trained machine learning model based on a difference to improve an accuracy of the trained machine learning model, wherein the difference is between the pose associated with the prediction time and the predicted pose associated with the prediction time.
[0195] Aspect 24. The method of any one of Aspects 20 to 23, wherein the predicted pose is prediction of a pose of the method at the prediction time.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO62
[0196] Aspect 25. The method of any one of Aspects 20 to 24, wherein the time indication of the prediction time identifies a prediction time window from a beginning time to an end time, wherein the prediction time is one of the beginning time or the end time.
[0197] Aspect 26. The method of Aspect 25, wherein the pose indication of the predicted pose includes a pose delta from a context pose to the predicted pose, wherein the context pose corresponds to a time within the context time window.
[0198] Aspect 27. The method of any one of Aspects 25 to 26, wherein the pose indication of the predicted pose includes a pose delta from the beginning time to the end time.
[0199] Aspect 28. The method of any one of Aspects 20 to 27, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes processing the plurality of IMU measurements, the time indication of the prediction time, and a plurality of images, wherein at least a subset of the plurality of images are captured during the context time window using an image sensor that is coupled to the IMU.
[0200] Aspect 29. The method of any one of Aspects 20 to 28, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes processing the plurality of IMU measurements, a plurality of IMU timestamps, the time indication of the prediction time, a plurality of images, and a plurality of image timestamps, wherein at least a subset of the plurality of IMU timestamps correspond to capture of at least a subset of the plurality of IMU measurements by the IMU within the context time window, and wherein at least a subset of the plurality of image timestamps correspond to capture of at least a subset of the plurality of images by an image sensor within the context time window.
[0201] Aspect 30. The method of any one of Aspects 20 to 29, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes processing the plurality of IMU measurements, a plurality of timestamps, and the time indication of the prediction time, wherein the plurality of timestamps correspond to capture of the plurality of IMU measurements by the IMU within the context time window.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO63
[0202] Aspect 31. The method of any one of Aspects 20 to 30, wherein dynamically initializing the map of the environment includes modifying a previously-determined pose based on the predicted pose.
[0203] Aspect 32. The method of any one of Aspects 20 to 31, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes calculating a double integral of an acceleration to determine the predicted pose associated with the prediction time.
[0204] Aspect 33. The method of any one of Aspects 20 to 32, wherein processing the plurality of IMU measurements and the time indication of the prediction time includes calculating a regression associated with the predicted pose associated with the prediction time.
[0205] Aspect 34. The method of any one of Aspects 20 to 33, further comprising: modifying the map of the environment based on the predicted pose within the map of the environment.
[0206] Aspect 35. The method of any one of Aspects 20 to 34, wherein dynamically initializing the map of the environment includes identifying that the predicted pose is within a particular map segment of a plurality of map segments that the map is divided into.
[0207] Aspect 36. The method of any one of Aspects 20 to 35, wherein dynamically initializing the map of the environment includes relocalizing the predicted pose within the map in response to a confidence being less than the relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein the relocalization threshold is based on a quantity of visual features in image data associated with the environment.
[0208] Aspect 37. The apparatus of any one of Aspects 20 to 36, wherein dynamically initializing the map of the environment includes relocalizing the predicted pose within the map in response to a confidence being less than the relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein the relocalization threshold is based on uncertainty associated with features extracted from image data associated with the environment.
[0209] Aspect 38. The method of any one of Aspects 20 to 37, wherein the predicted pose includes a three-dimensional position and an orientation, and wherein the orientation includes at least one of a pitch, a yaw, or a roll.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO64
[0210] Aspect 39. A method for relocalization in a mapping system, the method comprising: receiving inertial measurement unit (IMU) data from one or more inertial sensors of a device; processing the IMU data to generate a corrected pose of the device; determining, based on the corrected pose, whether to activate or deactivate a relocator for relocalization, wherein the relocator is activated when a displacement of the device is below a threshold indicating a possibility for a map connection, and the relocator is deactivated when the displacement exceeds the threshold; in response to activating the relocator, predicting, based on the corrected pose, a feature position on a map to reduce a search space for feature matching during relocalization; and dynamically initializing a map for relocalization based on the corrected pose, wherein dynamically initializing the map comprises selecting a map component from a plurality of map components based on the corrected pose.
[0211] Aspect 40. The method of Aspect 39, wherein the IMU data is processed using a deep learning (DL) network, and wherein the DL network is trained to compensate for at least one of drift, noise, or non-linearities in the IMU data.
[0212] Aspect 41. The method of any of Aspects 39 to 40, further comprising operations according to any of Aspects 1 to 38.
[0213] Aspect 42. The apparatus of any of Aspects 1 to 19, further comprising operations according to any of Aspects 39 to 40.
[0214] Aspect 43. The method of any of Aspects 20 to 38, further comprising operations according to any of Aspects 39 to 40.
[0215] Aspect 44. A non -transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any of Aspects 1 to 43.
[0216] Aspect 45. An apparatus for relocalization in a mapping system, the apparatus comprising one or more means for performing operations according to any of Aspects 1 to 43.Polsinelli Ref. No. 097922-892031
Claims
PATENTQualcomm Ref. No. 2504046WO65CLAIMSWHAT IS CLAIMED IS:
1. An apparatus for relocalization in a mapping system, the apparatus comprising: at least one memory; andat least one processor coupled to the at least one memory and configured to:process a plurality of inertial measurement unit (IMU) measurements and an time indication of a prediction time to generate a pose indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU; andin response to a comparison between a confidence associated with the pose indication of the predicted pose and a relocalization threshold, dynamically initialize a map of an environment based on the pose indication of the predicted pose to relocalize the predicted pose relative to the map.
2. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements and the time indication of the prediction time using a trained machine learning model that generates the pose indication of the predicted pose associated with the prediction time.
3. The apparatus of claim 2, wherein the trained machine learning model is a neural network.
4. The apparatus of claim 2, wherein the at least one processor is configured to: receive additional sensor data captured during the prediction time;determine a pose associated with the prediction time based on the additional sensor data; andPolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO66update the trained machine learning model based on a difference to improve an accuracy of the trained machine learning model, wherein the difference is between the pose associated with the prediction time and the predicted pose associated with the prediction time.
5. The apparatus of claim 1, wherein the predicted pose is prediction of a pose of the apparatus at the prediction time, and wherein the prediction time is after the context time window.
6. The apparatus of claim 1, wherein the time indication of the prediction time identifies a prediction time window from a beginning time to an end time, wherein the prediction time is one of the beginning time or the end time.
7. The apparatus of claim 6, wherein the pose indication of the predicted pose includes a pose delta from a context pose to the predicted pose, wherein the context pose corresponds to a time within the context time window.
8. The apparatus of claim 6, wherein the pose indication of the predicted pose includes a pose delta from the beginning time to the end time.
9. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, the time indication of the prediction time, and a plurality of images, wherein at least a subset of images are captured during the context time window using an image sensor that is coupled to the IMU.
10. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, a plurality of IMU timestamps, the time indication of the prediction time, a plurality of images, and a plurality of image timestamps, wherein at least a subset of the plurality of IMU timestamps correspond to capture of at least a subset of thePolsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO67plurality of IMU measurements by the IMU within the context time window, and wherein at least a subset of the plurality of image timestamps correspond to capture of at least a subset of the plurality of images by an image sensor within the context time window.
11. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to process the plurality of IMU measurements, a plurality of timestamps, and the time indication of the prediction time, wherein the plurality of timestamps correspond to capture of the plurality of IMU measurements by the IMU within the context time window.
12. The apparatus of claim 1, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to modify a previously-determined pose based on the predicted pose.
13. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to calculate a double integral of an acceleration to determine the predicted pose associated with the prediction time.
14. The apparatus of claim 1, wherein, to process the plurality of IMU measurements and the time indication of the prediction time, the at least one processor is configured to calculate a regression associated with the predicted pose associated with the prediction time.
15. The apparatus of claim 1, wherein the at least one processor is configured to: modify the map of the environment based on the predicted pose within the map of the environment.
16. The apparatus of claim 1, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to identify that the predicted pose is within a particular map segment of a plurality of map segments that the map is divided into.Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO6817. The apparatus of claim 1 , wherein, to dynamically initialize the map of the environment, the at least one processor is configured to relocalize the predicted pose within the map in response to a confidence being less than the relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein the relocalization threshold is based on a quantity of visual features in image data associated with the environment.
18. The apparatus of claim 1, wherein, to dynamically initialize the map of the environment, the at least one processor is configured to relocalize the predicted pose within the map in response to a confidence being less than the relocalization threshold, wherein the confidence is associated with the predicted pose, and wherein the relocalization threshold is based on uncertainty associated with features extracted from image data associated with the environment.
19. The apparatus of claim 1, wherein the predicted pose includes a three-dimensional position and an orientation, wherein the orientation includes at least one of a pitch, a yaw, or a roll.
20. A method of relocalization in a mapping system, the method comprising: processing a plurality of inertial measurement unit (IMU) measurements and a time indication of a prediction time to generate a pose indication of a predicted pose associated with the prediction time, wherein a context time window corresponds to capture of the plurality of IMU measurements by an IMU, and; andin response to a comparison between a confidence associated with the pose indication of the predicted pose and a relocalization threshold, dynamically initialize a map of an environment based on the pose indication of the predicted pose to relocalize the predicted pose relative to the map.
21. A method for relocalization in a mapping system, the method comprising:Polsinelli Ref. No. 097922-892031PATENTQualcomm Ref. No. 2504046WO69receiving inertial measurement unit (IMU) data from one or more inertial sensors of a device;processing the IMU data to generate a corrected pose of the device;determining, based on the corrected pose, whether to activate or deactivate a relocator for relocalization, wherein the relocator is activated when a displacement of the device is below a threshold indicating a possibility for a map connection, and the relocator is deactivated when the displacement exceeds the threshold;in response to activating the relocator, predicting, based on the corrected pose, a feature position on a map to reduce a search space for feature matching during relocalization; and dynamically initializing a map for relocalization based on the corrected pose, wherein dynamically initializing the map comprises selecting a map component from a plurality of map components based on the corrected pose.
22. The method of claim 21, wherein the IMU data is processed using a deep learning (DL) network, and wherein the DL network is trained to compensate for at least one of drift, noise, or non-linearities in the IMU data.Polsinelli Ref. No. 097922-892031
