Memory-augmented detection

Abstract

A method implements memory-augmented detection. The method involves generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model. The method further involves generating a detection proposal by processing the current data with a detector model. The method further involves generating a merged proposal by processing the memory proposal and the detection proposal with a merger model. The method further involves generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model. The method further involves updating multiple stored proposals including the stored proposal by storing the refined proposal to the memory bank.

Description

MEMORY-AUGMENTED DETECTIONCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U. S. Provisional Application 63 / 720,413, filed on November 14, 2024, which is hereby incorporated by reference herein.BACKGROUND

[0002] Self-driving vehicles operate within real-world environments by relying on artificial intelligence systems to make decisions that guide movement, interaction, and response to dynamic conditions. Autonomous navigation through geographic regions involves continuous interpretation of sensor data to identify agents and map elements. The autonomous system processes measurements from sensors such as light detection and ranging (LiDAR) and cameras to perceive the surrounding environment. The virtual driver, which is the decision-making software component of the autonomous system, interprets sensor inputs and generates control signals that direct actuators to perform actions such as steering, braking, and acceleration. The autonomous system interacts with both stationary map elements and dynamic agents, including pedestrians, vehicles, and other mobile entities, each of which may exhibit independent motion patterns and unpredictable behavior.

[0003] Object detection within the context of autonomous navigation involves identifying and localizing entities in three-dimensional space using sensor- derived data. Detection models analyze spatial and temporal features to generate bounding box proposals that represent the location, orientation, and dimensions of objects. The proposals are derived from sensor modalities such as LiDAR point clouds or camera images and may be processed through single- stage or multi-stage neural networks. Object detection faces challenges such as occlusion, limited visibility, and sparse sensor coverage, particularly for distant or fast-moving objects.SUMMARY

[0004] In general, in one or more aspects, the disclosure relates to a method for memory-augmented detection. The method involves generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model. The method further involves generating a detection proposal by processing the current data with a detector model. The method further involves generating a merged proposal by processing the memory proposal and the detection proposal with a merger model. The method further involves generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model. The method further involves updating multiple stored proposals including the stored proposal by storing the refined proposal to the memory bank.

[0005] In general, in one or more aspects, the disclosure relates to a system that includes at least one processor and an application that executes on the at least one processor. Executing the application performs generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model. Executing the application further performs generating a detection proposal by processing the current data with a detector model. Executing the application further performs generating a merged proposal by processing the memory proposal and the detection proposal with a merger model. Executing the application further performs generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model. Executing the application further performs updating multiple stored proposals including the stored proposal by storing the refined proposal to the memory bank.

[0006] In general, in one or more aspects, the disclosure relates to a non- transitory computer readable medium including instructions executable by at least one processor. Executing the instructions performs generating a memory proposal by processing current data and a stored proposal from a memory bankwith a retrieval model. Executing the instructions further performs generating a detection proposal by processing the current data with a detector model. Executing the instructions further performs generating a merged proposal by processing the memory proposal and the detection proposal with a merger model. Executing the instructions further performs generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model. Executing the instructions further performs updating multiple stored proposals including the stored proposal by storing the refined proposal to the memory bank.

[0007] Other aspects of one or more embodiments may be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1, FIG. 2, FIG. 3, FIG. 4.1, FIG. 4.2, FIG. 4.3, FIG. 4.4, and FIG.4.5 show diagrams in accordance with the disclosure.

[0009] FIG. 5 shows a method in accordance with the disclosure.

[0010] FIG. 3, FIG. 4, FIG. 5, FIG. 6, and FIG. 7 show examples in accordance with the disclosure.

[0011] FIG. 8.1 and 8.2 show a computing system in accordance with the disclosure.

[0012] Similar elements in the various figures may be denoted by similar names and reference numerals. The details of features and elements described in one figure may extend to similarly named features and elements in different figures.DETAILED DESCRIPTION

[0013] A transformer-based memory-augmentation module enhances object detection in autonomous navigation systems by integrating long-horizon temporal fusion into existing 3D detectors. The module processes detection proposals from current sensor data and memory proposals from past detections,which are stored in a memory bank. A proposal merging mechanism combines the current detection proposals and the historical memory proposals, followed by iterative refinement using cross-attention and factorized self-attention. The system forecasts object trajectories and aligns memory entries with current observations to improve detection accuracy, particularly under occlusion and sparse sensor conditions.

[0014] Conventional object detection systems in self-driving vehicles face limitations due to short temporal input windows, which may span less than one second. Detection performance degrades in scenarios involving occlusion, rapid motion, or distant objects with limited sensor coverage. Existing temporal fusion methods struggle with scalability, alignment errors, and reliance on explicit tracking, which may introduce compounding mistakes. Memory-based approaches may fail to recover missed detections and suffer from miscalibrated confidence scores when combining current and historical proposals.

[0015] The disclosed system addresses the limitations by introducing a memory- augmented detection framework that performs long-horizon temporal fusion without explicit object association. Forecasted trajectories align memory proposals with current sensor data for recovery of missed detections and refinement of object positions. Learned re-scoring mechanisms calibrate confidence scores across memory and detection proposals, improving proposal selection. A transformer-based refinement module applies cross-attention and factorized self-attention to enhance spatial and temporal reasoning, resulting in improved detection robustness and accuracy across diverse driving scenarios.

[0016] Turning to the Figures, FIGs. 1 and 2 show example diagrams of the autonomous system and virtual driver. Turning to FIG. 1, an autonomous system (116) is a self-driving mode of transportation that does not require a human pilot or human driver to move and react to the real-world environment. The autonomous system (116) may be completely autonomous or semi- autonomous. As a mode of transportation, the autonomous system (116) iscontained in a housing configured to move through a real-world environment. Examples of autonomous systems include self-driving vehicles (e.g., selfdriving trucks and cars), drones, airplanes, robots, etc. The autonomous system (116) includes a virtual driver (102) that is the decision-making portion of the autonomous system (116). The virtual driver (102) is an artificial intelligence system that learns how to interact in the real world and interacts accordingly. The virtual driver (102) is the software executing on a processor that makes decisions and causes the autonomous system (116) to interact with the real world including moving, signaling, and stopping or maintaining a current state. Specifically, the virtual driver (102) is decision-making software that executes on hardware (not shown). The hardware may include a hardware processor, memory or other storage device, and one or more interfaces. A hardware processor is any hardware processing unit that is configured to process computer readable program code and perform the operations set forth in the computer readable program code. A real-world environment is the portion of the real world through which the autonomous system (116), when trained, is designed to move. Thus, the real-world environment may include concrete and land, construction, and other objects in a geographic region along with agents. The agents are the other agents in the real-world environment that are capable of moving through the real-world environment. Agents may have independent decision-making functionality. The independent decision-making functionality of the agent may dictate how the agent moves through the environment and may be based on visual or tactile cues from the real -world environment. For example, agents may include other autonomous and non-autonomous transportation systems (e.g., other vehicles, bicyclists, robots), pedestrians, animals, etc.

[0017] In the real world, the geographic region is an actual region within the real- world that surrounds the autonomous system. Namely, from the perspective of the virtual driver, the geographic region is the region through which the autonomous system moves. The geographic region includes agents and mapelements that are located in the real world. Namely, the agents and map elements each have a physical location in the geographic region that denotes a place in which the corresponding agent or map element is located. The map elements are stationary in the geographic region, whereas the agents may be stationary or nonstationary in the geographic region. The map elements are the elements shown in a map (e.g., road map, traffic map, etc.) or derived from a map of the geographic region.

[0018] The real -world environment changes as the autonomous system (116) moves through the real -world environment. For example, the geographic region may change, and the agents may move positions, including new agents being added and existing agents leaving.

[0019] In order to interact with the real-world environment, the autonomous system (116) includes various types of sensors (104), such as LiDAR sensors amongst other types, which are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment. The autonomous system (116) may include other types of sensors as well. The sensors (104) provide input to the virtual driver (102). In addition to sensors (104), the autonomous system (116) includes one or more actuators (108). An actuator is hardware and / or software that is configured to control one or more physical parts of the autonomous system based on a control signal from the virtual driver (102).

[0020] In one or more embodiments, the control signal specifies an action for the autonomous system (e.g., turn on the blinker, apply breaks by a defined amount, apply accelerator by a defined amount, turn the steering wheel or tires by a defined amount, etc.). The actuator(s) (108) are configured to implement the action. In one or more embodiments, the control signal may specify a new state of the autonomous system, and the actuator may be configured to implement the new state to cause the autonomous system to be in the new state. For example, the control signal may specify that the autonomous system shouldturn by a certain amount while accelerating at a predefined rate, while the actuator determines and causes the wheel movements and the amount of acceleration on the accelerator to achieve a certain amount of turn and acceleration rate.

[0021] The testing and training of the virtual driver (102) of the autonomous systems in the real-world environment may be unsafe because of the accidents that an untrained virtual driver can cause. Thus, as shown in FIG. 2, a simulator (200) is configured to train and test a virtual driver (102) of an autonomous system. For example, the simulator may be a unified, modular, mixed-reality, closed-loop simulator for autonomous systems. The simulator (200) is a configurable simulation framework that enables not only evaluation of different autonomy components of the virtual driver (102) in isolation, but also as a complete system in a closed-loop manner. The simulator reconstructs “digital twins” of real-world scenarios automatically, enabling accurate evaluation of the virtual driver at scale.

[0022] The simulator (200) creates the simulated environment (204) which is a virtual world in which the virtual driver (102) is a player in the virtual world. The simulated environment (204) is a simulation of a real-world environment, which may or may not be in actual existence, in which the autonomous system is designed to move. As such, the simulated environment (204) includes a simulation of the objects (i.e., simulated objects or agents) and background in the real world, including the natural objects, construction, buildings and roads, obstacles, as well as other autonomous and non- autonomous objects. The simulated environment simulates the environmental conditions within which the autonomous system may be deployed. The simulated objects may include both stationary and nonstationary objects. Nonstationary objects are agents in the real-world environment. In the simulated environment, the geographic region is a realistic representation of a real-world region that may or may not be in actual existence. Namely, from the perspective of the virtual driver, thegeographic region appears the same as if the geographic region were in existence, if the geographic region does not actually exist, or the same as the actual geographic region present in the real world.

[0023] The geographic region in the simulated environment includes virtual agents and virtual map elements that would be actual agents and actual map elements in the real world. Namely, the virtual agents and virtual map elements each have a physical location in the geographic region that denotes an exact spot or place in which the corresponding agent or map element is located. The map elements are stationary in the geographic region, whereas the agents may be stationary or nonstationary in the geographic region. As with the real world, a map exists of the geographic region that specifies the physical locations of the map elements.

[0024] The simulator (200) includes an autonomous system model (216), sensor simulation models (214), and agent models (218). The autonomous system model (216) is a detailed model of the autonomous system in which the virtual driver (102) will execute. The autonomous system model (216) includes model, geometry, physical parameters (e.g., mass distribution, points of significance), engine parameters, sensor locations and type, firing pattern of the sensors, information about the hardware on which the virtual driver executes (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. The various parameters of the autonomous system model may be configurable by the user or another system.

[0025] The autonomous system model (216) includes an autonomous system dynamic model. The autonomous system dynamic model is used for dynamics simulation that takes the actuation actions of the virtual driver (e.g., steering angle, desired acceleration) and enacts the actuation actions on the autonomous system in the simulated environment to update the simulated environment and the state of the autonomous system. The interface between the virtual driver (102) and the simulator (200) may match the interface between the virtualdriver (102) and the autonomous system in the real world. Thus, to the virtual driver (102), the simulator simulates the experience of the virtual driver within the autonomous system in the real world.

[0026] In one or more embodiments, the sensor simulation model (214) models, in the simulated environment, active and passive sensor inputs. The sensor simulation models (114) are configured to simulate the sensor observations of the surrounding scene in the simulated environment (204) at each time step according to the sensor configuration on the vehicle platform. Passive sensor inputs capture the visual appearance of the simulated environment including stationary and nonstationary simulated objects from the perspective of one or more cameras based on the simulated position of the camera(s) within the simulated environment. Examples of passive sensor inputs include inertial measurement unit (IMU) and thermal. Active sensor inputs are inputs to the virtual driver of the autonomous system from the active sensors, such as LiDAR, radar, global positioning system (GPS), ultrasound, etc. Namely, the active sensor inputs include the measurements taken by the sensors, and the measurements being simulated based on the simulated environment, based on the simulated position of the sensor(s) within the simulated environment.

[0027] Agent models (218) represent an agent in a scenario. An agent is a sentient being that has an independent decision-making process. Namely, in a real world, the agent may be an animate being (e.g, person or animal) that makes a decision based on an environment. The agent makes active movement rather than or in addition to passive movement. An agent model, or an instance of an actor model may exist for each agent in a scenario. The agent model is a model of the agent. If the agent is in a mode of transportation, then the agent model includes the model of transportation in which the agent is located. For example, actor models may represent pedestrians, children, vehicles being driven by drivers, pets, bicycles, and other types of actors. The training of the virtual driver may be performed by using log data. Log data from the real world is usedto generate an initial virtual world. The log data may be used to define, at least in part, which asset and actor models are used in an initial positioning of assets. The simulator executes a sensor simulation model that may use beamforming and other techniques to replicate the view to the sensors of the autonomous system.

[0028] The simulated sensor output is passed to the virtual driver. The virtual driver executes based on the simulated sensor output to generate actuation actions. The actuation actions define how the virtual driver controls the autonomous system. For example, for a self-driving vehicle, the actuation actions may be the amount of acceleration, movement of the steering, triggering of a turn signal, etc. From the actuation actions, the autonomous system state in the simulated environment is updated. Further, actors’ actions in the simulated environment are modeled based on the simulated environment state. Concurrently with the virtual driver model, the actor models and asset models are executed on the simulated environment state to determine an update for each of the assets and actors in the simulated environment. The actors’ actions may use the previous output of the evaluator to test the virtual driver.

[0029] The updated simulated environment state is updated according to the actors’ actions and the autonomous system state. The updated simulated environment includes the change in positions of the actors and the autonomous system. The process can be repeated for a next timestep. Because the models execute independently of the real world, the update may reflect a deviation from the real world. Thus, the autonomous system is tested with new scenarios.

[0030] The virtual driver (302) is an implementation of the virtual driver (102) of FIG. 1 and is decision-making software that interprets sensor data and generates control signals for autonomous navigation. The virtual driver (302) includes the memory-augmented detection model (305) to perform object detection using both the current sensor data (310) and historical data stored in the memory bank (352). The virtual driver (302) processes inputs from thecurrent data (308) and the memory bank (352) to generate the refined object proposals (335) that are stored in the memory bank (352).

[0031] The memory-augmented detection model (305) is a transformer-based module that enhances object detection by performing long -horizon temporal fusion to combine information form current sensor data and historical observations over an extended time window. The memory-augmented detection model (305) processes the current data (308) and the stored proposals (355) to generate the memory proposals (320), merges the generated memory proposals (320) with the detection proposals (325), and refines the merged proposals (330) using cross-attention and factorized self-attention. The memory-augmented detection model (305) executes on a processor and includes models, such as the retrieval model (318), the detector model (322), the merger model (328), and the refinement model (332).

[0032] The current data (308) includes the sensor data (310), the current timestamp (312), and the current pose (315) of the autonomous system. The current data (308) is used to generate the detection proposals (325) and to align the stored proposals (355) retrieved from the memory bank (352). The current data (308) quantifies the present state of the autonomous system and the surrounding environment.

[0033] The sensor data (310) is a measurement of the surrounding environment captured by sensors such as LiDAR and cameras. For example, the LiDAR data within the sensor data (310) may include three-dimensional point clouds that quantify the spatial geometry of objects and surfaces by measuring the time-of- flight of laser pulses reflected from the environment. As another example, the camera data of the sensor data (310) may include two-dimensional images that quantify visual appearance, texture, and color information of the scene, which may be used to infer depth and object boundaries through monocular or stereo vision techniques. The sensor data (310) quantifies spatial and temporal features of objects and map elements in the environment, including objectcontours, motion cues, and occlusion patterns. The sensor data (310) is processed by the detector model (322) to generate the detection proposals (325) including bounding boxes, classification scores, and feature vectors.

[0034] The current timestamp (312) is a temporal marker indicating when the sensor data (310) was captured. The current timestamp (312) identifies the time of observation and is used to align the stored proposals (355) with the current observations. The current timestamp (312) is used by the retrieval model (318) to interpolate and extrapolate the memory proposals (320).

[0035] The current pose (315) is the spatial position and orientation of the autonomous system at the current timestamp (312). The current pose (315) identifies the location and heading of the autonomous system in a global coordinate frame. The current pose (315) is used to transform the stored proposals (355) into the current frame of reference during memory alignment.

[0036] The retrieval model (318) is a model that processes the current data (308) and the stored proposals (355) to generate the memory proposals (320). The retrieval model (318) receives the current timestamp (312), the current pose (315), and the stored proposals (355) as input, and outputs the memory proposals (320) aligned in space and time. The retrieval model (318) may include extrapolation algorithms and feature encoders executed on a processor to generate the memory proposals (320).

[0037] The memory proposals (320) are object proposals (see FIG. 4.1) derived from the stored proposals (355) and aligned with the current data (308). The memory proposals (320) include bounding boxes, trajectories, and features of objects observed in the past. The memory proposals (320) are processed by the merger model (328) and the refinement model (332) to improve detection accuracy.

[0038] The detector model (322) is a model that processes the current sensor data (310) to generate the detection proposals (325). The detector model (322) receives the sensor data (310) as input and outputs the detection proposals (325)including bounding boxes, confidence scores, and features. The detector model (322) may include convolutional neural networks or transformer-based architectures executed on a processor.

[0039] The detection proposals (325) are object proposals generated from the current sensor data (310) by the detector model (322). The detection proposals (325) may also include features obtained by interpolating intermediary feature maps inside the detector model (322). The detection proposals (325) identify the location, dimensions, and classification confidence of objects detected in the current frame. The detection proposals (325) are merged with the memory proposals (320) by the merger model (328).

[0040] The merger model (328) is a model that processes the detection proposals (325) and the memory proposals (320) to generate the merged proposals (330). The merger model (328) receives the detection and memory proposals as input and outputs the merged proposals (330) by re-scoring and filtering overlapping proposals. The merger model (328) may include multilayer perceptrons (MLPs) executed on a processor to calibrate confidence scores and perform nonmaximum suppression.

[0041] The merged proposals (330) are object proposals resulting from the combination of the detection proposals (325) and the memory proposals (320). The merged proposals (330) include bounding boxes, trajectories, and features selected based on calibrated confidence scores. The merged proposals (330) are refined by the refinement model (332) to improve spatial and temporal accuracy.

[0042] The refinement model (332) is a model that processes the merged proposals (330) and the memory proposals (320) to generate the refined proposals (335). The refinement model (332) receives the merged and memory proposals as input and outputs the refined proposals (335) using cross-attention and factorized self-attention. The refinement model (332) may includetransformer decoder blocks executed on a processor to generate updated bounding boxes, trajectories, and confidence scores.

[0043] The refined proposals (335) are object proposals generated by the refinement model (332) after iterative attention-based processing. The refined proposals (335) include updated bounding boxes, trajectories, and confidence scores for objects detected in the current frame. The refined proposals (335) are stored in the memory bank (352) for use in future inference steps.

[0044] The memory bank (352) is a data structure that stores the refined proposals (335) along with corresponding timestamps and poses. The memory bank (352) quantifies historical object detections and the spatial -temporal context. The memory bank (352) is accessed by the retrieval model (318) to generate the memory proposals (320) for future frames.

[0045] The stored proposals (355) are object proposals previously generated and refined by the refinement model (332). The stored proposals (355) include bounding boxes and trajectories of objects observed in past frames and may include features. The stored proposals (355) are stored together with the timestamp and pose corresponding to the time and location at which the proposals were generated. The stored proposals (355) are retrieved and aligned by the retrieval model (318) to generate the memory proposals (320).

[0046] FIG. 4.1 through FIG. 4.5 illustrate additional details of elements described in FIG. 3. Each of FIG. 4.1 through FIG. 4.5 expands upon the internal structure, subcomponents, and data relationships of the memory- augmented detection model (305) and associated models of FIG. 3.

[0047] Turning to FIG. 4.1, the object proposal (402) is a data structure used to represent an object detected or inferred within a scene. The object proposal (402) may correspond to any of the proposals described in FIG. 3, including the detection proposals (325), the memory proposals (320), the merged proposals (330), and the refined proposals (335). The object proposal (402) includes multiple components that describe spatial, temporal, and semanticcharacteristics of the object. The object proposal (402) is used throughout the memory-augmented detection model (305) to represent both detection proposals and memory proposals during inference and refinement. Each of the detection proposals (325), the memory proposals (320), the merged proposals (330), and the refined proposals (335) may include similar data as the object proposal (402), such as bounding boxes, trajectories, classification scores, features, pose, and timestamp.

[0048] The bounding box (405) is data that describes the spatial extent of an object in three-dimensional space. The bounding box (405) includes parameters such as the centroid coordinates, dimensions, and orientation of the object in a Bird’s-Eye View frame. The bounding box (405) is used to localize the object within the environment and is refined during processing by the refinement model (332).

[0049] The confidence score (408) is data that represents the likelihood that the object proposal (402) corresponds to a real object of a particular class. The confidence score (408) includes multi-class probabilities or logits that reflect classification certainty. The confidence score (408) is used during proposal merging and filtering to select high-confidence proposals for refinement.

[0050] The trajectory (410) is data that describes the predicted motion of the object over a sequence of future time steps. The trajectory (410) includes a series of waypoints representing the expected position and orientation of the object at each future timestamp. The trajectory (410) is used to align memory proposals (320) with current observations and is updated during refinement.

[0051] The proposal features (412) are data that represent latent characteristics of the object proposal (402) used for attention-based processing. The proposal features (412) include high-dimensional feature vectors derived from sensor data or memory encoding. The proposal features (412) are used as input to the merger model (328) and the refinement model (332) for scoring and attention operations.

[0052] The pose (415) is data that describes the spatial position and orientation of the autonomous system at the time the object proposal (402) was generated. The pose (415) includes translation and rotation parameters in a global coordinate frame. The pose (415) is used to transform stored proposals (355) into the current frame of reference during memory retrieval.

[0053] The timestamp (418) is data that indicates the time at which the object proposal (402) was generated. The timestamp (418) includes a temporal marker used to align proposals across time and to compute memory age. The timestamp (418) is used by the retrieval model (318) to interpolate and extrapolate object trajectories during memory alignment.

[0054] Turning to FIG. 4.2, the retrieval model (422) may be a neural network model that processes current data and stored proposals to generate a memory proposal. The retrieval model (422) outputs a memory proposal that represents a temporally aligned and contextually enriched object representation. The retrieval model (422) includes the extrapolation model (425) and the feature extraction model (428), which perform predictive alignment and feature encoding respectively.

[0055] The extrapolation model (425) is a predictive model that estimates current object locations by extrapolating from a stored proposal to the current timestamp. The extrapolation model (425) receives historical object proposals and temporal offset data as input. The extrapolation model (425) outputs spatial coordinates and motion- adjusted features that represent the expected object state at the current time. The extrapolation model (425) may include linear motion models, temporal convolutional networks, or recurrent neural networks.

[0056] The extrapolation in the retrieval model (422) may be simplified, since the stored proposal (355) includes the predicted trajectory (410), the alignment step in the retrieval model (422) may interpolate the trajectory (410) using the difference between the stored proposal (355), the timestamp (418), and thecurrent timestamp (312). If the time difference goes beyond the trajectory horizon, straight line (constant velocity) extrapolation may be applied.

[0057] The feature extraction model (428) is a neural network model that processes bounding boxes and trajectories to generate object features. The feature extraction model (428) outputs feature vectors that encode spatial, temporal, and semantic characteristics of each object. The feature extraction model (428) includes the positional encoding model (430) and the feature encoding model (432), which generate embeddings that form the object features.

[0058] The positional encoding model (430) is a model that generates spatial and temporal embeddings for object proposals. The positional encoding model (430) receives object coordinates, frame indices, and motion vectors as input. The positional encoding model (430) outputs positional embeddings that encode location and time information in a format suitable for neural processing. The positional encoding model (430) may include sinusoidal encodings, learned embeddings, or coordinate transformation layers.

[0059] The feature encoding model (432) is a neural network model that processes the positional embeddings into object features. The feature encoding model (432) receives positional embeddings from the positional encoding model (430) and outputs encoded representations that integrate spatial, semantic, and temporal information. The feature encoding model (432) may include multilayer perceptrons, attention layers, or fusion networks.

[0060] Turning to FIG. 4.3, the detector model (452) is a neural network model that processes current data to generate detection proposals. The detector model (452) receives sensor measurements and object motion data as input. The detector model (452) outputs detection proposals that include object locations, dimensions, and classification scores. The detector model (452) includes the sensor model (455) and the feature extraction model (458), which perform bounding box generation and feature encoding respectively.

[0061] The sensor model (455) is a processing module that interprets sensor data to generate bounding boxes and trajectories. The sensor model (455) receives measurements from sensors such as LiDAR, radar, and cameras. The sensor model (455) may output spatial proposals and may output motion paths that represent detected objects and corresponding movements across frames. The sensor model (455) may include point cloud processing algorithms, image segmentation networks, or motion estimation filters.

[0062] The feature extraction model (458) is a neural network model that generates object features from bounding boxes and trajectories. The feature extraction model (458) receives spatial proposals from the sensor model (455). The feature extraction model (458) outputs feature vectors that encode spatial, temporal, and semantic characteristics of each object. The feature extraction model (458) may be the same as the feature extraction model (428) of FIG. 4.2 and may include positional encoding and feature encoding components.

[0063] Turning to FIG. 4.4, the merger model (472) is a neural network model that compares memory proposals and detection proposals to generate merged proposals. The merger model (472) includes the detector confidence model (475) and the memory confidence model (478), which generate confidence scores used to weight and reconcile overlapping proposals.

[0064] The detector confidence model (475) is a scoring module that generates confidence scores for detection proposals. The detector confidence model (475) outputs numerical scores that quantify the reliability of each detection proposal. The detector confidence model (475) may include multilayer perceptrons, logistic regression layers, or learned scoring functions.

[0065] The memory confidence model (478) is a scoring module that generates confidence scores for memory proposals. The memory confidence model (478) outputs numerical scores that quantify the reliability of each memory proposal. The memory confidence model (478) may have an architecture that is similar to the architecture of the detector confidence model (475).

[0066] Turning to FIG. 4.5, the refinement model (482) is a neural network model that processes features from memory proposals and merged proposals to generate refined object outputs. The refinement model (482) generates refined proposal features and outputs refined bounding boxes and object trajectories. The refinement model (482) includes the blocks (485) and the update model (492), which perform attention-based feature processing and output generation respectively.

[0067] The blocks (485) are neural network components within the refinement model (482) that perform attention-based feature transformation. The blocks (485) receive object features from memory proposals and merged proposals. The blocks (485) include the cross-attention layer (488) and the factorized selfattention layer (490), which perform inter-object and intra-object feature refinement.

[0068] The cross-attention layer (488) is a neural network layer that performs cross-attention between memory proposal features and merged proposal features. The cross-attention layer (488) computes attention weights that quantify feature relevance across proposals. The cross-attention layer (488) outputs fused features that integrate historical and current object information.

[0069] The factorized self-attention layer (490) is a neural network layer that performs self-attention across object features and time steps. The factorized self-attention layer (490) receives object features organized by time and identity. The factorized self-attention layer (490) performs self-attention for each object across time steps and for each time step across objects. The factorized self-attention layer (490) outputs temporally and spatially refined features.

[0070] The update model (492) is a neural network model that processes refined features from the blocks (485) to generate bounding boxes and trajectories. The update model (492) receives attention-refined features and outputs object-level predictions. The update model (492) includes the box confidence models (495)and the trajectory model (498), which generate spatial and motion outputs respectively.

[0071] The box confidence models (495) are neural network models that generate bounding boxes based on refined features. The box confidence models (495) process spatial and semantic features to output bounding box coordinates and confidence scores. The box confidence models (495) may include multilayer perceptrons, regression heads, or classification layers.

[0072] The trajectory model (498) is a neural network module that generates object trajectories based on refined features. The trajectory model (498) receives temporally encoded features and outputs motion paths for each object. The trajectory model (498) may include gated recurrent units (GRUs), temporal convolutional layers, or sequence modeling networks.

[0073] FIG. 5 shows a flowchart of a method for memory-augmented detection of objects. The method of FIG. 5 may be implemented using the systems described in the other figures, and one or more of the steps may be performed on, or received at, one or more computer processors. The system may include at least one processor and an application that, when executing on the at least one processor, performs the method. A non-transitory computer readable medium may include instructions that, when executed by one or more processors, perform the method. The outputs from various components (including models, functions, procedures, programs, processors, etc.) for performing the method may be generated by applying a transformation to inputs using the components to create the outputs without using mental processes or human activities.

[0074] Turning to FIG. 5, the method (500) updates a memory bank with refined proposals for objects. The process (500) may include multiple steps (e.g., Block 502 through Block 512) that may execute on the components described in the other figures, including those of FIG. 1, FIG. 8.1, and FIG. 8.2.

[0075] Block 502 involves generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model. The retrieval model accesses stored proposals from the memory bank and aligns the stored proposals with the current timestamp and current pose using trajectory forecasts and ego-pose transformations. The aligned proposals are encoded into feature vectors using a feature extraction model, resulting in memory proposals that include bounding boxes, trajectories, and latent features representing historical object states.

[0076] The method (500) may involve processing the stored proposal with an extrapolation model of the retrieval model to extrapolate and align bounding boxes and trajectories with a current pose and a current timestamp from the current data. The extrapolation model interpolates or extrapolates the stored trajectory forecasts to the current timestamp, adjusting for both object motion and ego-vehicle motion using the relative transform between the stored pose and the current pose. The alignment updates the spatial location of each stored proposal to reflect the expected position at the current time.

[0077] The method (500) may involve processing output from the extrapolation model with the feature extraction model to generate the memory proposal. The feature extraction model encodes the aligned bounding boxes and trajectories into high-dimensional feature vectors by applying positional encodings and learned embeddings. The features are combined with additional data, such as memory age and object dimensions, to form the final memory proposal.

[0078] The method (500) may involve processing bounding boxes and trajectories from the memory bank with a feature extraction model to generate memory proposal features for the memory proposal. The feature extraction model applies a multilayer perceptron (MLP) to the bounding box parameters, trajectory waypoints, and temporal data (e.g., timestamps) to generate feature vectors for each memory proposal. The features are used in downstream attention operations and scoring mechanisms.

[0079] The method (500) may involve processing sensor data from the current data with the sensor model to generate bounding boxes and trajectories. The sensor model interprets raw sensor inputs such as LiDAR point clouds or camera images to detect objects and estimate the spatial extent. The output includes bounding boxes and initial trajectory estimates, assuming static motion for initialization.

[0080] The method (500) may involve processing the bounding boxes and trajectories from the sensor model with a feature extraction model to generate detection proposal features for the detection proposal. The feature extraction model interpolates a feature map from the detector backbone at the location of each bounding box to generate feature vectors. The vectors are augmented with learned embeddings of the box parameters and classification scores.

[0081] Block 505 involves generating a detection proposal by processing the current data with a detector model. The detector model applies a neural network architecture to the current sensor data to produce object proposals including bounding boxes, classification scores, and feature vectors. The detection proposals represent objects observed in the current frame.

[0082] Block 508 involves generating a merged proposal by processing the memory proposal and the detection proposal with a merger model. The merger model compares detection and memory proposals using learned confidence scores and applies non-maximum suppression to remove redundant proposals. The output is a merged set of proposals selected based on calibrated confidence scores.

[0083] The method (500) may involve generating a detector confidence score for the detection proposal with a detector confidence model. The detector confidence model applies an MLP to the feature vector of each detection proposal to produce a multi-class confidence score. The score reflects the likelihood that the proposal corresponds to a true object.

[0084] The method (500) may involve generating a memory confidence score for the memory proposal with a memory confidence model. The memory confidence model applies a separate MLP to the feature vector of each memory proposal, incorporating memory age and trajectory uncertainty. The resulting score quantifies the reliability of the memory proposal at the current time.

[0085] The method (500) may involve comparing the detector confidence score with the memory confidence score to select one of the memory proposal and the detection proposal as the merged proposal. The merger model compares the scores and retains the proposal with the higher confidence, resolving overlapping proposals in favor of the most reliable source. The selection is used to construct the merged proposal set for refinement.

[0086] Block 510 involves generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model. The refinement model uses a cross-attention layer that attends to queries from the merged proposal features. The refinement model applies keys and values derived from the memory proposal features during the cross-attention step. Additionally, the refinement model uses a factored self-attention layer. The factored self-attention layer performs time self-attention by attending to queries for an object across different time steps and performs object self-attention by attending to queries for each of the objects within a single time step. The refinement model applies cross-attention between memory and merged proposal features to enrich the merged proposals with historical context. Factorized self-attention is applied by attending across time steps for each object and across objects for each time step for efficient temporal and spatial reasoning.

[0087] The method (500) may involve generating one or more of the memory proposal, the stored proposal, the detection proposal, the merged proposal, and the refined proposal as an object proposal including a bounding box, a confidence score, a trajectory, proposal features, a pose, and a timestamp. Eachproposal is represented as a structured tuple including spatial coordinates, classification scores, predicted motion paths, latent features, and temporal data. The proposals are used throughout the detection and refinement pipeline.

[0088] Block 512 involves updating multiple stored proposals including the stored proposal by storing the refined proposal to the memory bank. The refined proposals are appended to the memory bank along with the current timestamp and pose. Older entries are removed to maintain a bounded memory size and increase relevance.

[0089] The method (500) may involve training one or more of the retrieval model, the detector model, the merger model, and the refinement model on progressively longer sequences of data with the training beginning with an initial sequence length that increases during the training. The training schedule begins with single-frame examples to maximize data diversity and gradually increases sequence length to match inference conditions. The curriculum improves generalization and temporal consistency.

[0090] The method (500) may involve training one or more of the retrieval model, the detector model, the merger model, and the refinement model using a cache to store intermediate proposals and features generated during training. A memory cache is maintained per training log to store model outputs for memory proposal generation during short-sequence training. The cache is updated after each iteration for memory availability throughout training.

[0091] The method (500) may involve activating a cache during training after meeting a training threshold including one of a predefined number of training iterations or a convergence criterion based on validation loss. The cache is activated after a warm-up phase to avoid storing inaccurate early outputs. The threshold improves the reliability of memory proposals used during training.

[0092] The method (500) may involve training the refinement model using a combination of a re-scoring loss, a detection refinement loss, and a forecasting refinement loss. The detection and forecasting losses may be computed at eachblock in the refinement model. The re-scoring loss is a focal loss for merged multi-class confidence scores. The detection refinement loss is combination of a binary focal loss for classification, an L1 loss for regression, and IoU loss. The forecasting refinement loss is an L1 loss against ground-truth trajectories, supervised for true-positive detections with IoU that may be greater than 0.5.

[0093] Turning to FIG. 6, memory-augmented detection may be implemented as a plug-and-play module that enhances an off-the-shelf 3D detector (kept frozen) with long-term memory. As an example, the module (600) enhances the detector (602).

[0094] 3D object detectors operate on a short temporal window of sensor data and produce a set of detections. Existing approaches may struggle with occlusion and distant objects that have limited sensor observations. Memory- augmented detection is introduced as a plug-and-play module that augments existing 3D object detectors with long-horizon temporal fusion capabilities. Each detection includes an object bounding box, multi-class confidence scores, and a feature vector capturing local context. The generality of memory- augmented detection is demonstrated by augmenting and improving various LiDAR-based and camera-based detectors.

[0095] Long-horizon temporal understanding is achieved through a flexible memory that stores relevant information about objects, including predicted motion. Trajectory forecasts are used to align memory objects in space and time with current detector proposals. Motion forecasts are computed by memory- augmented detection and are not required from the object detector. To account for self-driving vehicle motion, the system assumes localization is available and stores the vehicle pose in the memory along with model outputs.

[0096] At each inference step, memory-augmented detection receives as input the detection proposals, the current timestamp t (e.g., LiDAR sweep-end time or camera capture time), and the self-driving vehicle (SDV) pose Etin a global coordinate frame. Objects are retrieved from the memory bank and alignedspatially with Etand temporally to t. High-dimensional features are extracted from the aligned bounding boxes and trajectory forecasts. The aligned bounding boxes and trajectory forecasts, together with the extracted features, are referred to as memory proposals. A proposal merging mechanism fuses the detection proposals and memory proposals by re-scoring the confidence scores and applying standard post-processing. Object detections and trajectory forecasts in the merged proposals are iteratively refined using a transformer that applies cross-attention to the memory and factorized self-attention. Model outputs, referred to as refined proposals, are appended to the memory bank, and older model outputs are removed to maintain a bounded memory size.

[0097] An object proposal is defined as TT = (b, C, T, q), where b = (x,y,z,w, l,h,6') represents the object’s 3D centroid (%,y, z), heading 6 in a Bird’s-Eye-View (BEV) SDV-relative frame, and 3D box dimensions (w, I, ). Multi-class confidence scores are represented as c ∈ RC, where C is the number of actor classes. A trajectory forecast is represented as T = y>t+Sf, — > (,x> y> )t+7ys / ]> comprising a sequence of T waypoints at a time interval s. An object feature is represented as q ∈ R(Tf+1)×dencoding both local and global features, where d is the feature dimensionality. Superscripts are used to denote the source of the proposals: detection proposal πdetfrom the 3D detector, memory proposal πmergefrom the memory bank, merged proposal πrefrom the proposal merging module, and refined proposal πref from the output of the refinement transformer. A set of N proposals is denoted as P = (B, C, T, Q with tensors of boxes, scores, trajectory forecasts, and features represented as B E RNx7, C E RNxC, T E RNxTfx3’,and Q E RNx(.Tf+y)xd respectively. For each detection proposal πdet∈ Pdettrajectory Tdetinitialized by assuming the object remains static over time, due to the absence of forecast data from the detector. The object feature qdetobtained by interpolating the feature map prior to the detector header at the projected objectcentroid, repeating (Tf + 1) times to generate features for future timesteps, and adding a learned embedding of bdetand cdet, subsequent paragraphs describe the procedures for obtaining pmem, pmerge, and pref

[0098] The memory bank is a set of tuples (timestamp tm, SDV pose Etm, model outputssorted by the timestamp tmat which the outputs were generated. During inference at timestamp t, memory entries P^ are retrieved at a set of past target timestamps tme Tm, where Tm= {t — sm, t — 2sm...,t — Tmsm}. Tmdenotes the number of past target timestamps, and smdenotes the time stride of the retrieved entries. For tolerance with small sensor delays, the closest memory entry to each timestamp inmis retrieved.

[0099] For effective use of the memory at inference, each retrieved entry(tm, Etm, is aligned in space and time with the detection proposals at time t. SDV motion is handled by applying the relative transform Et→t = Et-1Etto Brtand Trt^. Object motion is handled by linearly interpolating the stored trajectory forecast to the current timestamp t to obtain the proposal box bmemto obtainproposal forecast Tmem5the stored trajectories are interpolated or extrapolated to produce waypoints at {t + s,..., t + T-s ] from stored waypoints at {tm+ s, tm+ TfSf. Latent features Qmemare extracted at t and at each future time step ty G Tf by computing sine / cosine positional embeddings for the centroid coordinates Bmemand encoding the positional embeddings with a lightweight MLP. Additional features including Bmem, the memory age t — tm, and a 2D vector indicating the location of the proposal in the current SDV frame at time tmare concatenated and encoded with another MLP. The outputs of both MLPs are summed to produce the final feature representation.

[0100] Memory proposals and detection proposals may be redundant, particularly in regions with high sensor coverage. To merge proposals, multi-class confidence scores are re-scored using learned models. The reliability of a memory proposal is based on the confidence score at a past timestampand on the proposal age t — tm, due to increasing forecasting uncertainty over time. For example, a fast-moving detection is less reliable than a stationary object, and a proposal observed 0.5 seconds ago is more trustworthy than one observed 5 seconds ago. Detection proposals originate from the 3D detector, while memory proposals are generated by memory-augmented detection, and detector outputs may be miscalibrated.

[0101] To calibrate scores across sources, two small MLPs are trained to separately map the features of detection proposals Qdetand memory proposals Qmemto new multi-class scores Cmerge. Training is performed under a unified detection loss applied to the merged proposals, allowing the model to learn which proposals to trust. Proposals are filtered using score thresholding and non-maximum suppression (NMS), and the top K merged proposals are retained, sorted by score (maximum over actor classes). The reduced proposal set allows the refinement transformer to process a limited number of queries.

[0102] Learned time positional embeddings are added to the features of merged proposals Qmergeto encode the time of the future trajectory forecast. At this stage, the merged proposals pmergeare prepared for refinement, with pmerge→ pref

[0103] A refinement transformer may be used that includes a transformer decoder. The transformer decoder refines the merged proposals pmerge= prefW iteratively over B blocks into Pref(1)...pref(B)where the final model outputs are Pref= pref(B)A memory cross-attention mechanism uses the proposal features Qref(l)to aggregate information from memory proposal features Qmem, including memory proposals that were filtered out during merging. Information from overlapping memory proposals includes evidence about object presence and location that supports refinement. Keys and values are constructed by projecting the memory proposal features, and cross-attention• • • Qref(l)is performed between present-time object queries Qt=0and the projected keys / values. To reduce computational cost, cross-attention is limited to the nearest k keys for each object query, determined by computing nearest neighbors of Bmemin Bmem. Factorized self-attention is performed in each refinement block, separating time self-attention and object self-attention for computational efficiency. Time self-attention attends to queries from the same object across a sequence length of Tf + 1, while object self-attention attends to queries from the same time step across a sequence length of N. Updated queries Qref(l+1)are passed to the next refinement block. Explicit proposal information is updated using a multilayer perceptron to produce Bref(l+1)and Cref(l+1)and a gated recurrent unit (GRU) is used to update future trajectory waypoints

[0104] The memory bank is updated. Refined proposals Prefare postprocessed using score thresholding, non-maximum suppression (NMS), and top-K filtering based on confidence scores to obtain Pref. The result Prefis added to the memory bank along with the corresponding timestamp t and self-driving vehicle pose Et. To maintain a bounded memory size during long-sequence inference, memory entries older than t — Tmsm— are removed, where Tmsmdefines the memory retrieval horizon and 6 is a buffer margin.

[0105] Training begins by applying the proposed training strategy of an off-the- shelf 3D detection model. Pre-training and freezing the 3D detector keeps the detection proposals consistent throughout memory-augmented detection training. If pre-trained weights for the 3D detector are available, the initial training stage may be omitted. Subsequent training is applied to the parameters of the memory-augmented detection module, while the 3D detector weights remain frozen.

[0106] Prior to describing the training procedure for memory-augmented detection, the space of training configurations and trade-offs for temporalfusion models is considered. Training on unordered examples with individual frames or short chunks of sequential frames satisfies the assumption of independent and identically distributed examples, which improves learning dynamics. However, such training configurations introduce a gap to evaluation, where the model is deployed on long sequences and consumes its own prior outputs. Training on long sequences of ordered data reduces the gap to evaluation but results in worse learning dynamics due to high correlation between consecutive examples. Accumulating gradients over a long sequence and updating model parameters once per sequence satisfies the i.i.d. assumption but increases training duration proportionally to sequence length if the number of updates is held constant. Despite the importance of training configuration choices, prior works on learned temporal fusion omit detailed discussion of training methodology.

[0107] To address the challenges of temporal fusion training, a training schedule is introduced. Training is performed on increasingly long chunks of ordered data, beginning with single frames and progressing to full sequences. To support object memory training on short chunks or single frames, a cache of memory banks is maintained throughout training. The cache is used to construct memory proposals for each training example, enabling temporal fusion even when training data lacks long sequences. Details of the training schedule, memory bank cache, memory-aware augmentations, and loss functions are described in subsequent sections.

[0108] During the first 25% of training, sampling is performed on individual frames, where consecutive examples are drawn as random frames from random driving snippets. For the remaining 75% of training, sequence chunks are sampled with progressively increasing length: 48 frames during the interval (25%, 50%], 96 frames during (50%, 75%], and 144 frames during (75%, 100%]. Training is performed using a cosine decay learning rate schedule without resets. High learning rates correspond to training on diverse andindependent examples, which reduces overfitting associated with sequential data. Low learning rates correspond to training on longer sequences, which more closely reflect the evaluation setting where prior outputs are consumed.

[0109] Following the schedule described above without modification results in a training regime where prior outputs are not consumed during early phases, preventing learning of memory usage. To address the limitation, a cache of memory banks is introduced. The cache maps each unique driving log identifier (logID) to a corresponding memory bank. Each driving log includes 10-20 seconds of continuous driving data organized by the dataset. At the beginning of training, the cache is empty. During training, the cache is populated with model outputs. For each training iteration, the memory bank corresponding to the current logID is retrieved from the cache if available, and memory proposals are constructed from the cached memory bank. At the end of each iteration, the cache is updated with the latest model outputs. If older outputs exist for a given logID, the outputs are replaced with the most recent outputs.

[0110] Training with an object memory cache introduces several challenges.Efficient training on large datasets uses a distributed data training scheme in which examples in each minibatch are split across multiple GPUs. Each GPU is assigned a unique index referred to as a rank. To avoid excessive RAM usage and synchronization overhead, each rank maintains a separate cache. For high cache hit rates, frames associated with a given logID are consistently sampled on the same rank throughout training.

[0111] The memory bank is populated with outputs from the memory-augmented detection model. Early in training, the outputs are inaccurate and may degrade the reliability of memory proposals if stored prematurely. To prevent degradation, cache population, and training with memory proposals begin after 2.5% of training has been completed, at which point model performance reaches a reasonable threshold.

[0112] To improve robustness against variable latency and the intermittent availability of memory proposals, the target timestamps used for memory retrieval during training are randomized. The randomization is achieved by sampling both the time stride and the number of target timestamps used to retrieve memory elements.

[0113] Data augmentations such as translation, rotation, flipping, and re-scaling are applied to improve detection performance. Augmentations are applied to the bounding boxes and trajectories of memory proposals after the memory alignment step. Before storing refined proposals in the memory bank, the inverse of the augmentations is applied to the bounding boxes and trajectories to maintain consistency across memory entries.

[0114] The training objective is a multi-task loss function defined asBL = LresCOre{Cmer^ + Ldet(Bre^, + Lfor(Tre^)i=l The objective combines a re-scoring loss, a detection refinement loss, and a forecasting refinement loss. The detection and forecasting losses are computed at each refinement block. The detection refinement loss includes a binary focal loss for classification, an LI loss for regression, and an intersection-over-union (loU) loss. Targets for the losses are determined by matching detections to ground truth bounding boxes using bipartite matching. The re-scoring loss includes a focal loss applied to the multi-class scores output by the re-scoring module. The forecasting refinement loss is an LI loss computed against the ground-truth trajectory and is supervised for true-positive detections with an loU greater than 0.5.

[0115] Turning to FIG. 7, the detector view (702) is generated using a 3D detector, which may include a false negative and display multiple detector proposals for objects surrounding an autonomous system. In the detector view(702), sensor data points may be shown in gray, with detections shown in green and labels shown in black.

[0116] The object memory view (705) shows objects identified from memory proposals stored in a memory bank for objects surrounding an autonomous system. In the object memory view (705), color may be used to indicate the age of memory proposals, with magenta representing older proposals and cyan representing newer proposals retrieved from the memory bank.

[0117] The memory-augmented detection view (708) shows refined proposals generated from the detector proposals and the memory proposals for objects surrounding an autonomous system. In the memory -augmented detection view (708), sensor data points may be shown in gray, with detections shown in green, and labels shown in black.

[0118] One or more embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure.

[0119] For example, as shown in FIG. 8.1, the computing system (800) may include one or more computer processor(s) (802), non-persistent storage device(s) (804), persistent storage device(s) (806), a communication interface (808) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (802) may be an integrated circuit for processing instructions. The computer processor(s) (802) may be one or more cores, or micro-cores, of a processor. The computer processor(s) (802) includes one or more processors. Thecomputer processor(s) (802) may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0120] The input device(s) (810) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device(s) (810) may receive inputs from a user that are responsive to data and messages presented by the output device(s) (812). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (800) in accordance with one or more embodiments. The communication interface (808) may include an integrated circuit for connecting the computing system (800) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN), such as the Internet, mobile network, or any other type of network) or to another device, such as another computing device, and combinations thereof.

[0121] Further, the output device(s) (812) may include a display device, a printer, external storage, or any other output device. One or more of the output device(s) (812) may be the same or different from the input device(s) (810). The input device(s) (810) and output device(s) (812) may be locally or remotely connected to the computer processor(s) (802). Many different types of computing systems exist, and the aforementioned input device(s) (810) and output device(s) (812) may take other forms. The output device(s) (812) may display data and messages that are transmitted and received by the computing system (800). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0122] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium, such as a solid state drive (SSD), compact disk (CD), digital video disk (DVD), storage device,a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by the computer processor(s) (802), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0123] The computing system (800) in FIG. 8.1 may be connected to, or be a part of, a network. For example, as shown in FIG. 8.2, the network (820) may include multiple nodes (e.g., node X (822) and node Y (824), as well as extant intervening nodes between node X (822) and node Y (824)). Each node may correspond to a computing system, such as the computing system shown in FIG. 8.1, or a group of nodes combined may correspond to the computing system shown in FIG. 8.1. By way of an example, embodiments may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (800) may be located at a remote location and connected to the other elements over a network.

[0124] The nodes (e.g., node X (822) and node Y (824)) in the network (820) may be configured to provide services for a client device (826). The services may include receiving requests and transmitting responses to the client device (826). For example, the nodes may be part of a cloud computing system. The client device (826) may be a computing system, such as the computing system shown in FIG. 8.1. Further, the client device (826) may include or perform all or a portion of one or more embodiments.

[0125] The computing system of FIG. 8.1 may include functionality to present data (including raw data, processed data, and combinations thereof), such as results of comparisons and other processing. For example, presenting data maybe accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown, as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0126] As used herein, the term “connected to” contemplates multiple meanings.A connection may be direct or indirect (e.g, through another component or network). A connection may be wired or wireless. A connection may be a temporary, permanent, or a semi-permanent communication channel between two entities.

[0127] The various descriptions of the figures may be combined and may include, or be included within, the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, or altered as shown in the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0128] In the application, ordinal numbers (e.g, first, second, third, etc.) may be used as an adjective for an element (z.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, ordinal numbers distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0129] Further, unless expressly stated otherwise, the conjunction “or” is an inclusive “or” and, as such, automatically includes the conjunction “and”, unless expressly stated otherwise. Further, items joined by the conjunction “or” may include any combination of the items with any number of each item, unless expressly stated otherwise.

[0130] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.

Claims

CLAIMSWhat is claimed is:

1. A method comprising:generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model;generating a detection proposal by processing the current data with a detector model;generating a merged proposal by processing the memory proposal and the detection proposal with a merger model;generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model using a cross-attention layer attending to queries of the merged proposal features of the merged proposal using keys and values of the memory proposal features of the memory proposal and using a factored self-attention layer performing time self-attention attending to queries for an object and performing object self-attention attending to queries for a time step; andupdating a plurality of stored proposals comprising the stored proposal by storing the refined proposal to the memory bank.

2. The method of claim 1, further comprising:processing the stored proposal with an extrapolation model of the retrieval model to extrapolate and align bounding boxes and trajectories with a current pose and a current timestamp from the current data, andprocessing output from the extrapolation model with the feature extraction model to generate the memory proposal.

3. The method of claim 1, further comprising:processing sensor data from the current data with the sensor model to generate bounding boxes and trajectories, andprocessing the bounding boxes and trajectories from the sensor model with a feature extraction model to generate detection proposal features for the detection proposal.

4. The method of claim 1, further comprisingprocessing bounding boxes and trajectories from the memory bank with a feature extraction model to generate memory proposal features for the memory proposal.

5. The method of claim 1, further comprising:generating a detector confidence score for the detection proposal with a detector confidence model;generating a memory confidence score for the memory proposal with a memory confidence model;comparing the detector confidence score with the memory confidence score to select one of the memory proposal and the detection proposal as the merged proposal.

6. The method of claim 1, further comprising:generating one or more of the memory proposal, the stored proposal, the detection proposal, the merged proposal, and the refined proposal as an object proposal comprising a bounding box, a confidence score, a trajectory, proposal features, a pose, and a timestamp.

7. The method of claim 1, further comprising:training one or more of the retrieval model, the detector model, the merger model, and the refinement model on progressively longer sequences of data with the training beginning with an initial sequence length that increases during the training.

8. The method of claim 1, further comprising:training one or more of the retrieval model, the detector model, the merger model, and the refinement model using a cache to store intermediate proposals and features generated during training.

9. The method of claim 1, further comprising:activating a cache during training after meeting a training threshold comprising one of a predefined number of training iterations or a convergence criterion based on validation loss.

10. The method of claim 1, further comprising:training the refinement model using a combination of a re-scoring loss, a detection refinement loss, and a forecasting refinement loss.

11. A system comprising:a computer processor; andan application that, when executing on the computer processor, performs operations comprising:generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model,generating a detection proposal by processing the current data with a detector model,generating a merged proposal by processing the memory proposal and the detection proposal with a merger model,generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model using a cross-attention layer attending to queries of the merged proposal features of the merged proposal using keys and values of the memory proposal features of the memory proposal and using a factored self-attention layer performing time self-attention attending to queries for an object and performing object self-attention attending to queries for a time step,updating a plurality of stored proposals comprising the stored proposal by storing the refined proposal to the memory bank.

12. The system of claim 11, wherein the application performs operations further comprising:processing the stored proposal with an extrapolation model of the retrieval model to extrapolate and align bounding boxes and trajectories with a current pose and a current timestamp from the current data, andprocessing output from the extrapolation model with the feature extraction model to generate the memory proposal.

13. The system of claim 11, wherein the application performs operations further comprising:processing sensor data from the current data with the sensor model to generate bounding boxes and trajectories, andprocessing the bounding boxes and trajectories from the sensor model with a feature extraction model to generate detection proposal features for the detection proposal.

14. The system of claim 11, wherein the application performs operations further comprising:processing bounding boxes and trajectories from the memory bank with a feature extraction model to generate memory proposal features for the memory proposal.

15. The system of claim 11, wherein the application performs operations further comprising:generating a detector confidence score for the detection proposal with a detector confidence model,generating a memory confidence score for the memory proposal with a memory confidence model,comparing the detector confidence score with the memory confidence score to select one of the memory proposal and the detection proposal as the merged proposal.

16. The system of claim 11, wherein the application performs operations further comprising:generating one or more of the memory proposal, the stored proposal, the detection proposal, the merged proposal, and the refined proposal as an object proposal comprising a bounding box, a confidence score, a trajectory, proposal features, a pose, and a timestamp.

17. The system of claim 11, wherein the application performs operations further comprising:training one or more of the retrieval model, the detector model, the merger model, and the refinement model on progressively longer sequences of data with the training beginning with an initial sequence length that increases during the training.

18. The system of claim 11, wherein the application performs operations further comprising:training one or more of the retrieval model, the detector model, the merger model, and the refinement model using a cache to store intermediate proposals and features generated during training.

19. The system of claim 11, wherein the application performs operations further comprising:activating a cache during training after meeting a training threshold comprising one of a predefined number of training iterations or a convergence criterion based on validation loss.

20. A non-transitory computer readable medium comprising instructions executable by a computer processor to perform:generating a memory proposal by processing current data and a stored proposal from a memory bank with a retrieval model;generating a detection proposal by processing the current data with a detector model;generating a merged proposal by processing the memory proposal and the detection proposal with a merger model;generating a refined proposal by processing the memory proposal and the merged proposal with a refinement model using a cross-attention layer attending to queries of the merged proposal features of the merged proposal using keys and values of the memory proposal features of the memory proposal and using a factored self-attention layer performing time self-attention attending to queries for an object and performing object self-attention attending to queries for a time step; andupdating a plurality of stored proposals comprising the stored proposal by storing the refined proposal to the memory bank.

Images