Generating realistic three-dimensional assets in latent space

Abstract

Generating three-dimensional assets in latent space includes performing operations. The operations include generating an asset latent code for the real asset. A decoder model processes the asset latent code to generate an asset model. A sensor simulation model processes the asset model to generate predicted sensor data. The asset latent code for the real asset is trained based on a comparison of the predicted sensor data with the real sensor data. Noise is added to the trained asset latent code to generate a noisy asset latent code. A diffusion model processes the noisy asset latent code to generate predicted asset latent code. The operations further include training the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model and deploying the trained diffusion model.

Description

GENERATING REALISTIC THREE-DIMENSIONAL ASSETS IN LATENT SPACE CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a non-provisional application of, and thereby claims benefit to, U.S. Patent Application Serial No. 63 / 720,375 filed on November 14, 2024. U.S. Patent Application Serial No. 63 / 720,375 is incorporated herein by reference in its entirety.BACKGROUND

[0002] A virtual world is an environment in which a player may move in three dimensions as if the player were in the real world. The player operates independently of the instructions of the virtual world. In addition to the player, the virtual world may also have computerized assets that are part of the virtual world. Some of the assets may be stationary and some of the assets may also move through the virtual world.

[0003] A goal of the virtual world is to be realistic or to have some elements of realism. Thus, even if the virtual world is not a replica of the real world, the virtual world has elements that could be seen in the real world. One source of realism is in the accuracy of the assets in the virtual world. Even new assets that do not exist in the real world should be realistic. For example, assets that are sedans should have four wheels, windows, and shaped as a sedan. Thus, a goal of asset generation is to create realistic assets.SUMMARY

[0004] In general, in one aspect, embodiments relate to a method that includes operations. The operations include obtaining a real sensor data for a real asset and generating an asset latent code for the real asset. A decoder model processes the asset latent code to generate an asset model. A sensor simulation model processes the asset model to generate predicted sensor data. The operations also include training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code, adding noise to the trained asset latent code to generate a noisy asset latentcode, and processing, by a diffusion model, the noisy asset latent code to generate predicted asset latent code. The operations further include training the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model and deploying the trained diffusion model.

[0005] In general, in one aspect, embodiments relate to a system that includes memory and a computer processor comprising computer readable program code for performing operations. The operations include obtaining a real sensor data for a real asset and generating an asset latent code for the real asset. A decoder model processes the asset latent code to generate an asset model. A sensor simulation model processes the asset model to generate predicted sensor data. The operations also include training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code, adding noise to the trained asset latent code to generate a noisy asset latent code, and processing, by a diffusion model, the noisy asset latent code to generate predicted asset latent code. The operations further include training the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model and deploying the trained diffusion model.

[0006] In general, in one aspect, embodiments relate to a non-transitory computer readable medium comprising computer readable program code for performing operations. The operations include obtaining a real sensor data for a real asset and generating an asset latent code for the real asset. A decoder model processes the asset latent code to generate an asset model. A sensor simulation model processes the asset model to generate predicted sensor data. The operations also include training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code, adding noise to the trained asset latent code to generate a noisy asset latent code, and processing, by a diffusion model, the noisy asset latent code to generate predicted asset latent code. The operations further includetraining the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model and deploying the trained diffusion model.

[0007] Other aspects of one or more embodiments will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1 shows a diagram of a simulator for testing or training an autonomous system with a virtual driver in accordance with one or more embodiments.

[0009] FIG. 2 shows a flowchart for performing a simulation for training a virtual driver in accordance with one or more embodiments of the invention.

[0010] FIG. 3 shows a diagram of a model generator in accordance with one or more embodiments of the invention.

[0011] FIG. 4 is a flowchart for generating a trained asset latent code in accordance with one or more embodiments.

[0012] FIG. 5 is a flowchart for training the diffusion model with the trained asset latent code in accordance with one or more embodiments.

[0013] FIG. 6 is a flowchart for deploying a diffusion model in accordance with one or more embodiments.

[0014] FIG. 7 shows an example implementation in accordance with one or more embodiments.

[0015] FIG. 8 shows an example in accordance with one or more embodiments.

[0016] FIG. 9.1 and FIG. 9.2 show a computing system in accordance with one or more embodiments.

[0017] Like elements in the various figures are denoted by like reference numerals for consistency.DETAILED DESCRIPTION

[0018] In general, embodiments are directed to a diffusion model for performing realistic asset generation. Specifically, one or more embodiments are directed to training a diffusion model to generate asset latent code for realistic assets from an asset latent code structure initialized with at least some noise. The asset latent code may then be processed by a decoder model to generate an asset model that is a model of the asset. Training the diffusion model is performed in an at least two-stage process. During an asset latent code training stage, asset latent code is generated for a corresponding real-world asset. In other words, the system learns a realistic asset latent code that corresponds to a realistic asset. The realistic asset latent code may be generated from noise. Over the course of multiple assets, training data in the form of asset latent codes are generated.Another stage of training the diffusion model is to use the generated asset latent codes combined with noise as training input to a diffusion model. The diffusion model is then trained to generate the asset latent codes without the noise.Through training the diffusion model, the diffusion model learns how to transform noisy data to a realistic asset latent code. Thus, during an inference phase, noise is transformed by the diffusion model into asset latent code for a new asset. The new asset, after processed by the decoder, may then be populated into a virtual world and used to generate simulated sensor data.

[0019] The virtual world is an environment in which at least one independent player may move in three dimensions as if the player were in the real world. The player in the virtual world may be a human, a virtual driver of an autonomous system, or other computer software. The virtual world may also include computerized agents that may also move through the virtual world like the player. To be realistic, assets and other agents should appear in the virtual world similar to how agents might appear in the real world. For example, if an agent is a car, the agent should look like a car.

[0020] In some applications, hundreds of thousands of initial scenes for various scenarios could be generated within a shortened time limit that is cost prohibitivefor humans to generate. Moreover, in some cases, humans have biases that do not generate fully randomized scenes. In such cases, the assets may be skewed to a preset set of properties. One or more embodiments include a diffusion model that performs asset generation. The diffusion model may learn how to transform randomly generated asset representations into realistic sets of assets that could potentially exist in the real world.

[0021] Embodiments of the invention may be used as part of generating a simulated environment for the training and testing of autonomous systems. An autonomous system is a self-driving mode of transportation that does not require a human pilot or human driver to move and react to the real-world environment. Rather, the autonomous system includes a virtual driver that is the decisionmaking portion of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world. The autonomous system may be completely autonomous or semi-autonomous. As a mode of transportation, the autonomous system is contained in a housing configured to move through a real-world environment. Examples of autonomous systems include self-driving vehicles (e.g., self-driving trucks and cars), drones, airplanes, robots, etc. The virtual driver is the software that makes decisions and causes the autonomous system to interact with the real world including moving, signaling, and stopping or maintaining a current state.

[0022] The real-world environment is the portion of the real world through which the autonomous system, when trained, is designed to move. Thus, the real- world environment may include interactions with concrete and land, people, animals, other autonomous systems, human driven systems, construction, and other objects as the autonomous system moves from an origin to a destination. In order to interact with the real-world environment, the autonomous system includes various types of sensors, such as LiDAR sensors amongst other types, which are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment.

[0023] The testing and training of the virtual driver of the autonomous systems in the real-world environment is unsafe because of the accidents that an untrained virtual driver can cause. Thus, as shown in FIG. 1, a simulator (100) is configured to train and test a virtual driver (102) of an autonomous system. For example, the simulator may be a unified, modular, mixed-reality, closed-loop simulator for autonomous systems. The simulator (100) is a configurable simulation framework that enables not only evaluation of different autonomy components in isolation but also as a complete system in a closed-loop manner. The simulator reconstructs “digital twins” of real-world scenarios automatically, enabling accurate evaluation of the virtual driver at scale. The simulator (100) may also be configured to perform mixed-reality simulation that combines real- world data and simulated data to create diverse and realistic evaluation variations to provide insight into the virtual driver’s performance. The mixed reality closed- loop simulation allows the simulator (100) to analyze the virtual driver’s action on counterfactual “what-if’ scenarios that did not occur in the real world. The simulator (100) further includes functionality to simulate and train on rare yet safety-critical scenarios with respect to the entire autonomous system and closed- loop training to enable automatic and scalable improvement of autonomy.

[0024] The simulator (100) creates the simulated environment (104) which is a virtual world. The virtual driver (102) is the player in the virtual world. The simulated environment (104) is a simulation of a real-world environment, which may or may not be in actual existence, in which the autonomous system is designed to move. As such, the simulated environment (104) includes a simulation of the objects (i.e., simulated objects or assets) and background in the real world, including the natural objects, construction, buildings and roads, obstacles, as well as other autonomous and non-autonomous objects. The simulated environment simulates the environmental conditions within which the autonomous system may be deployed. Additionally, the simulated environment (104) may be configured to simulate various weather conditions that may affect the inputs to the autonomous systems. The simulated objects may include bothstationary and nonstationary objects. Nonstationary objects are agents in the real- world environment.

[0025] The simulator (100) also includes an evaluator (110). The evaluator (110) is configured to train and test the virtual driver (102) by creating various scenarios in the simulated environment. Each scenario is a configuration of the simulated environment including, but not limited to, static portions, movement of simulated objects, actions of the simulated objects with each other, and reactions to actions taken by the autonomous system and simulated objects. The evaluator (110) is further configured to evaluate the performance of the virtual driver using a variety of metrics.

[0026] The evaluator (110) assesses the performance of the virtual driver throughout the performance of the scenario. Assessing the performance may include applying rules. For example, the rules may be that the automated system does not collide with any other agent, compliance with safety and comfort standards (e.g., passengers not experiencing more than a certain acceleration force within the vehicle), the automated system not deviating from executed trajectory), or other rule. Each rule may be associated with the metric information that relates a degree of breaking the rule with a corresponding score. The evaluator (110) may be implemented as a data-driven neural network that learns to distinguish between good and bad driving behavior. The various metrics of the evaluation system may be leveraged to determine whether the automated system satisfies the requirements of the success criterion for a particular scenario. Further, in addition to system level performance, for modular based virtual drivers, the evaluator may also evaluate individual modules such as segmentation or prediction performance for agents in the scene with respect to the ground truth recorded in the simulator.

[0027] The simulator (100) is configured to operate in multiple phases as selected by the phase selector (108) and modes as selected by a mode selector (106). The phase selector (108) and mode selector (106) may be a graphical user interface or application programming interface component that is configured to receive a selection of phase and mode, respectively. The selected phase and modedefine the configuration of the simulator (100). Namely, the selected phase and mode define which system components communicate and the operations of the system components.

[0028] The phase may be selected using a phase selector (108). The phase may be a training phase or a testing phase. In the training phase, the evaluator (110) provides metric information to the virtual driver (102), which uses the metric information to update the virtual driver (102). The evaluator (110) may further use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver. In the testing phase, the evaluator (110) does not provide the metric information to the virtual driver. In the testing phase, the evaluator (110) uses the metric information to assess the virtual driver and to develop scenarios for the virtual driver (102).

[0029] The mode may be selected by the mode selector (106). The mode defines the degree to which real -world data is used, whether noise is injected into simulated data, the degree of perturbations of real-world data, and whether the scenarios are designed to be adversarial. Example modes include open-loop simulation mode, closed-loop simulation mode, single module closed-loop simulation mode, fuzzy mode, and adversarial mode. In an open-loop simulation mode, the virtual driver is evaluated with real-world data. In a single module closed-loop simulation mode, a single module of the virtual driver is tested. An example of a single module closed-loop simulation mode is a localizer closed- loop simulation mode in which the simulator evaluates how the localizer estimated pose drifts over time as the scenario progresses in simulation. In a training data simulation mode, simulator is used to generate training data. In a closed-loop evaluation mode, the virtual driver and simulation system are executed together to evaluate system performance. In the adversarial mode, the agents are modified to perform adversarial. In the fuzzy mode, noise is injected into the scenario (e.g., to replicate signal processing noise and other types of noise). Other modes may exist without departing from the scope of the system.

[0030] The simulator (100) includes the controller (112) which includes functionality to configure the various components of the simulator (100) according to the selected mode and phase. Namely, the controller (112) may modify the configuration of each of the components of the simulator based on the configuration parameters of the simulator (100). Such components include the evaluator (110), the simulated environment (104), an autonomous system model (116), sensor simulation models (114), asset models (117), agent models (118), latency models (120), and a training data generator (122).

[0031] The autonomous system model (116) is a detailed model of the autonomous system in which the virtual driver will execute. The autonomous system model (116) includes model, geometry, physical parameters (e.g., mass distribution, points of significance), engine parameters, sensor locations and type, the firing pattern of the sensors, information about the hardware on which the virtual driver executes (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. The various parameters of the autonomous system model may be configurable by the user or another system.

[0032] For example, if the autonomous system is a motor vehicle, the modeling and dynamics may include the type of vehicle (e.g., car, truck), make and model, geometry, physical parameters such as the mass distribution, axle positions, type and performance of the engine, etc. The vehicle model may also include information about the sensors on the vehicle (e.g., camera, LiDAR, etc.), the sensors’ relative firing synchronization pattern, and the sensors’ calibrated extrinsics (e.g., position and orientation) and intrinsics (e.g., focal length). The vehicle model also defines the onboard computer hardware, sensor drivers, controllers, and the autonomy software release under test.

[0033] The autonomous system model includes an autonomous system dynamic model. The autonomous system dynamic model is used for dynamics simulation that takes the actuation actions of the virtual driver (e.g., steering angle, desired acceleration) and enacts the actuation actions on the autonomous system in thesimulated environment to update the simulated environment and the state of the autonomous system. To update the state, a kinematic motion model may be used, or a dynamics motion model that accounts for the forces applied to the vehicle may be used to determine the state. Within the simulator, with access to real log scenarios with ground truth actuations and vehicle states at each timestep, embodiments may also optimize analytical vehicle model parameters or learn parameters of a neural network that infers the new state of the autonomous system given the virtual driver outputs.

[0034] Asset models (117) include multiple models, each model modeling a particular type of individual asset in the real world. The assets may include inanimate objects, such as construction barriers or traffic signs, parked cars, and background (e.g., vegetation or sky), and animate objects. Each of the entities in a scenario may correspond to an individual asset. As such, an asset model, or instance of a type of asset model, may exist for each of the objects or assets in the scenario. The assets can be composed together to form the three-dimensional simulated environment. An asset model provides all the information needed by the simulator to simulate the asset. The asset model provides the information used by the simulator to represent and simulate the asset in the simulated environment. For simplifying representation and simulation, stationary objects may be grouped into one or more backgrounds. Examples of asset models include neural reflectance fields, mesh, Gaussian splat, triplane representation, textured meshes, voxel grids, and other representations.

[0035] Closely related to the asset models (117) are agent models (118). An agent model represents an agent in a scenario. An agent is a type of asset which is a sentient being that has an independent decision-making process. Namely, in the real world, the agent may be an animate being (e.g., a person or animal) that decides based on an environment. The agent makes active movement rather than in addition to passive movement. An agent model, or an instance of an agent model may exist for each agent in a scenario. The agent model is a model of the agent. If the agent is in a mode of transportation, then the agent model includesthe model of transportation in which the agent is located. For example, agent models may represent pedestrians, children, vehicles being driven by drivers, pets, bicycles, and other types of agents.

[0036] An agent model may encompass two parts. A first part of the agent model is an asset model modeling the appearance of the agent while a second part is a model of the actions taken by the agent. The actions of the agent may be an action model. Thus, the appearance part and the action part together form the agent model.

[0037] The action part of the agent model (118) leverages the scenario specification and assets to control all agents in the scene and their actions at each timestep. The agent’s behavior is modeled in a region of interest centered around the autonomous system. Depending on the scenario specification, the agent simulation will control the agents in the simulation to achieve the desired behavior. Agents can be controlled in various ways. One option is to leverage heuristic agent models, such as an intelligent-driver model (IDM) that tries to maintain a certain relative distance or time-to-collision (TTC) from a lead agent or heuristic-derived lane-change agent model. Another is to directly replay agent trajectories from a real log or to control the agent(s) with a data-driven traffic model. Through the configurable design, embodiments may mix and match different subsets of agents to be controlled by different behavior models. For example, far-away agents that initially may not interact with the autonomous system and can follow a real log trajectory, but when near the vicinity of the autonomous system may switch to a data-driven agent model. In another example, agents may be controlled by a heuristic or data-driven agent model that still conforms to the high-level route in a real log. This mixed-reality simulation provides control and realism.

[0038] Further, agent models may be configured to be in cooperative or adversarial mode. In cooperative mode, the agent model models agents to act rationally in response to the state of the simulated environment. In adversarialmode, the agent model may model agents to act irrationally, such as exhibiting road rage and bad driving.

[0039] The training data generator (122) is configured to generate training data.For example, the training data generator (122) may modify real -world scenarios to create new scenarios. The modification of real-world scenarios is referred to as mixed reality. For example, mixed-reality simulation may involve adding in new agents with novel behaviors, changing the behavior of one or more of the agents from the real world, and modifying the sensor data in that region while keeping the remainder of the sensor data the same as the original log. In some cases, the training data generator (122) converts a benign scenario into a safety-critical scenario. The training data generator may also include adding assets that are not in the real world or moving assets. The training data generator (124) may also create an entirely different simulation environment than is detected in the real world.

[0040] Continuing with FIG. 1, in one or more embodiments, the sensor simulation models (114) model, in the simulated environment, active and passive sensor inputs. Passive sensor inputs capture the visual appearance of the simulated environment including stationary and nonstationary simulated objects from the perspective of one or more cameras based on the simulated position of the camera(s) within the simulated environment. Examples of passive sensor inputs include inertial measurement unit (IMU) and thermal. Active sensor inputs are inputs to the virtual driver of the autonomous system from the active sensors, such as LiDAR, RADAR, global positioning system (GPS), ultrasound, etc.Namely, the active sensor inputs include the measurements taken by the sensors, and the measurements being simulated based on the simulated environment based on the simulated position of the sensor(s) within the simulated environment. By way of an example, the active sensor measurements may be measurements that a LiDAR sensor would make of the simulated environment over time and in relation to the movement of the autonomous system.

[0041] In one or more embodiments, the rendering system (301) shown in FIG.3 may be or include all or a portion of the sensor simulation models (114) or the training data generator (122). Specifically, the rendering system (301) includes functionality to generate a scene and render an image from a scene. The rendering may include LiDAR snapshots and camera images. A LiDAR snapshot is information gathered from a LiDAR sweep that is associated with a single point in time. For example, the LiDAR snapshot may be a list of LiDAR points (i.e., distances and angles of the LiDAR point) taken during the LiDAR sweep or a LiDAR point cloud (e.g., the LiDAR points translated to three-dimensional space based on location of the LiDAR sensor and the angles).

[0042] The sensor simulation models (114) are configured to simulate the sensor observations of the surrounding scene in the simulated environment (104) at each timestep according to the sensor configuration on the vehicle platform. When the simulated environment directly represents the real-world environment, without modification, the sensor output may be directly fed into the virtual driver. For light-based sensors, the sensor model simulates light as rays that interact with objects in the scene to generate the sensor data. Depending on the asset representation (e.g., of stationary and nonstationary objects), embodiments may use graphics-based rendering for assets with textured meshes, neural rendering, or a combination of multiple rendering schemes. Leveraging multiple rendering schemes enables customizable world building with improved realism. Because assets are compositional in 3D and support a standard interface of render commands, different asset representations may be composed in a seamless manner to generate the final sensor data. Additionally, for scenarios that replay what happened in the real world and use the same autonomous system as in the real world, the original sensor observations may be replayed at each timestep.

[0043] The latency model (120) represents timing latency that occurs when the autonomous system is in a real-world environment. Several sources of timing latency may exist. For example, a latency may exist from the time that an event occurs to the sensors detecting the sensor information from the event and sendingthe sensor information to the virtual driver. Another latency may exist based on the difference between the computing hardware executing the virtual driver in the simulated environment as compared to the computing hardware of the virtual driver. Further, another timing latency may exist between the time that the virtual driver transmits an actuation signal to the autonomous system changing (e.g., direction or speed) based on the actuation signal. The latency model (120) models the various sources of timing latency.

[0044] Stated another way, in the real world, safety-critical decisions in the real world may involve fractions of a second affecting response time. The latency model simulates the exact timings and latency of different components of the onboard system. To enable scalable evaluation without strict requirements on exact hardware, the latencies and timings of the different components of the autonomous system and sensor modules are modeled while running on different computer hardware. The latency model may replay latencies recorded from previously collected real-world data or have a data-driven neural network that infers latencies at each timestep to match the hardware in a loop simulation setup.

[0045] The simulator (100) is connected to a data repository (105). The data repository (105) is any type of storage unit or device that is configured to store data. The data repository (105) includes data gathered from the real world. For example, the data gathered from the real world includes real agent trajectories (126), real sensor data (128), real trajectories of the system capturing the real trajectory (130), and real latencies (132). Each of the real agent trajectories (126), real sensor data (128), real trajectory of the system capturing the real trajectory (130), and real latencies (132) is data captured by or calculated directly from one or more sensors from the real world (e.g., in a real-world log). In other words, the data gathered from the real world are actual events that happened in real life. For example, in the case that the autonomous system is a vehicle, the real-world data may be captured by a vehicle driving in the real world with sensor equipment.

[0046] Further, the data repository (105) includes functionality to store one or more scenario specifications (140). A scenario specification (140) specifies ascenario and evaluation setting for testing or training the autonomous system. For example, the scenario specification (140) may describe the initial state of the scene, such as the current state of the autonomous system (e.g., the full 6D pose, velocity and acceleration), the map information specifying the road layout, and the scene layout specifying the initial state of all the dynamic agents and objects in the scenario. The scenario specification may also include dynamic agent information describing how the dynamic agents in the scenario should evolve over time which are inputs to the agent models. The dynamic agent information may include route information for the agents, desired behaviors or aggressiveness. The scenario specification (140) may be specified by a user, programmatically generated using a domain-specification language (DSL), procedurally generated with heuristics from a data-driven algorithm, or adversarial-based generated. The scenario specification (140) can also be conditioned on data collected from a real- world log, such as taking place on a specific real-world map or having a subset of agents defined by their original locations and trajectories.

[0047] The interfaces between the virtual driver and the simulator match the interfaces between the virtual driver and the autonomous system in the real world. For example, the sensor simulation model (114) and the virtual driver match the virtual driver interacting with the sensors in the real world. The virtual driver is the actual autonomy software that executes on the autonomous system. The simulated sensor data that is output by the sensor simulation model (114) may be in or converted to the exact message format that the virtual driver takes as input as if the virtual driver were in the real world, and the virtual driver can then run as a black box virtual driver with the simulated latencies incorporated for components that run sequentially. The virtual driver then outputs the exact same control representation that it uses to interface with the low-level controller on the real autonomous system. The autonomous system model (116) will then update the state of the autonomous system in the simulated environment. Thus, the various simulation models of the simulator (100) run in parallel asynchronously at their own frequencies to match the real-world setting.

[0048] While FIG. 1 shows a configuration of components, other configurations may be used without departing from the scope of one or more embodiments. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components.

[0049] FIG. 2 shows a flow diagram for executing the simulator in one or more embodiments. In Block 201, an initial scene for a realistic scenario for the real world is generated as a simulated environment state. The initial scene may be generated by identifying a map of a geographic region. Log data from the real world may be used to generate an initial virtual world. The log data defines which asset models are used in the initial positioning of assets. Some assets may be moved, removed, and some assets may be added. Further, using the map of the geographic region, the training data generator may generate a set of assets in the initial scene. As discussed above, the training data generator may generate a wide variety of assets, which each correspond to an individual execution of FIG. 2. The operations of the training data generator are described in FIG. 3-6.

[0050] Continuing with FIG. 2, in Block 203, the sensor simulation model is executed on the simulated environment state to obtain simulated sensor output. The sensor simulation model may use beamforming and other techniques to replicate the view to the sensors of the autonomous system. Each sensor of the autonomous system has a corresponding sensor simulation model and a corresponding system. The sensor simulation model executes based on the position of the sensor within the virtual environment and generates simulated sensor output. The simulated sensor output is in the same form as would be received from a real sensor by the virtual driver. Location and viewing direction of the sensor with respect to the autonomous vehicle may be used to replicate the originating location of the corresponding virtual sensor on the simulated autonomous system. Thus, the various sensor inputs to the virtual driver match the combination of inputs if the virtual driver were in the real world.

[0051] The simulated sensor output is passed to the virtual driver. In Block 205, the virtual driver executes based on the simulated sensor output to generate actuation actions. The actuation actions define how the virtual driver controls the autonomous system. For example, for a self-driving vehicle, the actuation actions may be the amount of acceleration, movement of the steering, triggering of a turn signal, etc. From the actuation actions, the autonomous system state in the simulated environment is updated in Block 207. The actuation actions are used as input to the autonomous system model to determine the actual actions of the autonomous system. For example, the autonomous system dynamic model may use the actuation actions in addition to road and weather conditions to represent the resulting movement of the autonomous system. For example, in a wet or snowy environment, the same amount of acceleration action as in a dry environment may cause less acceleration than in the dry environment. As another example, the autonomous system model may account for possibly faulty tires (e.g., tire slippage), mechanical based latency, or other possible imperfections in the autonomous system.

[0052] In Block 209, agents’ actions in the simulated environment are modeled, by the agent, based on the simulated environment state. Concurrently with the virtual driver model, the agent model and asset models are executed on the simulated environment state to determine an update for each of the assets and agents in the simulated environment. Through the training of the agent model described in the following figures, the agent model causes the agent to take actions that are more realistic. For some of the agents, the agents’ actions may use the previous output of the evaluator to test the virtual driver. For example, if the agent is adversarial, the evaluator may indicate based on the previous action of the virtual driver, the lowest scoring metric of the virtual driver. Using a mapping of metrics to actions of the agent model, the agent model executes to exploit or test that particular metric.

[0053] Thus, in Block 211, the simulated environment state is updated according to the agents’ actions and the autonomous system state to generate anupdated simulated environment state. The updated simulated environment includes the change in positions of the agents and the autonomous system.Because the models execute independently of the real world, the update may reflect a deviation from the real world. Thus, the autonomous system is tested with new scenarios. In Block 213, a determination is made whether to continue. If the determination is made to continue, testing of the autonomous system continues using the updated simulated environment state in Block 203. At each iteration, during training, the evaluator provides feedback to the virtual driver. Thus, the parameters of the virtual driver are updated to improve the performance of the virtual driver in a variety of scenarios. During testing, the evaluator is able to test using a variety of scenarios and patterns including edge cases that may be safety critical. Thus, one or more embodiments improve the virtual driver and increase the safety of the virtual driver in the real world.

[0054] As shown, the virtual driver of the autonomous system acts based on the scenario and the current learned parameters of the virtual driver. The simulator obtains the actions of the autonomous system and provides a reaction in the simulated environment to the virtual driver of the autonomous system. The evaluator evaluates the performance of the virtual driver and creates scenarios based on the performance. The process may continue as the autonomous system operates in the simulated environment.

[0055] FIG. 3 shows a diagram of a system for training a diffusion model in accordance with one or more embodiments. As shown in FIG. 3, a data repository (105) is connected to a model generator (302) and a sensor simulation model (114). Each of these components is described below.

[0056] The data repository (105) includes functionality to store real sensor data (128), an asset model (142), an asset latent code structure (306), and an asset latent code (e.g., asset latent code X (308), asset latent code Y (310)). The real sensor data (128) and asset model (142) are the same as described above with reference to FIG. 1. The asset latent code structure (306) is a storage structure for asset latent code. Specifically, the asset latent code structure (306) is a templatedata structure for storing a latent code representation of an asset. For example, the asset latent code structure (306) may be an array, a grid structure, or other structure. Further, positions in the asset latent code structure may correspond to a location on the asset. Thus, a position in the asset latent code structure may reference a virtual location with respect to an origin on the asset (e.g., bottom right corner of the asset). The value of the position is a latent feature representation of the appearance of the asset at that location. In some cases, multiple positions may correspond to a same location on the asset. The asset latent code (e.g., asset latent code X (308), asset latent code Y (310)) is a latent representation of the geometry and appearance of the asset. The asset latent code is a compressed representation of the asset.

[0057] The model generator (302) includes a diffusion model (312), a diffusion model trainer (314), a latent code trainer (316), a noise insertion function (318), an asset latent code generator (320), and a decoder model (322). A diffusion model (312) is a machine learning model that performs a diffusion process. A forward diffusion process adds noise to a representation through a series of diffusion steps. A reverse diffusion process is a process that removes noise from a representation through a series of diffusion steps. The diffusion model (312) is trained to perform the reverse diffusion process. During training the reverse diffusion process should be starting from a noised version of asset latent code, which is obtained by adding random noise to an original clean asset latent code and predicting the original clean asset latent code. During inference, the reverse diffusion process starts with a randomly populated asset latent code structure (306) and generates an asset latent code that matches a real -world distribution. At each diffusion timestep of the reverse diffusion process, the diffusion model (312) attempts to reverse the noise added to generate the current asset latent code. To perform the reverse diffusion process, the mean of a distribution function may be parameterized by a noise prediction model. The noise prediction model is a machine learning model that is configured to generate a noise prediction value. The noise prediction value is a value configured to predict a level of noise in the current asset latent code.

[0058] The diffusion model trainer (314) is configured to train the diffusion model (312). Training the diffusion model (312) includes causing the diffusion model to remove noise from asset latent code and calculating a diffusion loss based on the execution of the diffusion model (312). The noise insertion function (318) is configured to inject noise into asset latent code to generate input data for training the diffusion model (312). Noise is random values that are added to the asset latent code that modifies one or more latent values or replaces one or more latent values.

[0059] The latent code trainer (316) is configured to generate asset latent code representing a real-world value from noise. Because the asset latent code has latent features, the corresponding asset does not naturally have a latent feature representation. The latent code trainer (316) is configured to generate the latent feature representation over the course of several iterations and applying a loss function. The asset latent code generated by the latent code trainer (316) is training data for training the diffusion model (312).

[0060] The decoder model (322) is a machine learning model that is configured to generate an asset model (117) from the asset latent code for the asset. For example, the decoder model (322) may be a convolutional neural network model.

[0061] The asset latent code generator (320) is configured to generate asset latent code by populating an asset latent code structure (306). In one or more embodiments, the asset latent code generator (320) may be configured to initially populate the asset latent code structure (306).

[0062] While FIG. 3 shows a configuration of components, other configurations may be used without departing from the scope of one or more embodiments. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components.

[0063] The operations of the various components of FIG. 3 are described in FIG. 4, FIG. 5, and FIG. 6. FIG. 4, FIG. 5, and FIG. 6 show flowcharts in accordance with one or more embodiments. The processing of FIG. 4, FIG. 5,and FIG. 6 may be performed using the system of FIG. 1 and FIG. 3 and with the flowchart of FIG. 2. Further the processing of FIG. 4, FIG. 5, and FIG. 6 may be performed on or received at one or more computer processors. While the various steps in these flowcharts are presented and described sequentially, at least some of the steps may be executed in different orders, may be combined or omitted, and at least some of the steps may be executed in parallel. Furthermore, the steps may be performed actively or passively.

[0064] FIG. 4 is a flowchart for generating a trained asset latent code in accordance with one or more embodiments. Block 402 includes obtaining real sensor data for a real asset. Real sensor data may be gathered for an asset from a real sensor moving through a real -world environment. For example, a real -world surveying or sensing vehicle may move through a real-world environment to obtain real sensor data. As the real sensor data is gathered, the extrinsics and intrinsics of the sensor are related to the real sensor data. One or more camera images and LiDAR snapshots may be obtained for an asset. The real sensor data is processed by an object detection model to identify assets. For example, the object detection model may include an image based neural network or point cloud based neural network that may process camera images and LiDAR to identify and classify individual assets. As part of identifying and classifying assets, the object detection model outputs a bounding box around the asset. In one or more embodiments, the object bounding box is the smallest rectangular cuboid.

[0065] Block 404 includes generating an asset latent code for the real asset.Generating the asset latent code for the real asset may include randomly initializing an asset latent code structure to generate the asset latent code.

[0066] Block 406 includes processing, by a decoder model, the asset latent code to generate an asset model. The decoder model generates an asset model from the asset latent code. The asset latent code may be concatenated with a vector embedding of the class of asset (e.g., car, person, truck, bicyclist, etc.). The processing by the decoder model may be passing the asset latent code through asequence of blocks of a neural network. The resulting output of Block 406 is an asset model that is an internal 3D representation of the asset.

[0067] Block 408 includes processing, by a sensor simulation model, the asset model to generate predicted sensor data. The sensor simulation model processes the asset model in accordance with the extrinsics and intrinsics of the corresponding sensor to generate predicted sensor data for the asset. Specifically, the actor pose (e.g., from human label, object detector, or other component) is used to place the asset model to the virtual world. The extrinsics of the sensor are used to place the virtual sensor in the virtual world. The intrinsics of the real sensor is used to determine how to capture the sensor data, such as the resolution and other features of the sensor data. Ray casting, splatting, or other operations may be performed by the sensor simulation model to generate one or more camera images, one or more LiDAR snapshots, or other sensor data from the asset model.

[0068] Block 410 includes comparing the predicted sensor data with the real sensor data to obtain loss components. Because the asset model is generated from asset latent code, during training of the asset latent code, the asset model may not be accurate, resulting in the output sensor data of Block 408 not being accurate. The comparison may be across the different types of predicted sensor data to generate multiple loss components that are combined. The loss components may include a color loss from a comparison of a real camera image in the real sensor data with a virtual camera image in the predicted sensor data. The color loss may be calculated as the difference between the Red Green Blue (RGB) values for each pixel in the respective image. The loss components may include a LiDAR loss from a real LiDAR snapshot in the real sensor data with a virtual LiDAR snapshot in the predicted sensor data. The LiDAR loss may be the differences in the detected distances to the object and angle of the object with respect to the virtual sensor. The loss may include a regularization value determined from the predicted sensor data. The regularization loss may be gradients between adjacent values in the predicted sensor data. The loss components may further include a perceptual loss generated by a machine learning model processing the predictedsensor data. The perceptual loss is a loss based on high level features and indicates how much the predicted sensor data looks realistic as determined by the machine learning model.

[0069] Block 412 includes generating a training loss from the loss components.The loss components may be combined using respective weights to generate a training loss.

[0070] Block 414 includes determining whether to continue training. If the training loss is less than a threshold or the number of training iterations satisfies an iteration threshold, then the latent code trainer may determine that the asset latent code represents a real asset. If the training loss does not satisfy a threshold, the flow may proceed to Block 416.

[0071] Block 416 includes updating the asset latent code for the real asset and, optionally, the decoder model based on the training loss. In one or more embodiments, the total loss is backpropagated through the asset latent code.Specifically, the total loss is used to adjust the various values in the asset latent code. In some cases, the decoder model is co-trained with the asset latent code. Specifically, the decoder model may also be adjusted based on the decoder loss. After an initial training of the decoder model and the asset latent code, to add new assets or new types of assets, updates to the decoder model may be frozen so as to not change the decoding of previous asset latent code in the system.

[0072] The processing of Blocks 402 to 416 includes training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code.

[0073] Training the asset latent code is performed over several iterations using snapshots in the real sensor data. The snapshots include camera images and LiDAR snapshots.

[0074] If the determination is made to not continue training of the asset latent code, the flow may proceed to Block 418. Block 418 includes training a diffusion model with the trained asset latent code. The training in FIG. 4 results in thegeneration of training data to generate asset latent code and a trained decoder model that can decode the asset latent code.

[0075] FIG. 5 is a flowchart for training the diffusion model with the trained asset latent code in accordance with one or more embodiments.

[0076] Block 502 includes obtaining asset latent code for an asset. The asset latent code may be obtained from a data repository.

[0077] Block 504 includes adding noise to the trained asset latent code to generate a noisy asset latent code. Randomized noise values are added to the various values in the asset latent code. The result of adding randomized noise is noisy asset latent code.

[0078] Block 506 includes processing, by a diffusion model, the noisy asset latent code to generate predicted latent code. As discussed above, through several diffusion steps, the diffusion model identifies and removes noise from the asset latent code to generate predicted latent code. The goal of the predicted latent code is to match the latent code in Block 502.

[0079] Block 508 includes comparing the predicted latent code with the trained asset latent code to obtain a diffusion loss. The difference between respective values of the predicted latent code and the trained asset latent code is a set of difference values reflecting the differences in the latent codes. The set of difference values are combined into a diffusion loss.

[0080] Block 510 includes determining whether to continue training. For example, the determination may be whether the diffusion loss is less than a threshold. If the diffusion loss is less than a threshold or the number of training iterations satisfies a threshold, then the diffusion model may be determined to be trained. If the diffusion loss is not less than a threshold, the flow may proceed to Block 512.

[0081] Block 512 includes updating, using the diffusion loss, the diffusion model. The diffusion loss may be backpropagated through the diffusion model togenerate a revised diffusion model. The flow may then proceed to Block 502 to continue training with asset latent codes from real assets.

[0082] The processing of Blocks 506 through 512 is directed to training the diffusion model based on a comparison of the noisy asset latent code with the trained asset latent code to generate a trained diffusion model. Through training in FIG. 5 with hundreds or thousands of asset latent codes generated for real assets using FIG. 4, the diffusion model learns to distinguish between noise and latent features of real assets. Thus, the diffusion model is trained to remove the noise.

[0083] When training is determined complete in Block 510, Block 514 involves deploying the trained diffusion model. The trained diffusion model is deployed and used in the simulator to generate new assets.

[0084] FIG. 6 is a flowchart for deploying a diffusion model in accordance with one or more embodiments. Block 602 includes generating a randomized asset latent code. An asset latent code structure is randomly initialized with values to generate an asset latent code that does not represent any real asset. Further, because of the random initialization, the asset latent code is likely not realistic because any asset generated by the decoder model would not have the geometry or appearance of an asset that is present in the real world.

[0085] Block 604 includes processing, by the trained diffusion model, the randomized asset latent code to generate a realistic asset latent code. Through the training in FIG. 5, the diffusion model is configured to remove noise from asset latent code. With the randomization, the “noise” reflects latent features that are not present in assets in the real world. Over the course of several diffusion steps of the reverse diffusion process, the diffusion model revises the asset latent code to remove the noise. The result is a revised and realistic asset latent code.Because the realistic asset latent code is generated from randomized asset latent code, the realistic asset latent code does not actually represent a seen real-world asset.

[0086] Block 606 includes processing, by the decoder model, the realistic asset latent code to generate a new asset model for a new asset. The decoder model executes as described in FIG. 4 to generate a new asset model.

[0087] Block 608 includes processing, by a sensor simulation model, the asset model structure to generate simulated sensor data. The sensor simulation model executes as described in FIG. 4 to generate sensor data. Specifically, the simulator may add the new asset model to the virtual world as described in FIG. 2. Thus, a simulated environment state of the new asset model structure in the virtual world is generated. The sensor simulation model is executed on the simulated environment state to obtain simulated sensor output showing the new asset.

[0088] Block 610 includes outputting the simulated sensor data. For example, the simulated sensor data may be output to a virtual driver. As another example, the simulated sensor data may be used in a gaming environment. Outputting the simulated sensor data may include displaying the simulated sensor data, transmitting the simulated sensor data on a network, or storing the simulated sensor data.

[0089] FIG. 7 shows an example implementation in accordance with one or more embodiments. The following example is for explanatory purposes only and not intended to limit the scope of one or more embodiments. Namely, while the example implementation is directed to training and testing of self-driving vehicles (SDV), embodiments are not so limited.

[0090] The example implementation has a goal to create a generative model (e.g., diffusion and decoder models) that can create, in a scalable manner, high- fidelity assets for self-driving simulation. The generative model should enable unconditional generation of diverse assets for a variety of classes (e.g., cars, buses, trucks, pedestrians), conditional generation given asset classes, and also support generating complete assets conditioned on new in-the-wild camera images and LiDAR point clouds for actors that may be occluded or far-away. To avoid the quality degradation that occurs when generative models trained on synthetic data are transferred to real data, the generative model should directly learn over acorpus of real-world driving scenes collected by a robot sensor platform. Towards this goal, a two-stage “reconstruct-then-generate” framework is presented in FIG.7. In the first stage (i.e., the asset latent code training stage (702)), one or more embodiments may jointly learn a set of asset latent codes through reconstructionbased neural rendering, where each asset latent code represents a foreground actor in the dataset. A compositional scene representation that enables rendering of actors of driving scenes in an occlusion-aware manner is generated, enabling us to decouple the foreground actors from the background (e.g., road, sky, vegetation).

[0091] In the second stage (i.e., the diffusion model training stage (704)), one or more embodiments may train a diffusion model to learn generative priors in this asset latent space, enabling the generation of realistic and diverse neural assets. This enables both unconditional generation and conditional generation through score-based guidance.

[0092] At inference stage (706), the sampled noise is passed through the trained diffusion model to generate an asset latent code. The asset latent code is passed through a decoder to generate a new asset (e.g., the truck in the example shown FIG. 7).

[0093] The following presents compositional scene representation, and then presents how actors may be compressed to be represented in latent space, and then show the diffusion process.

[0094] With respect to compositional scene representation, a generative 3D asset model is generated from real-world driving scenes. Unlike existing objectcentric approaches, one or more embodiments may use sensor data that contains many actors captured under different viewpoint, distances, and occlusions. To handle multiple instances and their occlusions, one or more embodiments may leverage a compositional scene representation. Each scene is decomposed into a static background B and a set of dynamic actors {c / Zwith each actor enclosed within a 3D box and localized by a trajectory of poses. One or more embodiments may model the static background and dynamic actors with separate neural feature fields parameterized by triplane representations. Specifically, for each dynamicactor’s volume, explicit features are aligned along three orthogonal planes, each with resolutionx x C, where is the spatial resolution, and C is the feature dimension. Similarly, triplane features of resolution NBX NBX C represent the background volume. For a spatial query coordinate x 6 IR3, one or more embodiments may project it onto each of the three feature planes t = (txy, txz, tyz) and bilinearly interpolate the corresponding feature vectors. The interpolated features are concatenated with view direction d 6 S2and processed by an MLP networkfeatto yield geometry as a signed-distance function (SDF) s and a neural feature f. This querying process is defined by eq. (1):f / ?eat({interp(xP,xz:yZ^, d), (1) where xprepresents the 2D projection of x onto feature plane p. To render the scene representation, one or more embodiments may first transform the objectcentric actor neural feature fields to world coordinates. Then, one or more embodiments may composite the background and actor feature fields. The composited geometry and appearance features are then rendered into sensor observations through neural volume rendering.

[0095] With respect to learning latent asset representations, the following operations may be performed. Encoding to a latent space may be performed by using per-scene reconstruction methods to train each scene separately, with its own set of actor and background triplanes. This prevents effective handling of ambiguities in occluded or unseen regions, resulting in actor representations that generalize poorly to unseen viewpoints. Naively training jointly over all the explicit triplanes for every actor in the dataset containing tens of thousands of objects requires vast amounts of GPU memory and still does not force the learned triplanes to have complete actor shape and appearance. To address these limitations, one or more embodiments may propose learning a latent code c£G ]Kn^xn^xcfor each actor representation and a class embedding E [R ^x^xc5combined with a shared asset triplane decoder fdecthat maps the latent code into the triplane representation shown in equation (2): / dec: e£, ctEtiG ^XN^X3C_ (2)

[0096] The class embedding ethas the same spatial resolution as the latent code and is shared among actors of the same class. One or more embodiments may concatenate the class embedding with the actor latent code before passing it to the decoder. The decoderdecupsamples the latent code by a factor f = NM / nM. The intuition is that different actors are observed from various viewpoints, so their feature planes t(capture different informative regions. By compressing the actors into a latent code bottleneck, one or more embodiments may enable learning shape and appearance priors, allowing inference of invisible parts from sparse observations.

[0097] For rendering sensor observations, with having decoded the asset neural fields {t ^=1from the latent codes {c ^=1, the next step is to composite the asset neural fields with the background neural fields tBand render into the sensor data. In this work, the example implementation focus on camera images and LiDAR point clouds, as they are primary sensory modalities for SDVs.

[0098] For camera rendering, one or more embodiments may use a hybrid volume and neural rendering framework for both efficiency and realism. Given a camera ray r(t) = o + hd sent from the camera center o to the pixel in direction d, one or more embodiments may sample 3D points along the ray, querying geometry stand neural feature ftvia equation (1). One or more embodiments may then aggregate these samples to obtain the pixel feature through volume rendering as shown in equation (3):Nt t-1f(r) = “aj)’ &t=i j=i where atis the opacity for the / -th point, derived from the SDF st. The following one or more embodiments may repeat this process to volume render all camera pixels, generating a 2D feature map FG. Then one or more embodiments may leverage a 2D CNN networkrgbto convert this feature map into an image Irgbas shown in equation (4): / rgb: F G ^HfyWfyCf -> IrgbG (4)

[0099] The CNN networkrgbmay up-sample the rendered map from resolution Hf X Wf to H X W. This allows us to significantly reduce the number of neural feature field queries. This approach also enhances model capacity and improves image quality by capturing spatial relationships effectively. In addition to camera images, one or more embodiments may use LiDAR rendering for additional geometry supervision. LiDAR sensors emit laser pulses and measure the time of flight to determine distances to reflective surfaces. This depth information provides valuable supervision for asset geometry. Using similar notation, one or more embodiments may define r(t) = o + hd as a ray cast from the LiDAR sensor center o in the direction d. One or more embodiments may render LiDAR depth similarly to equation (3) by aggregating sample depths via volume rendering: £>(r) = ’t iwtht, where htis the depth of the Lth sampled point, and wtis the sample weight computed as in equation (3).

[0100] With respect to learning, one or more embodiments may jointly optimize the asset code {c J^, class embedding {ej^, background neural fields ts, asset decoderdec, neural feature fields MLPfeat, and RGB CNN network / i-gb by minimizing the differences between rendered and observed sensor data. To achieve high-fidelity reconstruction and rendering, one or more embodiments may incorporate perceptual and patch-based adversarial objectives. One or more embodiments may also regularize the latent space by applying a Kullback-Leibler (KL) penalty. The full objective may be shown in equation (5):-^rgb d” perp -Tperp " T Xadv-^adv d” Mid -^lid d” RL TKL<where Lrgband Lperprepresent the f2photometric loss and the perceptual loss between rendered and observed images, Ladvis the adversarial objective using patch GAN, £lidis the f£depth loss between rendered and observed LiDAR points, and UKLis the injected KL penalty guiding the latent space towards a standard normal distribution, similar to £KL= || + ff — 1 — log (<r ) ||i,where [it and represent the mean and standard deviation components of latent code c£, i.e., c£= + ffiQe with e~J\f(O, I). This regularization preventsoverfitting and encourages the latent space to be compact, continuous, smooth, and low variance, enabling easier learning of generative models. Similarly, one or more embodiments may introduce additional perceptual losses on object patches for enhanced object-level supervision, where the patches are obtained by projecting the 3D instance boxes onto the image plane.

[0101] With respect to the diffusion model, given the learned asset code library {ci}Ni=1from the dataset, one or more embodiments may aim to learn generative priors using a diffusion model. Diffusion models are probabilistic frameworks that model the data distribution by progressively denoising a Gaussian noise variable. The diffusion model operates directly in the latent space and begins with Gaussian noise, progressively denoising it to recover the underlying latent distribution. The forward or diffusion process is a discrete-time Markov chain that iteratively adds Gaussian noise to the latent code c^° producing progressively noisier codes following the transition probability= N(y / 1 — fid) fiddif according to a noise schedule fid) After T steps, the code approximates pure Gaussian noise. This forward process can be directly sampled at timestep t using closed-form equation (6):c® = - / + - / l — a^e, (6) where cd = ]”[s=i with = 1 — fidf and e~7\f(0, I) is drawn from a Gaussian distribution. The goal of the diffusion model is to learn the reverse process, progressively denoising the noisy code to recover a clean version. One or more embodiments may parameterize the denoising networkdiffas a U-Net, which takes the noisy code c® and timestep t as inputs, predicting the noise estimate / diff(c^^ and removing it to yield a denoised code. The denoising network is trained with a weighted ℓ2objective as shown in the following diffusion loss equation (7):-^diff = Ec,6,t [| w(t)|| / diff(cW, t) - e||f|, (7)|_Z. where e~7\f(0, 1) is standard Gaussian noise,T) is uniformly sampled from the interval [1, T], and is derived from c® (Eq. (6)), whereis sampled from the asset code library. The term provides time-dependent weighting. Learning the diffusion process in the latent space, rather than in highdimensional triplane space like prior work, may have features of (i) focusing on essential contents of the data and (ii) operating in a computationally efficient, compact space.

[0102] Unconditional asset generation involves generating a latent code using the learned diffusion modeldiffand then decoding it into neural assets using the learned asset decoderdec. Sampling from the diffusion prior can be performed with various solvers, similar to methods used in image generation. For example, starting with a random Gaussian code1) and iteratively denoising it over T steps until reaching t = 0 as shown in equations (8) and (9):- / c(t) -, / 7?(Ue\ [ c^-U = -+pa~i)e, (8)\ J S. t. € = / diff(c(t), t), (9) where e is the noise estimate at timestep t from the denoising networkdiff. The final denoised code c® is a sample from the learned latent distribution. One or more embodiments may then pass this code to the asset decoder to generate the asset neural fields, t = / dec(c^0'))7for scene composition and rendering.

[0103] For conditional asset generation, one or more embodiments may aim to model the conditional distribution of the latent space p(c|y). This can be achieved with a conditional denoising networkdiff(c^ t, y), with condition signals such as actor class and time-of-day. Classifier-free diffusion guidance enables effective control over asset generation. Alternatively, for some inverse problems such as image-to-3D, conditional generation can be approximated from an unconditional model using classifier guidance, leveraging gradients of the rendering loss with respect to known observations. This flexibility removes the need for separate models tailored to different inverse problems. Given camera image and / or LiDAR point cloud of a new actor to reconstruct, during the sampling process (Eq. (8)) one or more embodiments may compute the gradient of the rendering loss with respectto the current code: g = Vc(t) (£rgb+xiid Aid), and then steer the sampling process by updating the noisy code as shown in equation (10):i) <_c(t i) —x- g, (10)where is computed following equation (8) and c^-1) is the updated code; x is a small guidance weight.

[0104] FIG. 8 shows an example in accordance with one or more embodiments.The left side (802) shows ground truth, and the right side (804) shows the results from the modeling system. Embodiments may use only a sparse number of views such as only the views available when a sensing vehicle is driving down a road. While other approaches may struggle to learn robust geometry, leading to severe artifacts in new viewpoints, one or more embodiments are able to learn a more robust geometry by extracting the assets from a limited number of views (806), learning the asset latent representations, and being able to construct 360° degree views (810).

[0105] However, for new sensor poses (front-left camera position in boxed regions), previously unobserved regions become visible, requiring the model to “hallucinate” these areas. For instance, the front-right part of the SUV (812) is invisible in the front camera but becomes visible in front-left camera views. By learning generative priors, one or more embodiments are capable of rendering complete and high-quality simulations in this setting (808), comparing left image to right image. There are multiple cameras on the vehicle. The left is camera image captured from front-left camera, and the right is camera image captured from front camera.

[0106] The rectangle boxes are regions that are not observable from front camera, but we can generate those parts.

[0107] Sensor simulation involves not only simulating novel viewpoints but also synthesizing entirely new scenarios. Consider a four- way intersection with a car crossing in front of the autonomous system from a perpendicular lane. To explore a situation where that car suddenly turns sharply into your lane, actors arerendered in significantly different poses than those seen in the original scene. To that end, a 360° view synthesizes the setting by rotating actors in the scene from 0° to 360° (810), simulating various orientations.

[0108] One or more embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure.

[0109] For example, as shown in FIG. 9 A, the computing system (900) may include one or more computer processor(s) (902), non-persistent storage device(s) (904), persistent storage device(s) (906), a communication interface (908) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (902) may be an integrated circuit for processing instructions. The computer processor(s) (902) may be one or more cores, or micro-cores, of a processor. The computer processor(s) (902) includes one or more processors. The computer processor(s) (902) may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0110] The input device(s) (910) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device(s) (910) may receive inputs from a user that are responsive to data and messages presented by the output device(s) (912). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (900) in accordance with one or more embodiments. The communication interface (908) may include an integrated circuit for connecting the computing system (900) to a network (not shown) (e.g., a localarea network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) or to another device, such as another computing device, and combinations thereof.

[0111] Further, the output device(s) (912) may include a display device, a printer, external storage, or any other output device. One or more of the output device(s) (912) may be the same or different from the input device(s) (910). The input device(s) (910) and output device(s) (912) may be locally or remotely connected to the computer processor(s) (902). Many different types of computing systems exist, and the aforementioned input device(s) (910) and output device(s) (912) may take other forms. The output device(s) (912) may display data and messages that are transmitted and received by the computing system (900). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0112] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a solid state drive (SSD), compact disk (CD), digital video disk (DVD), storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by the computer processor(s) (902), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0113] The computing system (900) in FIG. 9A may be connected to, or be a part of, a network. For example, as shown in FIG. 9B, the network (920) may include multiple nodes (e.g., node X (922) and node Y (924), as well as extant intervening nodes between node X (922) and node Y (924)). Each node may correspond to a computing system, such as the computing system shown in FIG.9A, or a group of nodes combined may correspond to the computing system shown in FIG. 9A. By way of an example, embodiments may be implemented ona node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (900) may be located at a remote location and connected to the other elements over a network.

[0114] The nodes (e.g., node X (922) and node Y (924)) in the network (920) may be configured to provide services for a client device (926). The services may include receiving requests and transmitting responses to the client device (926). For example, the nodes may be part of a cloud computing system. The client device (926) may be a computing system, such as the computing system shown in FIG. 9A. Further, the client device (926) may include or perform all or a portion of one or more embodiments.

[0115] The computing system of FIG. 9A may include functionality to present data (including raw data, processed data, and combinations thereof) such as results of comparisons and other processing. For example, presenting data may be accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown, as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0116] As used herein, the term “connected to” contemplates multiple meanings. A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be a temporary, permanent, or a semi-permanent communication channel between two entities.

[0117] The various descriptions of the figures may be combined and may include, or be included within, the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, or altered as shown in the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0118] In the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (z.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before,” “after,” “single,” and other such terminology. Rather, ordinal numbers distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0119] Further, unless expressly stated otherwise, the conjunction “or” is an inclusive “or” and, as such, automatically includes the conjunction “and,” unless expressly stated otherwise. Further, items joined by the conjunction “or” may include any combination of the items with any number of each item, unless expressly stated otherwise.

[0120] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.

Claims

CLAIMSWhat is claimed is:

1. A method comprising:obtaining a real sensor data for a real asset;generating an asset latent code for the real asset;processing, by a decoder model, the asset latent code to generate an asset model;processing, by a sensor simulation model, the asset model to generate predicted sensor data;training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code;adding noise to the trained asset latent code to generate a noisy asset latent code;processing, by a diffusion model, the noisy asset latent code to generate predicted asset latent code;training the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model; anddeploying the trained diffusion model.

2. The method of claim 1, further comprising:generating a randomized asset latent code;processing, by the trained diffusion model, the randomized asset latent code to generate a realistic asset latent code;processing, by the decoder model, the realistic asset latent code to generate a new asset model for a new asset; andadding the new asset model to a virtual world.

3. The method of claim 2, further comprising:generating a simulated environment state of the new asset model in the virtual world; andexecuting a sensor simulation model on the simulated environment state to obtain simulated sensor output showing the new asset.

4. The method of claim 3, further comprising:training a virtual driver of an autonomous system using the simulated sensor output.

5. The method of claim 1, wherein generating the asset latent code for the real asset comprises randomly initializing an asset latent code structure to generate the asset latent code.

6. The method of claim 1, wherein training the asset latent code is performed over a plurality of iterations using a plurality of snapshots in the real sensor data.

7. The method of claim 6, wherein the plurality of snapshots comprises a plurality of camera images and a plurality of LiDAR snapshots.

8. The method of claim 1, further comprising:comparing the predicted sensor data with the real sensor data to obtain a plurality of loss components;generating a training loss from the plurality of loss components; and updating the asset latent code based on the training loss.

9. The method of claim 8, wherein the plurality of loss components comprises a color loss from a real camera image in the real sensor data with a virtual camera image in the predicted sensor data, a LiDAR loss from a real LiDAR snapshot in the real sensor data with a virtual LiDAR snapshot in the predicted sensor data, and a regularization value determined from the predicted sensor data.

10. The method of claim 9, wherein the plurality of loss components further comprises a perceptual loss generated by a machine learning model processing the predicted sensor data.

11. The method of claim 1, further comprising:co-training the decoder model with the asset latent code.

12. The method of claim 1, wherein the asset model is a neural reflectance field structure.

13. A system comprising:memory; anda computer processor comprising computer readable program code for performing operations comprising:obtaining a real sensor data for a real asset,generating an asset latent code for the real asset,processing, by a decoder model, the asset latent code to generate an asset model,processing, by a sensor simulation model, the asset model to generate predicted sensor data,training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code,adding noise to the trained asset latent code to generate a noisy asset latent code,processing, by a diffusion model, the noisy asset latent code to generate predicted asset latent code,training the diffusion model based on a comparison of the predicted asset latent code with the trained asset latent code to generate a trained diffusion model, anddeploying the trained diffusion model.

14. The system of claim 13, wherein the operations further comprise:generating a randomized asset latent code;processing, by the trained diffusion model, the randomized asset latent code to generate a realistic asset latent code;processing, by the decoder model, the realistic asset latent code to generate a new asset model for a new asset; andadding the new asset model to a virtual world.

15. The system of claim 13, wherein generating the asset latent code for the real asset comprises randomly initializing an asset latent code structure to generate the asset latent code.

16. The system of claim 13, wherein training the asset latent code is performed over a plurality of iterations using a plurality of snapshots in the real sensor data.

17. The system of claim 13, wherein the operations further comprise:comparing the predicted sensor data with the real sensor data to obtain a plurality of loss components;generating a training loss from the plurality of loss components; and updating the asset latent code based on the training loss.

18. The system of claim 13, wherein the operations further comprise:co-training the decoder model with the asset latent code.

19. The system of claim 13, wherein the asset model is a neural reflectance field structure.

20. A non-transitory computer readable medium comprising computer readable program code for performing operations comprising:obtaining a real sensor data for a real asset;generating an asset latent code for the real asset;processing, by a decoder model, the asset latent code to generate an asset model;processing, by a sensor simulation model, the asset model to generate predicted sensor data;training the asset latent code for the real asset based on a comparison of the predicted sensor data with the real sensor data to generate a trained asset latent code;adding noise to the trained asset latent code to generate a noisy asset latent code;processing, by a diffusion model, the noisy asset latent code;training the diffusion model based on a comparison of the noisy asset latent code with the trained asset latent code to generate a trained diffusion model; anddeploying the trained diffusion model.

Images