Sparse local fields for simulation

Abstract

Sparse local fields for simulation include obtaining a scene that includes object voxel grids for objects. The object voxel grids include an object voxel grid with the object voxel grid including a voxel positioning property, a geometry field property, and a color field property of the voxel in the object voxel grid. The operations further include identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene. The operations also include performing, using the object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot. The operations also include outputting the first virtual sensor snapshot and the second virtual sensor snapshot.

Description

SPARSE LOCAL FIELDS FOR SIMULATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a non-provisional application of, and thereby claims benefit to U. S. Patent Application Serial Number 63 / 720,430 filed on November 14, 2024, which is incorporated herein by reference in its entirety.BACKGROUND

[0002] A virtual world is a computer-simulated environment, which enables a player to interact in a three-dimensional space as if the player were in the real world. In some cases, the virtual world is designed to replicate at least some aspects of the real world. For example, the virtual world may include objects and background reconstructed from the real world. The reconstructing of objects and background from the real world allows the system to replicate aspects of the real world.

[0003] For example, one way to bring realism is by obtaining sensor data from the real world describing a scenario, modifying the scenario to create a modified scenario, and then allowing the player to interact with the modified scenario. When the player interacts with the modified scenario, different objects may be in different relative positions than in the real world. Thus, in order to modify the real world, an accurate set of models should be created and used in the virtual world.

[0004] A tradeoff exists between accuracy and performance. Representing an object using polygonal surfaces causes significant tradeoff between the accuracy and the amount of storage. Decreasing the size of the polygons to increase the resolution may cause a more accurate object model (z.e., the object voxel grid) with the cost of an increased memory requirement to store the object model and generate the virtual object from the object model. Machine learning may be used to generate implicit representations of objects. The implicit representation does not store the properties of the object itself, but rather information that when processed by the neural network model generates theproperties. Training machine learning models and generating the implicit representation to accurately predict the object in the virtual world can cause computer system performance degradation in terms of processing time and resource usage. Three-dimensional (3D) Gaussian splatting may be used. This degradation is especially noticeable given the number of objects in a virtual world. 3D Gaussian splatting represents scenes using collections of tiny, translucent ellipsoids and may be used to generate images for pinhole cameras. However, 3D Gaussian splatting does not represent objects in a manner that can generate virtual LiDAR point clouds. Thus, a separate object representation would be needed, further increasing the storage requirements. The resource requirements are further exacerbated when a large number of objects exist in the in the virtual world due to each object having its own object model. In such a scenario, the computer system may have slower responsiveness and cost prohibitive training.SUMMARY

[0005] In general, in one aspect, embodiments are directed to a computer- implemented method that includes operations. The operations include obtaining a scene that includes object voxel grids for objects. The object voxel grids include an object voxel grid with the object voxel grid including a voxel positioning property, a geometry field property, and a color field property of the voxel in the object voxel grid. The operations further include identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene. The operations also include performing, using the object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot. The rasterization is performed according to the voxel positioning property and the color field property of the voxel. The ray casting is performed according to the voxel positioning property and the geometry field property. The operations also include outputting the first virtual sensor snapshot and the second virtual sensor snapshot.

[0006] In general, in one aspect, embodiments are directed to a system that includes a memory and a computer processor including computer readable program code for performing operations. The operations include obtaining a scene that includes object voxel grids for objects. The object voxel grids include an object voxel grid with the object voxel grid including a voxel positioning property, a geometry field property, and a color field property of the voxel in the object voxel grid. The operations further include identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene. The operations also include performing, using the object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot. The rasterization is performed according to the voxel positioning property and the color field property of the voxel. The ray casting is performed according to the voxel positioning property and the geometry field property. The operations also include outputting the first virtual sensor snapshot and the second virtual sensor snapshot.

[0007] In general, in one aspect, embodiments are directed to a non-transitory computer readable medium comprising computer readable program code for performing operations. The operations include obtaining a scene that includes object voxel grids for objects. The object voxel grids include an object voxel grid with the object voxel grid including a voxel positioning property, a geometry field property, and a color field property of the voxel in the object voxel grid. The operations further include identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene. The operations also include performing, using the object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot. The rasterization is performed according to the voxel positioning property and the color field property of the voxel. The ray casting is performed according to the voxelpositioning property and the geometry field property. The operations also include outputting the first virtual sensor snapshot and the second virtual sensor snapshot.

[0008] Other aspects of one or more embodiments will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0009] FIG. 1 shows a diagram of an autonomous training and testing system in accordance with one or more embodiments.

[0010] FIG. 2 shows a flowchart of the autonomous training and testing system in accordance with one or more embodiments.

[0011] FIG. 3 shows a diagram of a rendering system in accordance with one or more embodiments.

[0012] FIG. 4 shows a diagram of a voxel grid in accordance with one or more embodiments.

[0013] FIG. 5 shows a flowchart for voxel grid training in accordance with one or more embodiments.

[0014] FIG. 6 shows a flowchart for generating virtual sensor data in a virtual world in accordance with one or more embodiments.

[0015] FIG. 7 shows an example of a voxel grid in accordance with one or more embodiments.

[0016] FIG. 8A shows an example of rasterization and ray casting of the same object voxel grid in accordance with one or more embodiments.

[0017] FIG. 8B shows an example algorithm in accordance with one or more embodiments.

[0018] FIG. 9 shows an example of a training process of a voxel grid in accordance with one or more embodiments.

[0019] FIG. 10 shows an example of images generated through using a unified representation in accordance with one or more embodiments.

[0020] FIG. 11A and FIG. 11B show a computing system in accordance with one or more embodiments of the invention.

[0021] Like elements in the various figures are denoted by like reference numerals for consistency.DETAILED DESCRIPTION

[0022] One or more embodiments are directed to a sparse local fields representation for an objects in a virtual environment. Objects in a virtual environment are represented as object voxel grids, which is a grid of voxels. Each voxel corresponds to a particular three-dimensional (3D) portion of the object that store properties of the portion of the object.

[0023] For example, the properties may include voxel positioning properties and appearance properties. The voxel positioning properties describe the positioning information of the voxel in the voxel grid. The appearance property specifies an appearance of a portion of an object within the voxel. For example, the appearance properties include a geometry field, which is a mathematical representation defining the geometry of the object at a particular location within the voxel. Similarly, the color field is a mathematical representation defining the color of the object at a particular location within the voxel. Because of the representation in the object voxel grid, the same voxel grid may be used for both ray casting and for rasterization. Rasterization is a faster method to render a scene while ray casting may be used to render LiDAR point clouds. Thus, a unified representation is achieved for multiple sensor types.

[0024] Additionally, to increase performance, voxels that are empty, e.g., correspond to a region not having a portion of the object, may be pruned from the object voxel grid. Similarly, voxels that have high gradients are partitioned to sub-voxels to increase accuracy. Through pruning and partitioning, the accuracy of the object voxel model is increased, and the computing system resource usage is decreased.

[0025] In one or more embodiments, the processing by a system may be used to generate a virtual world that mimics the real world, but with different scenariosimplemented. For example, the changed scenarios may be that dynamic and / or static objects are in different locations, the perspective of the player is changed because the player is in a different location than the player was in the real world, or other aspects of the real world are different.

[0026] Embodiments of the invention may be used as part of generating a simulated environment for the training and testing of autonomous systems. An autonomous system is a self-driving mode of transportation that does not require a human pilot or human driver to move and react to the real-world environment. Rather, the autonomous system includes a virtual driver that is the decision-making portion of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world. The autonomous system may be completely autonomous or semi-autonomous. As a mode of transportation, the autonomous system is contained in a housing configured to move through a real-world environment. Examples of autonomous systems include self-driving vehicles (e.g., self-driving trucks and cars), drones, airplanes, robots, etc. The virtual driver is the software that makes decisions and causes the autonomous system to interact with the real world including moving, signaling, and stopping or maintaining a current state.

[0027] The real-world environment is the portion of the real world through which the autonomous system, when trained, is designed to move. Thus, the real- world environment may include interactions with concrete and land, people, animals, other autonomous systems, human driven systems, construction, and other objects as the autonomous system moves from an origin to a destination. In order to interact with the real-world environment, the autonomous system includes various types of sensors, such as LiDAR sensors amongst other types, which are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment.

[0028] The testing and training of the virtual driver of the autonomous systems in the real-world environment is unsafe because of the accidents that an untrained virtual driver can cause. Thus, as shown in FIG. 1, a simulator (100)is configured to train and test a virtual driver (102) of an autonomous system. For example, the simulator may be a unified, modular, mixed reality, closed- loop simulator for autonomous systems. The simulator (100) is a configurable simulation framework that enables not only evaluation of different autonomy components in isolation but also as a complete system in a closed-loop manner. The simulator reconstructs “digital twins” of real-world scenarios automatically, enabling accurate evaluation of the virtual driver at scale. The simulator (100) may also be configured to perform mixed-reality simulation that combines real-world data and simulated data to create diverse and realistic evaluation variations to provide insight into the virtual driver’s performance. The mixed reality closed-loop simulation allows the simulator (100) to analyze the virtual driver’s action on counterfactual “what-if’ scenarios that did not occur in the real world. The simulator (100) further includes functionality to simulate and train on rare yet safety-critical scenarios with respect to the entire autonomous system and closed-loop training to enable automatic and scalable improvement of autonomy.

[0029] The simulator (100) creates the simulated environment (104) which is a virtual world. The virtual driver (102) is the player in the virtual world. The simulated environment (104) is a simulation of a real-world environment, which may or may not be in actual existence, in which the autonomous system is designed to move. As such, the simulated environment (104) includes a simulation of the objects (z.e., simulated objects or assets) and background in the real world, including the natural objects, construction, buildings and roads, obstacles, as well as other autonomous and non- autonomous objects. The simulated environment simulates the environmental conditions within which the autonomous system may be deployed. Additionally, the simulated environment (104) may be configured to simulate various weather conditions that may affect the inputs to the autonomous systems. The simulated objects may include both stationary and non-stationary objects. Non-stationary objects are actors in the real-world environment.

[0030] The simulator (100) also includes an evaluator (110). The evaluator (110) is configured to train and test the virtual driver (102) by creating various scenarios in the simulated environment. Each scenario is a configuration of the simulated environment including, but not limited to, static portions, movement of simulated objects, actions of the simulated objects with each other, and reactions to actions taken by the autonomous system and simulated objects. The evaluator (110) is further configured to evaluate the performance of the virtual driver using a variety of metrics.

[0031] The evaluator (110) assesses the performance of the virtual driver throughout the performance of the scenario. Assessing the performance may include applying rules. For example, the rules may be that the automated system does not collide with any other actor, compliance with safety and comfort standards (e.g., passengers not experiencing more than a certain acceleration force within the vehicle, the automated system not deviating from executed trajectory), or other rule. Each rule may be associated with the metric information that relates a degree of breaking the rule with a corresponding score. The evaluator (110) may be implemented as a data-driven neural network that learns to distinguish between good and bad driving behavior. The various metrics of the evaluation system may be leveraged to determine whether the automated system satisfies the requirements of the success criterion for a particular scenario. Further, in addition to system level performance, for modular based virtual drivers, the evaluator may also evaluate individual modules such as segmentation or prediction performance for actors in the scene with respect to the ground truth recorded in the simulator.

[0032] The simulator (100) is configured to operate in multiple phases as selected by the phase selector (108) and modes as selected by a mode selector (106). The phase selector (108) and mode selector (106) may be a graphical user interface or application programming interface component that is configured to receive a selection of phase and mode, respectively. The selected phase and mode define the configuration of the simulator (100). Namely, the selectedphase and mode define which system components communicate and the operations of the system components.

[0033] The phase may be selected using a phase selector (108). The phase may be a training phase or a testing phase. In the training phase, the evaluator (110) provides metric information to the virtual driver (102), which uses the metric information to update the virtual driver (102). The evaluator (110) may further use the metric information to further train the virtual driver (102) by generating scenarios for the virtual driver. In the testing phase, the evaluator (110) does not provide the metric information to the virtual driver. In the testing phase, the evaluator (110) uses the metric information to assess the virtual driver and to develop scenarios for the virtual driver (102).

[0034] The mode may be selected by the mode selector (106). The mode defines the degree to which real -world data is used, whether noise is injected into simulated data, the degree of perturbations of real-world data, and whether the scenarios are designed to be adversarial. Example modes include open-loop simulation mode, closed-loop simulation mode, single module closed-loop simulation mode, fuzzy mode, and adversarial mode. In an open-loop simulation mode, the virtual driver is evaluated with real-world data. In a single module closed-loop simulation mode, a single module of the virtual driver is tested. An example of a single module closed-loop simulation mode is a localizer closed-loop simulation mode in which the simulator evaluates how the localizer estimated pose drifts over time as the scenario progresses in simulation. In a training data simulation mode, the simulator is used to generate training data. In a closed loop evaluation mode, the virtual driver and simulation system are executed together to evaluate system performance. In the adversarial mode, the actors are modified to perform adversarial. In the fuzzy mode, noise is injected into the scenario (e.g., to replicate signal processing noise and other types of noise). Other modes may exist without departing from the scope of the system.

[0035] The simulator (100) includes the controller (112) which includes functionality to configure the various components of the simulator (100) according to the selected mode and phase. Namely, the controller (112) may modify the configuration of each of the components of the simulator based on the configuration parameters of the simulator (100). Such components include the evaluator (110), the simulated environment (104), an autonomous system model (116), sensor simulation models (114), asset models (117), actor models (118), latency models (120), and a training data generator (122).

[0036] The autonomous system model (116) is a detailed model of the autonomous system in which the virtual driver will execute. The autonomous system model (116) includes model, geometry, physical parameters (e.g., mass distribution, points of significance), engine parameters, sensor locations and type, the firing pattern of the sensors, information about the hardware on which the virtual driver executes (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. The various parameters of the autonomous system model may be configurable by the user or another system.

[0037] For example, if the autonomous system is a motor vehicle, the modeling and dynamics may include the type of vehicle (e.g., car, truck), make and model, geometry, physical parameters such as the mass distribution, axle positions, type and performance of the engine, etc. The vehicle model may also include information about the sensors on the vehicle (e.g., camera, LiDAR, etc.), the sensors’ relative firing synchronization pattern, and the sensors’ calibrated extrinsics (e.g., position and orientation) and intrinsics (e.g., focal length). The vehicle model also defines the onboard computer hardware, sensor drivers, controllers, and the autonomy software release under test.

[0038] The autonomous system model includes an autonomous system dynamic model. The autonomous system dynamic model is used for dynamics simulation that takes the actuation actions of the virtual driver (e.g., steering angle, desired acceleration) and enacts the actuation actions on the autonomoussystem in the simulated environment to update the simulated environment and the state of the autonomous system. To update the state, a kinematic motion model may be used, or a dynamics motion model that accounts for the forces applied to the vehicle may be used to determine the state. Within the simulator, with access to real log scenarios with ground truth actuations and vehicle states at each time step, embodiments may also optimize analytical vehicle model parameters or learn parameters of a neural network that infers the new state of the autonomous system given the virtual driver outputs.

[0039] In one or more embodiments, the sensor simulation models (114) models, in the simulated environment, active and passive sensor inputs. Passive sensor inputs capture the visual appearance of the simulated environment including stationary and nonstationary simulated objects from the perspective of one or more cameras based on the simulated position of the camera(s) within the simulated environment. Examples of passive sensor inputs include inertial measurement unit (IMU) and thermal. Active sensor inputs are inputs to the virtual driver of the autonomous system from the active sensors, such as LiDAR, RADAR, global positioning system (GPS), ultrasound, etc. Namely, the active sensor inputs include the measurements taken by the sensors, and the measurements being simulated based on the simulated environment based on the simulated position of the sensor(s) within the simulated environment. By way of an example, the active sensor measurements may be measurements that a LiDAR sensor would make of the simulated environment over time and in relation to the movement of the autonomous system. In one or more embodiments, all or a portion of the sensor simulation models (114) may be or include the rendering system (300) shown in FIG. 3. In such a scenario, the rendering system of the sensor simulation models (114) may perform the operations of FIGs. 5 and 6.

[0040] Measurements or sensor data that are associated with the same sensor (virtual or real) at a single point in time, even if acquired over a duration of acquisition time (e.g., 20-100 microseconds) is referred to as a snapshot. Thus,a virtual sensor snapshot is a set of measurements that is each associated or otherwise assigned the same point in time and acquired by a virtual sensor.

[0041] The sensor simulation models (114) are configured to simulate the sensor observations of the surrounding scene in the simulated environment (104) at each time step according to the sensor configuration on the vehicle platform. When the simulated environment directly represents the real-world environment, without modification, the sensor output may be directly fed into the virtual driver. For light-based sensors, the sensor model simulates light as rays that interact with objects in the scene to generate the sensor data. Depending on the asset representation (e.g., of stationary and nonstationary objects), embodiments may use graphics-based rendering for assets with textured meshes, neural rendering, or a combination of multiple rendering schemes. Leveraging multiple rendering schemes enables customizable world building with improved realism. Because assets are compositional in 3D and support a standard interface of render commands, different asset representations may be composed in a seamless manner to generate the final sensor data. Additionally, for scenarios that replay what happened in the real world and use the same autonomous system as in the real world, the original sensor observations may be replayed at each time step.

[0042] Asset models (117) include multiple models, each model modeling a particular type of individual asset in the real world. The assets may include inanimate objects such as construction barriers or traffic signs, parked cars, and background (e.g., vegetation or sky). Each of the entities in a scenario may correspond to an individual asset. As such, an asset model, or instance of a type of asset model, may exist for each of the objects or assets in the scenario. The assets can be composed together to form the three-dimensional simulated environment. An asset model provides all the information needed by the simulator to simulate the asset. The asset model provides the information used by the simulator to represent and simulate the asset in the simulated environment.

[0043] Closely related to, and possibly considered part of the set of asset models (117) are actor models (118). An actor model represents an actor in a scenario. An actor is a sentient being that has an independent decision-making process. Namely, in the real world, the actor may be an animate being (e.g., a person or animal) that makes a decision based on an environment. The actor makes active movement rather than or in addition to passive movement. An actor model, or an instance of an actor model may exist for each actor in a scenario. The actor model is a model of the actor. If the actor is in a mode of transportation, then the actor model includes the model of transportation in which the actor is located. For example, actor models may represent pedestrians, children, vehicles being driven by drivers, pets, bicycles, and other types of actors.

[0044] The actor model leverages the scenario specification and assets to control all actors in the scene and their actions at each time step. The actor’s behavior is modeled in a region of interest centered around the autonomous system. Depending on the scenario specification, the actor simulation will control the actors in the simulation to achieve the desired behavior. Actors can be controlled in various ways. One option is to leverage heuristic actor models, such as an intelligent-driver model (IDM) that try to maintain a certain relative distance or time-to-collision (TTC) from a lead actor or heuristic-derived lanechange actor models. Another is to directly replay actor trajectories from a real log or to control the actor(s) with a data-driven traffic model. Through the configurable design, embodiments may mix and match different subsets of actors to be controlled by different behavior models. For example, far-away actors that initially may not interact with the autonomous system and can follow a real log trajectory, but when near the vicinity of the autonomous system may switch to a data-driven actor model. In another example, actors may be controlled by a heuristic or data-driven actor model that still conforms to the high-level route in a real log. This mixed-reality simulation provides control and realism.

[0045] Further, actor models may be configured to be in cooperative or adversarial mode. In cooperative mode, the actor model models actors to act rationally in response to the state of the simulated environment. In adversarial mode, the actor model may model actors acting irrationally, such as exhibiting road rage and bad driving.

[0046] In one or more embodiments, the actor models (118), asset models (117), and background may be part of the rendering system (described below with reference to FIG. 3). As another example, the system may be a bifurcated system whereby the operations (e.g., trajectories or positioning) of the assets and actors are defined separately from the appearance, which is part of the rendering system. Actors and assets are objects, and the visual representation of the object may be represented as the object voxel grids. The trajectories of the actors and assets may be represented in a separate model or in data related to the corresponding object voxel grid.

[0047] The latency model (120) represents timing latency that occurs when the autonomous system is in a real-world environment. Several sources of timing latency may exist. For example, a latency may exist from the time that an event occurs to the sensors detecting the sensor information from the event and sending the sensor information to the virtual driver. Another latency may exist based on the difference between the computing hardware executing the virtual driver in the simulated environment as compared to the computing hardware of the virtual driver. Further, another timing latency may exist between the time that the virtual driver transmits an actuation signal to the autonomous system changing (e.g., direction or speed) based on the actuation signal. The latency model (120) models the various sources of timing latency.

[0048] Stated another way, in the real world, safety-critical decisions in the real world may involve fractions of a second affecting response time. The latency model simulates the exact timings and latency of different components of the onboard system. To enable scalable evaluation without strict requirements on exact hardware, the latencies and timings of the different components of theautonomous system and sensor modules are modeled while running on different computer hardware. The latency model may replay latencies recorded from previously collected real-world data or have a data-driven neural network that infers latencies at each time step to match the hardware in a loop simulation setup.

[0049] The training data generator (122) is configured to generate training data.For example, the training data generator (122) may modify real-world scenarios to create new scenarios. The modification of real-world scenarios is referred to as mixed reality. For example, mixed-reality simulation may involve adding in new actors with novel behaviors, changing the behavior of one or more of the actors from the real world, and modifying the sensor data in that region while keeping the remainder of the sensor data the same as the original log. In some cases, the training data generator (122) converts a benign scenario into a safety-critical scenario.

[0050] The simulator (100) is connected to a data repository (105). The data repository (105) is any type of storage unit or device that is configured to store data. The data repository (105) includes data gathered from the real world. For example, the data gathered from the real world include real actor trajectories (126), real sensor data (128), real trajectories of the system capturing the real world (130), and real latencies (132). Each of the real actor trajectories (126), real sensor data (128), real trajectory of the system capturing the real world (130), and real latencies (132) is data captured by or calculated directly from one or more sensors from the real world (e.g., in a real-world log). In other words, the data gathered from the real world are actual events that happened in real life. For example, in the case that the autonomous system is a vehicle, the real-world data may be captured by a vehicle driving in the real world with sensor equipment.

[0051] Further, the data repository (105) includes functionality to store one or more scenario specifications (140). A scenario specification (140) specifies a scenario and evaluation setting for testing or training the autonomous system.For example, the scenario specification (140) may describe the initial state of the scene, such as the current state of the autonomous system (e.g., the full 6D pose, velocity and acceleration), the map information specifying the road layout, and the scene layout specifying the initial state of all the dynamic actors and objects in the scenario. The scenario specification may also include dynamic actor information describing how the dynamic actors in the scenario should evolve over time which are inputs to the actor models. The dynamic actor information may include route information for the actors, desired behaviors or aggressiveness. The scenario specification (140) may be specified by a user, programmatically generated using a domain-specification language (DSL), procedurally generated with heuristics from a data-driven algorithm, or adversarial-based generated. The scenario specification (140) can also be conditioned on data collected from a real-world log, such as taking place on a specific real-world map or having a subset of actors defined by their original locations and trajectories.

[0052] The interfaces between the virtual driver and the simulator match the interfaces between the virtual driver and the autonomous system in the real world. For example, the sensor simulation model (114) and the virtual driver match the virtual driver interacting with the sensors in the real world. The virtual driver is the actual autonomy software that executes on the autonomous system. The simulated sensor data that is output by the sensor simulation model (114) may be in or converted to the exact message format that the virtual driver takes as input as if the virtual driver were in the real world, and the virtual driver can then run as a black box virtual driver with the simulated latencies incorporated for components that run sequentially. The virtual driver then outputs the exact same control representation that it uses to interface with the low-level controller on the real autonomous system. The autonomous system model (116) will then update the state of the autonomous system in the simulated environment. Thus, the various simulation models of the simulator (100) run in parallel asynchronously at their own frequencies to match the real- world setting.

[0053] FIG. 2 shows a flow diagram for executing the simulator in a closed-loop mode. In Block 201, a digital twin of a real-world scenario is generated as a simulated environment state. Log data from the real world is used to generate an initial virtual world. The log data defines which asset and actor models are used in the initial positioning of assets. For example, using convolutional neural networks on the log data, the various asset types within the real world may be identified. As other examples, offline perception systems and human annotations of log data may be used to identify asset types. Accordingly, corresponding asset and actor models may be identified based on the asset types and add to the positions of the real actors and assets in the real world. Thus, the asset and actor models create an initial three-dimensional virtual world. The positions of the asset and actor models within a virtual world form a scene. The planned trajectories of the actors through the virtual world create a scenario.

[0054] In Block 203, the sensor simulation model is executed on the simulated environment state to obtain simulated sensor output. The sensors that are simulated may include wide angle camera sensors, LiDAR sensors, and standard camera sensors using the same object voxel models and background models as described in FIG. 6. Each sensor of the autonomous system has a corresponding sensor simulation model and a corresponding system. The rendering system of FIG. 3 with the sensor simulation model executes based on the position of the sensor within the virtual environment and generates simulated sensor output. The simulated sensor output is in the same form as would be received from a real sensor by the virtual driver. Sensor attributes of the sensor, including the sensor intrinsics and extrinsics, are used to simulate the sensor. For example, location, viewing direction, and resolution of the sensor with respect to the autonomous vehicle may be used to replicate the originating location of the corresponding virtual sensor on the simulated autonomous system. Thus, the various sensor inputs to the virtual driver match the combination of inputs if the virtual driver were in the real world.

[0055] The simulated sensor output is passed to the virtual driver. In Block 205, the virtual driver executes based on the simulated sensor output to generate actuation actions. The actuation actions define how the virtual driver controls the autonomous system. For example, for a self-driving vehicle, the actuation actions may be the amount of acceleration, movement of the steering, triggering of a turn signal, etc. From the actuation actions, the autonomous system state in the simulated environment is updated in Block 207. The actuation actions are used as input to the autonomous system model to determine the actual actions of the autonomous system. For example, the autonomous system dynamic model may use the actuation actions in addition to road and weather conditions to represent the resulting movement of the autonomous system. For example, in a wet or snowy environment, the same amount of acceleration action as in a dry environment may cause less acceleration than in the dry environment. As another example, the autonomous system model may account for possibly faulty tires (e.g., tire slippage), mechanical based latency, or other possible imperfections in the autonomous system.

[0056] In Block 209, actors’ actions in the simulated environment are modeled based on the simulated environment state. Concurrently with the virtual driver model, the actor models and asset models are executed on the simulated environment state to determine an update for each of the assets and actors in the simulated environment. Here, the actors’ actions may use the previous output of the evaluator to test the virtual driver. For example, if the actor is adversarial, the evaluator may indicate based on the previous action of the virtual driver, the lowest scoring metric of the virtual driver. Using a mapping of metrics to actions of the actor model, the actor model executes to exploit or test that particular metric.

[0057] Thus, in Block 211, the simulated environment state is updated according to the actors’ actions and the autonomous system state to generate an updated simulated environment state. The updated simulated environment includes the change in positions of the actors and the autonomous system. Because themodels execute independently of the real world, the update may reflect a deviation from the real world. Thus, the autonomous system is tested with new scenarios. In Block 213, a determination is made whether to continue. If the determination is made to continue, testing of the autonomous system continues using the updated simulated environment state in Block 203. At each iteration, during training, the evaluator provides feedback to the virtual driver. Thus, the parameters of the virtual driver are updated to improve the performance of the virtual driver in a variety of scenarios. During testing, the evaluator is able to test using a variety of scenarios and patterns including edge cases that may be safety critical. Thus, one or more embodiments improve the virtual driver and increase the safety of the virtual driver in the real world.

[0058] As shown, the virtual driver of the autonomous system acts based on the scenario and the current learned parameters of the virtual driver. The simulator obtains the actions of the autonomous system and provides a reaction in the simulated environment to the virtual driver of the autonomous system. The evaluator evaluates the performance of the virtual driver and creates scenarios based on the performance. The process may continue as the autonomous system operates in the simulated environment.

[0059] FIG. 3 shows a diagram of the rendering system (300) in accordance with one or more embodiments. The rendering system (300) is a system configured to generate virtual sensor input using voxel grids for objects. In particular, the rendering system (300) may be configured to render virtual sensor snapshots. A virtual sensor snapshot is a set of measurements that is each associated or otherwise assigned the same point in time and acquired by a virtual sensor. For example, a virtual sensor snapshot may be a camera snapshot (e.g., camera images) and LiDAR snapshot (e.g., LiDAR point clouds). The rendering system (300) includes a data repository (302) connected to a model framework (304). The data repository (302) includes real sensor data (128), sensor attributes (303), object voxel grids (e.g., object voxel grid X (306), object voxel grid Y (308)), and a background voxel grid (310).

[0060] The real sensor data (128) is the sensor data described above with reference to FIG. 1. The real sensor data (128) may include LiDAR point clouds and actual images. LiDAR point clouds are point clouds captured by LiDAR sensors performing a LiDAR sweep of a geographic region. The term LiDAR point clouds used herein means any LiDAR data captured through a LiDAR sweep with or without transformation into a LiDAR image. For example, a LiDAR sweep of a region may be captured to generate a set of LiDAR points, each LiDAR point having a distance and an angle at which an object is detected. A LiDAR image may be generated from the set of LiDAR points that is a transformation of the distance and angle into two-dimensional or three-dimensional space. Thus, the LiDAR point cloud may be the original set of LiDAR points, may refer to the LiDAR image, or may be another representation of LiDAR data. Actual images are images captured by one or more cameras of the geographic region. For example, as a sensing vehicle is moving through a geographic region the sensing vehicle may have cameras and LiDAR sensors that gather sensor data from the geographic region.

[0061] In some embodiments, the real sensor data (128) may be the time series of data that is captured along the trajectory of the sensing vehicle. As such, the real sensor data (128) generally omits several side views of three-dimensional objects. Cameras on the sensing vehicle can capture directly, the sides of other vehicles that are parked along the street as well as a small amount of the front and back of the parked vehicles that are not hidden. The camera may also capture images of another vehicle being driven in front of the sensing vehicle. When the other vehicle turns, the camera may capture a different side but does not capture the front. As such, the real sensor data (128) may be imperfect as it does not capture the three-hundred-and-sixty-degree view of the objects.

[0062] The sensor attributes (303) are attributes of a sensor, real or virtual. For example, during training, the sensor attributes are attributes of real-world sensors used to capture real sensor data. The sensor attributes (303) are properties of the sensor that affect how the sensor data is captured. Forexample, the sensor attributes (303) include sensor extrinsics, such as the location of the sensor and the orientation of the sensor. The sensor attributes (303) also include sensor intrinsics including the resolution of the sensor, scanning speed, viewing angle of the sensor, focal information, type of sensor, and other information about the corresponding sensor.

[0063] The object voxel grids (e.g., object voxel grid X (306), object voxel grid Y (308)) are three-dimensional models of objects. The object voxel grids (e.g., object voxel grid X (306), object voxel grid Y (308)) is a grid of voxels defined for an object. A voxel is a rectangular cuboid in which the sides have ninety degree angles with respect to each other. In some embodiments, the rectangular cuboid is a cube having all sides of equal length.

[0064] Each location on the object has a corresponding location in the object voxel grid, whereby the relative locations between two locations on the object match the relative locations. In one or more embodiments, the object voxel grid may represent the object at a variety of resolutions. Specifically, certain part of the object may be represented at a higher resolution than other parts of the object. The variation in resolutions may be performed by partitioning certain voxels into sub-voxels. Each sub-voxel may have an individual set of properties that is defined for the sub-voxel. In one or more embodiments, the object voxel grid is stored as a tree, such as an octree. In an octree, a volume is partitioned into eight sub-volumes. At a highest level, the root may have any number of children. When a voxel is partitioned, the voxel may be partitioned into eight sub-voxels, which may be further partitioned. The sub-voxels of the voxel and the voxel have a parent child relationship whereby a sub-voxel is a child voxel of the parent voxel according to containment. Some of the voxels may be pruned from the octree in one or more embodiments.

[0065] A background voxel grid (310) is a voxel grid that is defined for the background region. Namely, the background voxel grid (310) may be the same as an object voxel grid but defined for a background region and include multiple objects. Specifically, objects in the background region may be treated as aunitary whole with other objects in the background and, thus, not move separately from the other objects. Multiple background models, such as background voxel grid (310), may be defined for different levels of background (e.g., a target region background and an external region background).

[0066] In one or more embodiments, the voxel grid (400) with the voxels (e.g., voxel M (402), voxel N (404)) is shown in FIG. 4. The voxel grid (400) may be a background voxel grid or object voxel grid in FIG. 3. Specifically, FIG. 4 shows an exploded view of a voxel. As shown in FIG. 4, the object voxel grid (400) includes multiple voxels (e.g., voxel M (402), voxel N (404)). FIG. 4 shows an exploded view (406) of a voxel that may be any of the multiple voxels. Further, each of the multiple voxels may have the same properties as in the exploded view (406).

[0067] As shown in FIG. 4, a voxel's properties are defined by the voxel positioning properties (408) and appearance properties (411). The voxel positioning properties (408) define the voxel's spatial pose. The voxel positioning properties (408) include a position property (410) specifying translation, a size property (412) specifying scale, and an orientation property (414) specifying rotation. The position property (410) is a three-dimensional coordinate that defines the voxel's location. The size property (412) defines the voxel's dimensions, such as a single value for a cubic voxel or distinct lengths for a rectangular cuboid. The orientation property (414) defines the voxel's rotational state with respect to the global coordinate system. The voxel positioning properties (408) collectively define a transformation from the global coordinate system to the voxel's local coordinate system.

[0068] The appearance properties (411) define the appearance (e.g., visual, LiDAR, etc.) of the portion of the object represented by the voxel. Namely, the appearance properties (411) define the values returned for the voxel while the voxel positioning properties (408) identify which voxels to use to return the values. In one or more embodiments, the appearance properties are field properties. A field property does not identify a corresponding appearance ofthe voxel as a whole voxel (e.g., the color of the voxel as a whole) or a value for a single point in the voxel. Rather, a field property refers to a value or set of values that describes how a corresponding measurable attribute (i.e., a particular appearance property) varies over a region (i.e., the portion of the object represented by the voxel). The field property defines a set of values, which, when mathematically combined with location identifier, outputs a corresponding appearance value for the location. Namely, the field property directly identifies the corresponding values rather than a weight that may be used to predict the values. The combination may be matrix multiplication or other such similar small computation. The field properties are the parameters (e.g., a 3x3 matrix for color) for a simple, computationally shallow function that is defined locally within each voxel.

[0069] The appearance properties (411) may include a geometry field property (416), a color field property (418), and a spherical harmonic field property (420). The geometry field property (416) is a set of learnable parameters that defines the volumetric density of the content within the voxel (e.g., density of the volumetric content). In one or more embodiments, the set of learnable parameters is a matrix of coefficients. To determine the density at a specific point inside the voxel, the point's local coordinates are mathematically combined with the learnable parameters. For example, the mathematical combination may be between matrices. The result is then processed through an activation function, such as an exponential function, to yield a final density value. This density value represents how opaque or transparent the object is at that specific point.

[0070] The appearance properties (411) also define the view-dependent color of the volumetric content (e.g., content within the voxel). The view-dependent color may be achieved through a combination of a color field property (418) and a spherical harmonic field property (420). The color field property (418) may define the voxel's base color, independent of the viewing angle. The color field property may be represented as a matrix of coefficients. Similar to thegeometry field, the local coordinates of a point may be combined with this matrix to produce a base color value.

[0071] The spherical harmonic field property (420) is a set of parameters, such as a matrix of coefficients, that models view-dependent lighting effects, such as shininess or reflections. To calculate this effect, the incoming viewing direction may be first encoded into a set of spherical harmonic basis values. These basis values may then be combined with the property's matrix of coefficients to produce a color modification or offset.

[0072] The final color for a point may be determined by combining the base color from the color field property with the view-dependent modification from the spherical harmonic field property. This combined result is typically passed through a final activation function, such as a sigmoid function, to ensure the output values are in a valid color range. This composite structure allows the voxel to represent surfaces that realistically change appearance depending on the angle from which they are viewed.

[0073] Although FIG. 4 shows some of the properties of a voxel, the voxel may include other properties not shown in FIG. 4 without departing from the scope of the invention.

[0074] Returning to FIG. 3, the rendering system (300) includes a model framework (304) connected to a data repository (302). The model framework (304) includes a ray casting engine (344), a rasterization engine (346), a voxel grid generator (348), and a loss function (350). A ray casting engine (344) is configured to define and cast rays to a target object from a sensor. For an object, the ray has a first endpoint at the virtual sensor and a second endpoint on an opposing side of the target object or where the ray intersects the target object. The ray may pass through the target object. As such, the ray passes through at least a near point on the surface of the target object and a far point on the surface of the target object. Because the ray is a line, an infinite number of locations are along the ray.

[0075] One or more embodiments use a sampling technique to sample locations along the ray and determine feature descriptors for each location using the rest of the model framework (304). In one or more embodiments, the ray casting engine (344) is configured to aggregate the feature descriptors along the locations of the ray. A single ray may pass through multiple objects. Thus, the accumulation may be through the multiple objects. At each sampled point, the ray casting engine (344) is configured to select the voxel having the portion of the object for the sampled point and calculate the corresponding value from the selected voxel. The ray casting engine is further configured to accumulate the values along the ray.

[0076] The rasterization engine (346) is configured to perform rasterization to generate an image. Rasterization involves identifying an image plane for an image, projecting the voxel grids onto the image plane, and compositing the voxels. The image plane is a virtual two-dimensional surface on which the image is formed. The image plane is defined by the position of the corresponding virtual sensor and is defined by the intrinsics of the virtual sensor. The projecting includes matching locations in the voxel to a location on the image plane.

[0077] In one or more embodiments, rasterization is performed for pinhole cameras. The ray casting engine may be used to generate a LiDAR point cloud and to handle other types of cameras. The voxel grids provide a uniform representation for both types of image generation.

[0078] The voxel grid generator (348) is configured to generate a voxel grid for an object. Specifically, the voxel grid generator (348) is configured to generate a voxel grid for an object or a background. In one or more embodiments, the voxel grid generator (348) may perform the operations of FIG. 5. The loss function (350) operates with the voxel grid generator (348) to determine a loss for the voxel grid. The loss reflects the difference between the actual sensor data and predicted sensor data. In one or more embodiments, the loss function further includes a regularization term. The regularization term compares eachpoint to the point’s neighbors to have a smooth transition between adjacent points.

[0079] FIGs. 5 and 6 show flowcharts in accordance with one or more embodiments. FIG. 5 shows a flowchart for training the voxel grid and FIG. 6 shows a flowchart for rendering using the voxel grids. While the various steps in these flowcharts are presented and described sequentially, at least some of the steps may be executed in different orders, may be combined or omitted, and at least some of the steps may be executed in parallel. Furthermore, the steps may be performed actively or passively.

[0080] FIG. 5 shows a flowchart for training object voxel grids in accordance with one or more embodiments. Block 502 includes initializing a voxel grid by partitioning a bounding box of an object into voxels. The real sensor data is processed by an object detection model to identify objects. For example, the object detection model may include an image based neural network or point cloud based neural network that may process camera snapshots and LiDAR snapshots to identify and classify individual objects. As part of identifying and classifying objects, the object detection model outputs a bounding box around the object. In one or more embodiments, the object bounding box is the smallest rectangular cuboid. The object bounding box is partitioned into voxels. In one or more embodiments, the voxels are non-overlapping and span the object bounding box. The voxels may be equal sized and large. For example, a vehicle may initially be partitioned into eight or twelve voxels. Other partitioning schemes may be used.

[0081] Block 504 includes setting a size property value, a position property value, and an orientation property value for the voxels. The voxel positioning properties are based on partitioning a bounding box around an object. Namely, the size property value is determined directly from the size of the voxel. The size may be relative to the size of the overall object or relative to a uniform size. In one or more embodiments, the position property value is directly determined from the origin point of the object. For example, the positionproperty value may be defined as the location of the particular voxel as compared to the voxels of the same level of the hierarchy (e.g., the overall object level initially). The orientation property value, represented as a quaternion, is initialized to an identity quaternion. The scene may be represented compositionally, with static voxels defined in a global coordinate system and dynamic actor voxels defined in a local, canonical coordinate system. For rendering, the set of voxels corresponding to a dynamic actor is transformed into the global scene using the actor's pose (i.e., its global orientation and position) at a given timestamp. The orientation of dynamic elements is defined as identity quaternion in the object-centric coordinates, and transformed to global coordinates, dictated by the actor-level transformation.

[0082] Block 506 includes obtaining real sensor data for voxels in the voxel grid.Real sensor data may be obtained as described above based on actual sensors moving through a real-world environment and capturing images and LiDAR point clouds.

[0083] Block 508 includes initializing an opacity value for the voxels according to the real sensor data. The opacity value may be defined based on a number of LiDAR points detected as existing in the voxel based on the LiDAR point cloud. If the LiDAR point cloud indicates that no points are in the particular voxel, then the opacity value may be set to a low value (e.g., less than 0.2). Conversely, voxels that correspond to more than a threshold number of LiDAR points may be initialized to a higher opacity value (e.g., greater than 1).

[0084] Block 510 includes initializing the geometry field property, the color field property, and the spherical harmonic field property. In one or more embodiments the appearance field properties are randomly initialized. For example, a linear or other random initialization may be performed. Thus, the processing of Blocks 504-510 initializes multiple voxel properties of a current voxel object grid.

[0085] The processing of Blocks 502-510 may be performed for each of multiple objects and the background that are present in training data. Thus, the systemmay be concurrently trained for the various objects in the scenario and for the background. If a new object is to be added, actual sensor data may be captured for the new object and the voxel grid for the new object trained without training the remaining voxel grids.

[0086] Block 512 includes composing a training scene to match the real sensor data. The object voxel grids as well as the background voxel grid are added to the virtual environment according to the locations in the real sensor data. Specifically, the background is added to the virtual environment. The object voxel grids are assigned the corresponding location where the object is detected in the real sensor data.

[0087] Block 514 includes performing ray casting and rasterization of the training scene with the current object voxel grid according to sensor attributes of the sensors generating the real sensor data to generate predictive sensor data. In one or more embodiments, the extrinsics of a particular sensor are used to determine the location and viewing direction of the corresponding virtual sensor in the virtual environment. The intrinsics of the particular sensor are used to predict the sensor data. Namely, the intrinsics of the particular sensor may be used to determine the viewing angle, viewing direction, focal, and other aspects for generating predictive sensor data. Performing the ray casting and rasterization is described in reference to FIG. 6.

[0088] Continuing with FIG. 5, Block 516 includes updating the voxel properties based on a comparison between the real sensor data and the predictive sensor data. Because the sensor properties of the actual sensor are replicated in the virtual sensor, the predictive sensor data should match the actual sensor data. A color loss is calculated based on the difference between the actual color value of each point in the predictive camera snapshot and the actual camera snapshot. A depth loss is calculated between the actual LiDAR point cloud and the predictive LiDAR point cloud. The color loss may be combined with the LiDAR loss. Further, regularization and smoothness terms may be added to the color loss combined with the LiDAR loss to generate a total loss. The totalloss may be used to update the corresponding voxel grids. Thus, the object voxel grids are trained using a loss calculated from the virtual camera snapshot and the virtual LiDAR snapshot. The background voxel grids may be similarly trained.

[0089] In one or more embodiments, Block 516 includes updating the learnable parameters of the voxels based on a total loss function. The total loss is calculated on the entire rendered scene, which is a composition of the static background voxel grids and one or more dynamic object voxel grids. Specifically, a predictive camera snapshot and a predictive LiDAR snapshot are rendered from the composite scene. The total loss function, L, is then computed as a weighted sum of multiple components. A color loss, L_color, is calculated as the l1 distance between the predictive camera snapshot and a ground-truth camera snapshot (e.g., actual camera image). A depth loss, L_depth, is calculated as the l1 distance between the predictive LiDAR point cloud and a ground-truth LiDAR point cloud (e.g., actual LiDAR point cloud). Furthermore, the total loss may include a regularization term, L_reg, which enforces spatial consistency between adjacent voxels, such as maintaining smooth surface (SDF) and color transitions. The resulting total loss L is then minimized using an optimizer, which backpropagates gradients to simultaneously update the parameters of all voxels, both background and object, which contributes to the rendered scene.

[0090] In Block 518, a determination is made whether to continue updating the voxel grids. If the determination is made to continue updating the voxel grids, then the processing continues with Block 512. The processing of Blocks 512— 518 may be performed for several iterations (e.g., 300-500 iterations).

[0091] Block 520 includes pruning a subset of the voxels having an opacity value failing to satisfy an opacity threshold. If the opacity value after the updating of Block 516 is determined to be below the opacity threshold, then the voxel may be determined to correspond to an empty space through which no part of the object exists. Thus, the voxel may be removed from the object voxel grid.

[0092] Block 522 includes partitioning a voxel in the voxel grid based on a gradient of a color field property of the voxel satisfying a gradient threshold. If the gradient of the color field property is greater than a threshold, then the system determines that the portion of the object exhibits high variability. Thus, the voxel is partitioned into sub-voxels in order to create a more accurate model of the object. Notably, voxels that do not satisfy the gradient threshold are not partitioned. By having different levels of partitioning, the object voxel grid is able to manage resource usage while having an accurate model. The partitioning of a voxel may be performed in a similar manner to partitioning the bounding box as described above with reference to Block 502. In the partitioning, the sub-voxels are added as child voxels of the voxel that is partitioned. The sub-voxels may be assigned a position property value relative to the voxel that is partitioned. The size property and orientation property may be defined based on the size and orientation of the sub-voxel. Further, the subvoxels may each inherit the appearance properties of the parent of the subvoxel. Through training described above, the appearance properties are updated.

[0093] Block 524 includes determining whether to continue. If the determination is made to continue, the processing continues with Block 512.

[0094] FIG. 6 includes rendering operations using a voxel grid in accordance with one or more embodiments. Block 602 includes obtaining a scene including multiple object voxel grids for multiple objects and a background voxel grid. As discussed above, the object voxel grid includes a voxel positioning property, a geometry field property, and a color field property of the voxel in the object voxel grid.

[0095] Block 604 includes identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene. The image plane is defined by the extrinsics and intrinsics of the corresponding virtual sensor. Similarly, the virtual sensor properties of the second virtual sensor may be similarly acquired.

[0096] Block 606 includes performing, using the object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot. The rasterization is performed according to the voxel positioning property and the color field property of the voxel. The ray casting is performed according to the voxel positioning property and the geometry field property. Performing rasterization of the scene on the image plane using the object voxel grids to generate a virtual snapshot. Rasterization may further include adding a background voxel grid for multiple background objects. The voxels of the background voxel grid and the object voxel grid are projected onto the image plane based on the distance of the respective voxel grid in the scene to the image plane. Thus, closer voxel grids to the image plane may be projected initially and farther voxel grids may be projected after the closer voxel grids. The result is a projection. The voxels in the projection are composited. Compositing voxels is based on the opacity of the voxels. For example, the compositing may be based on whether the voxels represent a transparent portion of the object or an opaque portion of the object. If opaque, then the closest voxel defines the color and the later voxels, or portions thereof, that are covered by the closest voxel are ignored. Thus, the rasterization is performed according to the voxel positioning property.

[0097] For color, such as in a virtual camera image, the color field property may further be used to perform rasterization. For a particular voxel, locations within the voxel may be multiplied using matrix multiplication by the color field property of the voxel to generate a resulting color value for the location. The spherical harmonic field property may also be used to augment the color value. For example, the location may also be combined by matrix multiplication with the spherical harmonic field property to generate a spherical harmonic value. The spherical harmonic value may be mathematically combined with the color value to generate a color of the voxel.

[0098] Ray casting through the scene may be performed using the object voxel grids according to the voxel positioning property and the geometry field property to generate the second virtual sensor snapshot. During ray casting, the hierarchy of the object voxel grid may be traversed. The traversal may be performed according to a hierarchical containment relationship between the voxels. For example, if the ray passes through a parent voxel, then a determination is made which of the child voxels are touched by the ray. The traversal then continues with each of the child voxels that are touched by the ray. Thus, the traversing includes detecting a parent voxel of the voxels and traversing to a child voxel of the parent voxel in the voxels responsive to detecting an intersection of a ray in ray casting with the child voxel.

[0099] Once the voxels touched by the ray are identified, multiple locations along the ray are sampled. For a particular voxel that is intersected by the ray, one or more points along the segment of the ray that intersects the voxel may be sampled. Namely, the segment is a portion of the ray that passes through the particular voxel. For example, the sampling may include, for a voxel, selecting a midpoint of a segment of the ray that intersects the voxel. The midpoint may be used as the value for the point. For example, a density value and a color value at the midpoint may be calculated using the geometry field property and the color field property. As discussed above, the location of the midpoint may be mathematically combined using matrix multiplication with the geometry field property and the color field property to generate a depth in the case of a LiDAR point cloud or a color value for a pixel of the virtual camera image.

[0100] Block 608 includes outputting the first virtual sensor snapshot and the second virtual sensor snapshot. The processing of Blocks 602-612 generates virtual sensor data, the virtual sensor data is output by the rendering system and may be used as described in FIG. 2 or to a different system.

[0101] FIG. 7-10 and the description below show an example implementation in accordance with one or more embodiments. As described below, the example implementation includes a volumetric representation that supports efficient tile-based rasterization and flexible, high-fidelity ray casting of complex scenes. The following is a description of the scene representation and rasterization and ray casting rendering algorithms. The coarse-to-fine densification may be performed for compactness and efficient training.

[0102] FIG. 7 shows an example representation of an object voxel grid (700) in accordance with one or more embodiments. Turning to the representation, scenes may be represented using a sparse grid of local implicit fields that map global 3D coordinates and view-directions to spatial properties such as density and color. Let V c3be a 3D volume in an axis-aligned bounding box (AABB) with dimensions (V^, Vw, Vd). One or more embodiments partition V into a regular grid Cj of dimensions [1 / s0] x [Vw / s0] x [Vd / s0], where each voxel is a cube with edge length s0, having a height h, width w, and depth d. Each voxel supports recursive sub-division into eight smaller voxels, up to K levels s0...sk). To efficiently handle large-scale scenes, one or more embodiments may employ a sparse representation that stores only non-empty voxels.

[0103] Each voxel may be characterized by its static geometric parameters (e.g., positioning properties): position p G IK3, scale s 6 ]K, and rotation q E IK4, q represents the orientation relative to the global coordinate system, with all voxels initialized to the identity quaternion in the global frame. Each voxel may also contain a geometry field WaE IKlx4and a color field I / FcG IK3x3, along with 2nd order spherical harmonics WshE IK3x4for view-dependent lighting effects. For any point inside the voxel, letx G [— 1,1]3denote its normalized local coordinates, and x = [x, 1] be its homogeneous representation. The geometry fieldCT(x; 14^.): [—1,1]3IK+computes the density as equation (1):o- = fa(x; Wa) = exp(WaxT). (1)

[0104] Similarly, the color fieldc(x, m; 14^, I4 / J: [— 1,1]3X §2— > [0,l]3computes the color value as equation (2):c = sigmoid (l / FcxT+ Wshy (m)) (2)where ) E §2is the view direction and y: §2— > IE4maps the view direction to spherical harmonics basis coefficients.

[0105] One or more embodiments may render the color C for each ray by accumulating the color and opacity values of intersected voxels along the ray, following the volume rendering equation in equation (3):NcC = ^ Ttatct, (3)i=l where Ncis the number of intersected voxels along the ray, Ttrepresents the accumulated transmittance Tt= 2}=i(l— aj) andat denotes the opacity computed from the density valueand ray segment length 8 within the z-th voxel:= 1 - exp(— < J£5£). (4)

[0106] The density < J£and color ctvalues may be sampled at the mid-point of each ray-voxel intersection segment using Eq. (1) and Eq. (2), respectively.

[0107] With respect to surface parameterization of the object’s surface, autonomous driving scenes have high-quality surfaces to enable accurate LiDAR simulation and secondary ray effects such as reflection. Instead of directly representing density as 14^., a signed distance function (SDF) for better surface parameterization may be used. The geometry SDF field denoted as Ws, quantifies the signed distance (s±) between a point x and the surface using equation (5):s±= WST(5)

[0108] In equation (5), s±may be transformed to density <7 using equation (7):a = - + -sign(s±)(l - e“ls±l / / ?). (6)

[0109] The final learnable parameters of each voxel are the geometry field I / Fs, the color field Wc, the spherical harmonics Wsh, and the SDF-to-density parameters a, b.

[0110] As shown in FIG. 8 A, one or more embodiments is interoperable and can be rendered via ray casting (800) through the volume and computing ray -voxel intersections, or by splatting and compositing voxels (802) onto the image plane (804) (z.e., rasterization). Both can be implemented following the volume rendering equation in Eq. (3), with rasterization (802) being faster but making more approximations in the image formation process. The ray casting (800) approach is more flexible and can handle more complex physics phenomena and sensor models. One or more embodiments now discuss in more detail our efficient implementations of each.

[0111] Ray casting may be performed with octree acceleration by casting rays through the voxels from the sensor origin, sampling points, and accumulating their color and opacity. To accelerate raymarching and ray-voxel intersection checks, one or more embodiments employ an octree data structure. The octree recursively partitions the volume into eight sub-volumes, where non-leaf nodes maintain pointers to their sub-volumes, and leaf nodes either store -1 for empty space or a pointer to the corresponding voxel. This hierarchical structure enables fast ray traversal through the volume. Through a single ray-box intersection test, empty regions can be bypassed. Upon encountering nonempty nodes, traversal selectively descends into intersected child octants with logarithmic complexity.

[0112] Octree-based ray marching combines hierarchical space partitioning with efficient memory layout for fast volume rendering. This includes at least two aspects, the linear octree storage format and the hierarchical ray marching algorithm. FIG. 8B shows an example set of algorithms (806) for ray casting.

[0113] Tile-based splatting may be performed as follows. Images may be rendered by splatting, which projects voxels onto the image plane, compositing them with alpha blending. The image plane may be divided into a grid of 16 * 16 tiles. For each tile, one or more embodiments perform view-frustum culling to identify and sort relevant voxels. Each tile may be rendered by a thread block to iterate over the relevant voxels. The relevant voxels may be preloaded intoshared memory to reduce global memory access, which is a key optimization that significantly accelerates rendering. The opacity for each voxel is computed based on current pixel’s ray travel distance within each voxel using Eq. (4), and the color is sampled from the color field at the intersection point. For a pinhole camera with primary rays, the splatting may exhibit significant acceleration through tile-based processing and shared memory.

[0114] For the tile-based rasterization, one or more embodiments converts each pixel into a 3D ray and computes ray -voxel intersections to obtain the traversal distance (interval 5) and midpoint. The density may then be computed using this midpoint, and opacity is derived using equation (4). One or more embodiments may use predefined static size, position, and orientation parameters using additional voxels high-fidelity scene representation.

[0115] With respect to initialization and densification, naive uniform voxelization of large-scale scenes at high resolution may have prohibitive memory consumption that scales cubically with scene size. To address this challenge, one or more embodiments may adopt a coarse-to-fine approach as shown in FIG. 9. During training, one or more embodiments initialize the scene with a coarse representation (900) and apply an adaptive densification and pruning strategy (902). Voxels exhibiting significant color field gradients may be subdivided (partitioned) into eight child voxels in an octree-aligned manner. Voxels with negligible opacity values may be removed from the sparse set. Densification and pruning together enables preserving of fine details while maintaining a compact memory footprint.

[0116] The Axis-Aligned Bounding Box (AABB) that encompasses actor trajectories in the world coordinate frame. The boundary may be extended by 10m upward, 5m downward, and 40m laterally to define an inner region that typically contains dense LiDAR coverage. The inner region may be partitioned into 1 -meter voxels and extended hierarchically by factors of 2x, 4x, 8x, and 16x to create outer regions, with voxel sizes scaled proportionally. To optimize the scene initialization, the aggregated static LiDAR point cloud data may beleveraged through a two-step process. First, voxels may be pruned in the inner region that contain no LiDAR points, then voxels containing LiDAR points may be subdivided, thereby creating a multi-scale scene representation. For parameter initialization, voxels containing at least one LiDAR point may be assigned high opacity values by setting parameter a in Equation (6) to 2.0, while voxels without LiDAR points may be initialized with a = 0.1. Parameter b may be initialized to 0.2 across the voxels. The learnable parameters (SDF Field, RGB Field, Spherical Harmonics) may be randomly initialized using PyTorch’s default settings in nn. Linear. The adaptive splitting and pruning process maintains a budget of M total voxels, with N voxels at the current step. N_prune voxels whose opacity remains below 0.005 may be pruned. Accumulated training gradients may be used as a measure of local geometric complexity to identify the top (M+N_prune-N) / (8><5) voxels for splitting. The factor of 8 accounts for subdivision into sub-voxels, while the factor of 5 ensures gradual splitting to prevent premature budget allocation. Upon splitting, the eight sub-voxels inherit identical color and geometry fields from their parent voxel.

[0117] For panoramic visualization, the model is trained using six cameras and then render 360° views through ray-casting. Rolling shutter and motion blur effects occur when the sensor is moving with respect to parts of the scene. Rolling shutter and motion blur effects may be simulated by computing individual per-ray origin points based on the sensor’s velocity and the exact timestamp when each ray of the sensor is registered. The poses of dynamic objects may be interpolated according to the individual ray timestamps. Cast individual camera rays also may allow for injecting new objects into the scene with reflections, refraction, and shadows. For the glass sphere demo, additional refracted and reflected rays are casted, then contributions blended according to the Fresnel formula. A shadow of a synthetically injected object may be simulated.

[0118] The following is a description of how embodiments may be used for selfdriving sensor simulation, which involves large scenes and dynamic actors, while demanding real-time rendering and multi-sensor support. One or more embodiments model dynamic actors and backgrounds as distinct bounding volumes that one or more embodiments compose into a global frame for rendering. For dynamic actors, one or more embodiments initialize voxel sets by partitioning each actor’s canonical bounding box. For rasterization, one or more embodiments use actor labels to transform the dynamic voxels to the global coordinate system at each timestamp, combine the transformed voxels with the static voxels, and project the voxels to the image plane. For ray casting, one or more embodiments construct a separate octree for each dynamic actor. One or more embodiments precompute ray-box intersections between rays and the bounding boxes of dynamic actors, sorting the entry and exit points by camera distance. These precomputed intersections allow us to determine the traversal order between the static scene’s octree and the dynamic actors’ octrees, enabling efficient ray marching.

[0119] The outdoor driving environment uses efficient representation of large- scale scenes (e.g., sky, far-away buildings). The initialization strategy begins by identifying a core region of interest that the self-driving vehicle will traverse, which one or more embodiments discretize at a base resolution. Recognizing that distant scene elements do not use high-resolution voxels, one or more embodiments surround the core static foreground region with increasingly coarser outer regions. These outer regions extend at 2x, 4x, 8x, and 16x the base volume, with their voxel sizes scaling proportionally, naturally matching the diminishing detail requirements of distant scene elements. One or more embodiments leverage LiDAR point clouds to optimize scene initialization through a three-step process: inner region voxel pruning based on point absence, subdivision of point-containing voxels, and high-opacity initialization for point-occupied voxels.

[0120] The scene representation may be optimized through the following loss function:-^color + ^1 -^depth ^reg (7) where Lcolorand £depthmeasure thedistance between rendered and ground-truth images and LiDAR depth, respectively. £regare regularization terms including enforcing spatial consistency between adjacent voxels by maintaining smooth SDF and color transitions and reduced opacity in outer regions to facilitate subsequent pruning. Please refer to the supp. for more details.

[0121] The loss formulation includes of an L color loss (weight: 1.0) and an Li depth loss (weight: 10.0), along with regularization terms. Our regularization framework consists of multiple weighted loss terms, all utilizing the Li norm. The Eikonal loss (weight: 0.1) enforces the SDF field property by constraining the norm of the first three W_s components to unity (for f=ax+by+cz+d, ensuring a2+b2+c2=l). The smoothness loss (weight: 3.0) minimizes the difference between color and geometry fields of neighboring voxel pairs along their shared plane. For opacity regularization (weight: 10.0), opacity values close to 1 for LiDAR points may be enforced by computing opacity at 20cm traversal distance using queried SDF values. The empty space loss (weight: 0.1) encourages the lowest 20% of voxel opacity values in the outer region to approach 0 at maximum traversal distance.

[0122] High-fidelity sensor simulation of light-based sensors such as cameras and LiDARs is important for safe and accurate autonomy testing. One or more embodiments represents volumes as a sparse set of 3D voxel primitives, where each voxel is a local implicit field. One or more embodiments may have fast training (<30 min) and rendering capabilities (50+ FPS for camera and 600+ FPS for LiDAR), may include adaptive pruning and densification to easily handle large scenes, and can support non-pinhole cameras and spinning LiDARs. FIG. 10 shows an example of a variety of camera images (1000) that are generated using the uniform representation of the object voxel grids.

[0123] One or more embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure.

[0124] For example, as shown in FIG. 11 A, the computing system (1100) may include one or more computer processor(s) (1102), non-persistent storage device(s) (1104), persistent storage device(s) (1106), a communication interface (1108) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (1102) may be an integrated circuit for processing instructions. The computer processor(s) (1102) may be one or more cores, or micro-cores, of a processor. The computer processor(s) (1102) includes one or more processors. The computer processor(s) (1102) may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0125] The input device(s) (1110) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device(s) (1110) may receive inputs from a user that are responsive to data and messages presented by the output device(s) (1112). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (1100) in accordance with one or more embodiments. The communication interface (1108) may include an integrated circuit for connecting the computing system (1100) to a network (not shown) (e.g., a local area network (FAN), a wide area network (WAN) such as theInternet, mobile network, or any other type of network) or to another device, such as another computing device, and combinations thereof.

[0126] Further, the output device(s) (1112) may include a display device, a printer, external storage, or any other output device. One or more of the output device(s) (1112) may be the same or different from the input device(s) (1110). The input device(s) (1110) and output device(s) (1112) may be locally or remotely connected to the computer processor(s) (1102). Many different types of computing systems exist, and the aforementioned input device(s) (1110) and output device(s) (1112) may take other forms. The output device(s) (1112) may display data and messages that are transmitted and received by the computing system (1100). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0127] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a solid state drive (SSD), compact disk (CD), digital video disk (DVD), storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by the computer processor(s) (1102), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0128] The computing system (1100) in FIG. 11A may be connected to, or be a part of, a network. For example, as shown in FIG. 11B, the network (1120) may include multiple nodes (e.g., node X (1122) and node Y (1124), as well as extant intervening nodes between node X (1122) and node Y (1124)). Each node may correspond to a computing system, such as the computing system shown in FIG. 11 A, or a group of nodes combined may correspond to the computing system shown in FIG. 11 A. By way of an example, embodimentsmay be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (1100) may be located at a remote location and connected to the other elements over a network.

[0129] The nodes (e.g., node X (1122) and node Y (1124)) in the network (1120) may be configured to provide services for a client device (1126). The services may include receiving requests and transmitting responses to the client device (1126). For example, the nodes may be part of a cloud computing system. The client device (1126) may be a computing system, such as the computing system shown in FIG. 11 A. Further, the client device (1126) may include or perform all or a portion of one or more embodiments.

[0130] The computing system of FIG. 11 A may include functionality to present data (including raw data, processed data, and combinations thereof) such as results of comparisons and other processing. For example, presenting data may be accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown, as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0131] As used herein, the term “connected to” contemplates multiple meanings.A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be atemporary, permanent, or a semi-permanent communication channel between two entities.

[0132] The various descriptions of the figures may be combined and may include, or be included within, the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, or altered as shown in the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0133] In the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before,” “after,” “single,” and other such terminology. Rather, ordinal numbers distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0134] Further, unless expressly stated otherwise, the conjunction “or” is an inclusive “or” and, as such, automatically includes the conjunction “and,” unless expressly stated otherwise. Further, items joined by the conjunction “or” may include any combination of the items with any number of each item, unless expressly stated otherwise.

[0135] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising:obtaining a scene comprising a plurality of object voxel grids for a plurality of objects comprising an object voxel grid, the object voxel grid comprising a voxel positioning property, a geometry field property, and a color field property of a voxel in the object voxel grid;identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene;performing, using the plurality of object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot, wherein the rasterization is performed according to the voxel positioning property and the color field property of the voxel, wherein the ray casting is performed according to the voxel positioning property and the geometry field property; andoutputting the first virtual sensor snapshot and the second virtual sensor snapshot.

2. The computer-implemented method of claim 1, wherein the first virtual sensor and the second virtual sensor comprise a virtual camera and a virtual LiDAR sensor.

3. The computer-implemented method of claim 1, wherein the ray casting generates a virtual LiDAR point cloud and wherein the rasterization generates a virtual camera image.

4. The computer-implemented method of claim 1, further comprising:training the plurality of object voxel grids using a loss calculated from the first virtual sensor snapshot and the second virtual sensor snapshot.

5. The computer-implemented method of claim 1, wherein the object voxel grid further comprises a spherical harmonic field property, and wherein performing rasterization further uses the spherical harmonic field property of the voxel.

6. The computer-implemented method of claim 1, wherein performing rasterization of the scene comprises:projecting a plurality of voxels comprising the voxel onto the image plane to obtain a projection; andcompositing the plurality of voxels in the projection.

7. The computer-implemented method of claim 1, wherein the ray casting comprises:sampling a plurality of locations along a ray, wherein the sampling comprises, for a voxel, selecting a midpoint of a segment of the ray that intersects the voxel; andcalculating is performed by calculating a density value and a color value at the midpoint using the geometry field property and the color field property.

8. The computer-implemented method of claim 1, further comprising:traversing a plurality of voxels in the object voxel grid according to a hierarchical containment relationship between the plurality of voxels,wherein traversing comprises detecting a parent voxel of the plurality of voxels and traversing to a child voxel of the parent voxel in the plurality of voxels responsive to detecting an intersection of a ray in ray casting with the child voxel.

9. The computer-implemented method of claim 1, wherein performing ray casting comprises matrix multiplication between a location identifier of a location within the voxel and the geometry field property and the color field property to generate a color value for a pixel of the second virtual sensor snapshot.

10. The computer-implemented method of claim 1, wherein the scene further comprises a background voxel grid for a plurality of background objects and wherein the rasterization is further performed using the background voxel grid.

11. The computer-implemented method of claim 1, further comprising:setting a size property value, a position property value, and an orientation property value for a plurality of voxels comprising the voxel based on partitioning a bounding box around an object;obtaining real sensor data for the object;initializing a plurality of voxel properties of a current voxel object grid; composing a training scene to match the real sensor data;performing ray casting and rasterization of the training scene with the current object voxel grid according to sensor attributes of sensors generating the real sensor data to generate predictive sensor data; andupdating the plurality of voxel properties based on a comparison between the real sensor data and the predictive sensor data.

12. The computer-implemented method of claim 11, wherein initializing the plurality of voxel properties comprises:initializing an opacity value for the plurality of voxels according to the real sensor data; andrandomly initializing the geometry field property, the color field property, and a spherical harmonic field property.

13. The computer-implemented method of claim 11, further comprising:pruning a subset of the plurality of voxels having an opacity value failing to satisfy an opacity threshold.

14. The computer-implemented method of claim 11, further comprising:partitioning a voxel in the object voxel grid based on a gradient of the color field property of the voxel satisfying a gradient threshold.

15. A system comprising:memory; anda computer processor comprising computer readable program code for performing operations comprising:obtaining a scene comprising a plurality of object voxel grids for a plurality of objects comprising an object voxel grid, the object voxel grid comprising a voxel positioning property, a geometry field property, and a color field property of a voxel in the object voxel grid; identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene; performing, using the plurality of object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot, wherein the rasterization is performed according to the voxel positioning property and the color field property of the voxel, wherein the ray casting is performed according to the voxel positioning property and the geometry field property; andoutputting the first virtual sensor snapshot and the second virtual sensor snapshot.

16. The system of claim 15, wherein the operations further comprise:training the plurality of object voxel grids using a loss calculated from the first virtual sensor snapshot and the second virtual sensor snapshot.

17. The system of claim 15, wherein the object voxel grid further comprises a spherical harmonic field property, and wherein performing rasterization further uses the spherical harmonic field property of the voxel.

18. The system of claim 15, wherein performing rasterization of the scene comprises:projecting a plurality of voxels comprising the voxel onto the image plane to obtain a projection; andcompositing the plurality of voxels in the projection.

19. The system of claim 15, wherein the ray casting comprises:sampling a plurality of locations along a ray, wherein the sampling comprises, for a voxel, selecting a midpoint of a segment of the ray that intersects the voxel; andcalculating is performed by calculating a density value and a color value at the midpoint using the geometry field property and the color field property.

20. A non-transitory computer readable medium comprising computer readable program code for performing operations comprising:obtaining a scene comprising a plurality of object voxel grids for a plurality of objects comprising an object voxel grid, the object voxel grid comprising a voxel positioning property, a geometry field property, and a color field property of a voxel in the object voxel grid;identifying an image plane of a first virtual sensor and a set of virtual sensor properties of a second virtual sensor capturing the scene;performing, using the plurality of object voxel grids, rasterization of the scene on the image plane and ray casting through the scene using the set of virtual sensor properties to generate a first virtual sensor snapshot and a second virtual sensor snapshot, wherein the rasterization is performed according to the voxel positioning property and the color field property of the voxel, wherein the ray casting is performed according to the voxel positioning property and the geometry field property; andoutputting the first virtual sensor snapshot and the second virtual sensor snapshot.

Images