Just in space / just in time 3-dimensional gaussian splatting
By segmenting background and foreground features in 3D Gaussian splatting models, the method optimizes computational resources and time, allowing for efficient real-time 3D reconstruction and novel view synthesis by updating only dynamic foreground elements.
Patent Information
- Application Number
- PCT/US2026/024918
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-07-02
- Filing Date
- 2026-04-23
- Publication Date
- 2026-11-19
AI Technical Summary
Existing Gaussian splatting and novel view synthesis techniques require significant computational resources and time for rendering 3D models, especially when dynamic foreground elements need to be integrated with static background elements, which limits their efficiency in real-time applications.
Utilizing preexisting 3D Gaussian splatting (3DGS) models by segmenting background and foreground features, allowing for updating only foreground elements while leveraging the preexisting background content, thereby reducing the need for complete model re-rendering.
This approach significantly reduces computational resources and time required for rendering 3D models by minimizing the need to re-render static background elements, enabling real-time or near real-time 3D reconstruction and novel view synthesis.
Smart Images

Figure US2026024918_19112026_PF_FP_ABST
Abstract
Description
Qualcomm Ref. No.: 2503356WO1 / 51JUST IN SPACE / JUST IN TIME 3-DIMENSIONAL GAUSSIAN SPLATTING CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 837,799 filed on July 2, 2025, U.S. Provisional Patent Application No. 63 / 804,910 filed on May 13, 2025, U.S. Provisional Patent Application No. 63 / 831,815 filed on June 27, 2025, U.S. Provisional Patent Application No.63 / 810,880 filed on May 23, 2025, U.S. Provisional Patent Application No. 63 / 825,138 filed on June 17, 2025, and U.S. Provisional Patent Application No. 63 / 820,238 filed on June 9, 2025, which are hereby expressly incorporated by reference herein in their entireties.INTRODUCTIONField of the Disclosure
[0002] Aspects of the present disclosure relate to techniques for updating preexisting 3-dimensional Gaussian splatting (3DGS) models.Description of Related Art
[0003] Computer graphic technology is the field of creating, displaying, and storing images using computers and specialized software. Computer graphic technology includes multiple different techniques for creating 3D representations from 2D images and optionally other data sources, such as LiDAR point clouds. Some of these techniques include Photogrammetry, NeRFs (Neural Radiance Fields), and Gaussian splatting. Photogrammetry uses multiple photographs to reconstruct a 3D model by triangulating common points across images. NeRFs employ neural networks to model the entire scene as a continuous volumetric function, allowing for any viewing angle generation. Gaussian splatting represents a scene as a collection of 3D Gaussian blobs (e.g., spherical / elliptical objects with color, transparency, and size attributes). Gaussian splatting is faster to process and more efficient than NeRFs and Photogrammetry.
[0004] Gaussian splatting is a volume rendering technique that renders a 3-dimensional (3D) reconstruction of a scene or object using Gaussian elements to generate 2-dimensional (2D) rendered images. From the 3D reconstruction of the scene it is possible to render novel viewpoints that are not captured with a method referred to asD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO2 / 51novel view synthesis (NVS). In other words, 3D reconstruction enables NVS. NVS generates new 2D views of a scene from a previously generated 3D model. This allows the creation of realistic images of a scene from any desired viewpoint, even if the original images were not taken from that angle.
[0005] As Gaussian splatting and NVS techniques continue to become more popular in the field of computer graphic technology, the variety of applications and variations of each technique continue to expand. Consequently, there exists a need for further improvements in Gaussian splatting and NVS techniques to enable their implementation in solving computer graphic related problems.SUMMARY
[0006] Certain aspects provide a method that includes obtaining abase 3-dimensional Gaussian splatting (3DGS) model of an environment corresponding to a first time, wherein the base 3DGS model comprises Gaussians classified as either background or foreground; obtaining foreground features of the environment; updating base model Gaussians corresponding to the base 3DGS model with one or more Gaussians for each of the foreground features corresponding to a second time after the first time; and outputting an updated 3DGS model comprising Gaussians corresponding to background features from the base 3DGS model and the one or more Gaussians for each of the foreground features.
[0007] Other aspects provide: one or more apparatuses operable, configured, or otherwise adapted to perform any portion of any method described herein (e.g., such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more non-transitory, computer-readable media comprising instructions that, when executed by one or more processors of one or more apparatuses, cause the one or more apparatuses to perform any portion of any method described herein (e.g., such that instructions may be included in only one computer-readable medium or in a distributed fashion across multiple computer-readable media, such that instructions may be executed by only one processor or by multiple processors in a distributed fashion, such that each apparatus of the one or more apparatuses may include one processor or multiple processors, and / or such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more computer program products embodied on one or more computer-readable storage media comprising code for performing anyD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO3 / 51portion of any method described herein (e.g., such that code may be stored in only one computer-readable medium or across computer-readable media in a distributed fashion); and / or one or more apparatuses comprising one or more means for performing any portion of any method described herein (e.g., such that performance would be by only one apparatus or by multiple apparatuses in a distributed fashion). By way of example, an apparatus may comprise a processing system, a device with a processing system, or processing systems cooperating over one or more networks. An apparatus may comprise one or more memories; and one or more processors configured to cause the apparatus to perform any portion of any method described herein. In some examples, one or more of the processors may be preconfigured to perform various functions or operations described herein without requiring configuration by software.
[0008] The following description and the appended figures set forth certain features for purposes of illustration.BRIEF DESCRIPTION OF DRAWINGS
[0009] The appended figures depict certain features of the various aspects described herein and are not to be considered limiting of the scope of this disclosure.
[0010] FIG. 1 depicts an illustrative environment of a road scene.
[0011] FIG. 2 depicts an illustrative sensor and computing system equipped apparatus.
[0012] FIG. 3A is an illustration of 3D Gaussian primitives that are rendered into a two-dimensional (2D) plane based on camera orientation.
[0013] FIG. 3B is an illustration of alpha-blending the 3D Gaussians primitives to render a tile in connection with a volume grouping function.
[0014] FIG. 3C is an illustration of alpha-blending the 3D Gaussians to render a tile in connection with a volume grouping function.
[0015] FIG. 4A illustrates a training system for a 3D Gaussian splatting model.
[0016] FIG. 4B is a diagram illustrating an example of a process for 3D Gaussian splatting.
[0017] FIG. 4C is a diagram illustrating a projector and differentiable tile rasterizer.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO4 / 51
[0018] FIG. 5A is an illustration of an environment from the viewpoint of a driver of a vehicle at a first time.
[0019] FIG. 5B is an illustration of an environment from the viewpoint of a driver of a vehicle at a second time that is later than the first time.
[0020] FIG. 6A is a block diagram of techniques for utilizing a base 3DGS model for a new Gaussian instance of the same environment.
[0021] FIG. 6B is another block diagram of techniques for utilizing a base 3DGS model for a new Gaussian instance of the same environment.
[0022] FIG. 7 depicts an illustrative data structure for a 3DGS model.
[0023] FIG. 8 depicts aspects of an example method for utilizing abase 3DGS model for a new Gaussian instance of the same environment.
[0024] FIG. 9 depicts aspects of an example system according to techniques of the disclosure.DETAILED DESCRIPTION
[0025] Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums related to techniques for updating preexisting 3-dimensional Gaussian splatting (3DGS) models (e.g., referred to herein as base 3DGS models). More specifically, the process of rendering a 3DGS model can be expedited when representations of a background of an environment do not need to be rendered each time a 3DGS model is generated for an environment. Since backgrounds, such as structures, fixtures, and the like may change less frequently than dynamic content such as vehicles, humans, animals, moveable furniture such as chairs and tables and the like, techniques of the present disclosure leverage preexisting 3DGS models as opposed to rendering the complete environment (e.g., background and foreground) from scratch. While rendering a 3DGS model from scratch is faster than other methods, the time and computing resources required to render new 3DGS models can still be optimized by leveraging previously built models, identifying content that can or has changed (e.g., foreground content) and implementing the present foreground content with existing background content from the preexisting 3DGS model.
[0026] For example, in one instance, a person may always drive from home to office and office to home every day. A 3DGS model can be built on day one and stored inD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO5 / 51memory (e.g., in local memory or a cloud-based storage device) for later retrieval and use. On day two, the person can use location information to retrieve day one’s model, then update it and render it with foreground content from day two, while minimizing or eliminating the need to render background content. In another instance, two people may be driving in the same route, but spaced apart in time. The lead driver’s vehicle may generate a base 3DGS model and share the base 3DGS model with one or more other drivers who can leverage the base 3DGS model as it corresponds to their location and optionally time to reduce the computing resources and time required to generate fully 3DGS models of the same location. In a further example, an establishment (e.g., a coffee shop, retail store, museum, event space, or the like) can build a base 3DGS model for its establishment. Customers can access, download, and / or store the base 3DGS model and utilize it to update from time to time with their own data generated for the same establishment. These are only a few example implementation scenarios.
[0027] Aspects described herein are directed to how a base 3DGS model can be utilized and updated to include information (e.g., foreground information) for a time later than when the base 3DGS model was rendered. In general, a base 3DGS model associated with a location of a device configured to generate a current 3DGS model can be obtained, for example, from a storage location. In some aspects, features represented in the base 3DGS model can be annotated by a previously executed process that indicated whether the feature is part of the background or foreground. In some instances, background features may be static (e.g., not currently moving) and / or immovable features (e.g., features that a human cannot readily move like a table or chair). Foreground features may be features that are dynamic (e.g., in motion objects) and / or movable features, such as a table, a chair, parked vehicle, or the like. In some aspects, the base 3DGS model can be processed using an object detection tool, semantic segmentation tool, or image flow tool to determine features (e.g., objects) within the environment. Features that are identified as background and foreground may be labeled as such and stored in a database. The background and foreground segmented base 3GDS model is associated with at least a location and optionally a time or other context data so that the database can be queried by other devices or users to obtain a location relevant base 3DGS model for use.
[0028] As a subsequent user experiences an environment, for example, traversing an environment in a vehicle, the database storing base 3DGS models can be queried for a base 3DGS model that corresponds to the location of the environment being experiencedD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO6 / 51by the subsequent user. The query may also include parameters such as time, field of view, class, or other parameters to obtain the base 3DGS model that best corresponds to the subsequent user’s current situation and environment. It should be noted that a base 3DGS model may be a model associated with the environment that has been updated one or more times in the past. In this way, the number of updates needed to a base 3DGS model can be minimized if prior users of the model have incorporated intermediate updates to the model over time since the model was originally rendered.
[0029] In certain aspects, updates to the base 3DGS model can be readily implemented due to the structured storage of the base 3DGS model, which will be described in more detail herein. For example, a hierarchal structure based on location, time, field of view, foreground and background segmentation, and / the corresponding Gaussian representations (e.g., the plurality of Gaussians and their associated Gaussian parameters) of each may be labeled in a structured format.
[0030] A subsequent user’s device may be configured to obtain one or more images or video of the environment through sensors coupled with the subsequent user’s device. The one or more images or video (e.g.. that may be processed by frame) may be processed with an object detection tool, semantic segmentation tool, and / or image flow tool to determine features (e.g., objects) within the environment that are classified as foreground features (e.g., features in motion and / or that are moveable). In some aspects, the imagebased features that are classified as foreground features may be processed using a Gaussian splatting process to render a 3DGS model of the feature.
[0031] Once a user’s device obtains a base 3DGS model that at least corresponds to their present environment, the foreground features present in the base 3DGS model may be compared to the foreground features corresponding to the current environment being experienced by the subsequent user’s device to determine differences therebetween. That is, the foreground of the base 3DGS model can be updated to include new features obtained from the subsequent user’s device. Additionally, in some aspects, features that are not present in the foreground features from the subsequent user’s device, but which are present in the foreground of the base 3DGS model may be removed from the base 3DGS model. Once the comparison of the foreground features from the base 3DGS model and the foreground features from the subsequent user’s device is completed (e.g., features are added or removed), an updated 3DGS model may be rendered by the subsequent user’s device. This process eliminates the need for the subsequent user’sD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO7 / 51device to carry out Gaussian splatting and rendering processes for every feature (e.g., background and foreground features) of an environment obtained from one or more images of the sensors of the subsequent user’s device. The updated 3DGS model rendering process may follow the Gaussian splatting pipeline. The aforementioned process may alternatively be performed in the image space instead of the Gaussian space. For example, when performed in the image space, the base 3DGS model is used to create one or more images having a viewpoint that corresponds to the one or more images obtained by the subsequent user’s device so that the background and foreground features may be compared in the image-space.
[0032] Aspects of the present disclosure are made possible through the implementation of Gaussian splatting and novel view synthesis (NVS). Gaussian splatting enables the real-time or near real-time rendering of a 3D reconstruction of the environment, such as of one or more objects, using Gaussian elements. From the 3D reconstruction of the environment, it is possible to render novel viewpoints that are not captured with a method referred to as NVS. In other words, 3D reconstruction enables NVS. NVS generates new 2D views of a scene from the previously generated 3D model. Gaussian splatting and NVS enable the creation of realistic images of a scene from any desired viewpoint, even if the original images were not taken from that angle.
[0033] FIG. 1 depicts an illustrative environment 100 of a road scene. The road scene includes a first vehicle 102 equipped with a plurality of sensors, optionally having different viewing angles of the environment and, optionally different modalities. The plurality of sensors may each have a corresponding field of view illustratively depicted by the dashed lined fields. The plurality of sensors may generate images, point clouds, or other data representations of other vehicles (e.g., a second vehicle 103) and / or other objects in the environment such as structures 112, 114, signs or poles 116, 118, and / or other features not specifically depicted in FIG. 1. Features in the environment may be physical objects or components of objects such as corners, edges, or other structures or shapes that may be invariant to rotation, translation, and illumination.
[0034] As depicted in FIG. 1, the first vehicle 102 is travelling in the right-hand lane of traffic. The second vehicle 103 may be travelling in a similar or opposite direction in an adjacent lane. The situation depicted in the illustrated environment 100 will be referenced throughout as aspects of the present disclosure are described. The present instance that is depicted may be referred to as the present time or time tO.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO8 / 51
[0035] Aspects of the present disclosure provide techniques for utilizing base 3DGS models to reduce the need to completely render a 3DGS model for an environment each time a different device captures images or video of the environment for the purposes of rendering a 3DGS model of the same environment, irrespective of the viewpoint. The process of rendering a 3DGS model can be expedited when representations of background features of the environment do not need to be rendered each time a 3DGS model is generated for an environment. For example, an autonomous vehicle may use 3DGS models and NVS models or images generated thereby for navigation, autonomous maneuvers, collision avoidance operations, and / or the like. In other aspects, a robotic system may utilize 3DGS and NVS models for controlling the manipulation of a robotic arm, traversal or navigation of an environment, and / or the like. These tasks may be expedited if the rendering of the 3DGS models that the processes associated with the task can be expedited.
[0036] It should be understood that while the illustrative example discussed herein include a vehicle 102 traversing a road environment, the techniques described herein may be implemented on various apparatuses such as a robot operating in a facility or other environment. For example, the vehicle 102 may be a terrestrial vehicle such as an automobile, a truck, a motorcycle, a bicycle, a robotic device, or the like; an amphibious vehicle, such as a boat, a sailboat, a ship, a submarine, an unmanned amphibious vehicle, or the like; or an aerial vehicle such as a helicopter, a quadcopter, an unmanned aerial vehicle or the like.
[0037] Given the road scene depicted in FIG. 1, specific aspects will be described herein.Electronic Device and System
[0038] FIG. 2 depicts an illustrative sensor and computing system equipped apparatus, such as a manually operated vehicle, a semi-autonomous vehicle, or an autonomous vehicle (AV) corresponding to aspects described herein. The apparatus 202 may be an example of the first vehicle 102 and / or the second vehicle 103 depicted in FIG.1 or other apparatus such as a robot. The apparatus 202 depicted in FIG.2 is depicted by way of an example schematic of a vehicle including sensor resources and a computing device. Not every vehicle is required to be equipped with the same set of sensor resources, nor is every vehicle required to be configured with the same set of systems for perceivingD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO9 / 51attributes of an environment. FIG. 2 only provides one example configuration of sensor resources and systems equipped within a vehicle.
[0039] FIG. 2 provides an example schematic of apparatus 202 including a variety of sensor resources, which may be utilized by the apparatus 202 to perceive and collect sensor data about the environment. For example, the apparatus 202 may include a computing device 240 comprising one or more processors 242 and a non-transitory computer readable memory 244 (also referred to herein as one or more memories), one or more cameras 252, a global positioning system (GPS) 254, a radar system 256, an IMU 258, a LiDAR system 260, one or more microphones 262, one or more speakers 264, a display 266, and network interface hardware 270. The aforementioned components of the apparatus are merely examples as some apparatuses may have more or less components and / or different sensors for perceiving the environment. These and other components of the apparatus 202 may be communicatively connected to each other via a communication path 230. It should be noted that non-transitory computer readable memory 244 may include volatile and / or non-volatile memory or storage.
[0040] The communication path 230 may be formed from any medium that is capable of transmitting a signal such as, for example, conductive wires, conductive traces, optical waveguides, or the like. The communication path 230 may also refer to the expanse in which electromagnetic radiation and their corresponding electromagnetic waves traverse. Moreover, the communication path 230 may be formed from a combination of mediums capable of transmitting signals. In some aspects, the communication path 230 comprises a combination of conductive traces, conductive wires, connectors, and buses that cooperate to permit the transmission of electrical data signals to components such as processors, memories, sensors, input devices, output devices, and communication devices. Accordingly, the communication path 230 may comprise a bus. Additionally, it is noted that the term “signal” means a waveform (e.g., electrical, optical, magnetic, mechanical or electromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave, square-wave, vibration, and the like, capable of traveling through a medium. As used herein, the term “communicatively coupled” means that coupled components are capable of exchanging signals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and the like.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO10 / 51
[0041] The computing device 240 may be any device or combination of components comprising one or more processors 242 and non-transitory computer readable memory, referred to herein as one or more memories 244. The one or more processors 242 may be any device capable of executing the processor-executable instructions stored in the one or more memories 244. Accordingly, the one or more processors 242 may be an electric controller, an integrated circuit, a microchip, a computer, or any other computing device. The one or more processors 242 are communicatively coupled to the other components of the apparatus 202 by the communication path 230. Accordingly, the communication path 230 may communicatively couple any number of processors 242 with one another, and allow the components coupled to the communication path 230 to operate in a distributed computing environment. Specifically, each of the components may operate as a node that may send and / or receive data.
[0042] The one or more memories 244 may comprise RAM, ROM, flash memories, hard drives, or any non-transitory memory device capable of storing processor-executable instructions such that the processor-executable instructions can be accessed and executed by the one or more processors 242. The machine-readable instruction set may comprise logic or algorithm(s) written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, for example, machine language that may be directly executed by the one or more processors 242, or assembly language, object-oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into processor-executable instructions and stored in the one or more memories 244. Alternatively, the processor-executable instructions may be written in a hardware description language (HDL), such as logic implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), or their equivalents. Accordingly, the functionality described herein may be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components.
[0043] The apparatus 202 may further include one or more cameras 252. The one or more cameras 252 may be any device having an array of sensing devices (e.g., a CCD array or active pixel sensors) capable of detecting radiation in an ultraviolet wavelength band, a visible light wavelength band, or an infrared wavelength band. The one or more cameras 252 may have any resolution. The one or more cameras 252 may be an omnidirection camera or a panoramic camera. In some aspects, one or more opticalD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO11 / 51components, such as a mirror, fish-eye lens, or any other type of lens may be optically coupled to the one or more cameras 252. The image data collected by the one or more cameras 252 may be stored in the one or more memories 244.
[0044] Still referring to FIG. 2, a global positioning system, GPS 254, may be coupled to the communication path 230 and communicatively coupled to the computing device 240 of the apparatus 202. The GPS 254 is capable of generating location information indicative of a location of the apparatus 202 by receiving one or more GPS signals from one or more GPS satellites. The GPS signal communicated to the computing device 240 via the communication path 230 may include location information comprising a National Marine Electronics Association (NMEA) message, a latitude and longitude data set, a street address, a name of a known location based on a location database, or the like. Additionally, the GPS 254 may be interchangeable with any other system capable of generating an output indicative of a location. For example, a local positioning system that provides a location based on cellular signals and broadcast towers or a wireless signal detection device capable of triangulating a location by way of wireless signals received from one or more wireless signal antennas. The sensor data collected by the GPS 254 may be stored in the one or more memories 244.
[0045] The apparatus 202 may also include a radar system 256. The radar system 256 measures the distance to objects over wide distances. It is also possible to measure the relative speed of the detected object. The radar system 256 may be a continuous wave (CW), frequency-modulated continuous wave (FMCW), 3D-radar (3D FMCW multipleinput and multiple-output (MIMO)), or 4D-radar (4D FMCW MIMO). The sensor data collected by the radar system 256 may be stored in the one or more memories 244.
[0046] The apparatus 202 may include an inertial measurement unit (IMU) 258. The IMU 258 is an electronic device that measures and reports an apparatus’s specific force, angular rate, and sometimes the orientation of the apparatus, using a combination of accelerometers, gyroscopes, and sometimes magnetometers. The sensor data collected by the IMU 258 may be stored in the one or more memories 244.
[0047] In certain aspects, the apparatus 202 may include a LiDAR system 260. The LiDAR system 260 is communicatively coupled to the communication path 230 and the computing device 240. A LiDAR system 260 or light detection and ranging is a system and method of using pulsed laser light to measure distances from the LiDAR system 260D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO12 / 51to objects that reflect the pulsed laser light. A LiDAR system 260 may be made as solid-state devices with few or no moving parts, including those configured as optical phased array devices where its prism-like operation permits a wide field-of-view without the weight and size complexities associated with a traditional rotating LiDAR system 260. The LiDAR system 260 is particularly suited to measuring time-of-flight, which in turn can be correlated to distance measurements with objects that are within a field-of-view of the LiDAR system 260. By calculating the difference in return time of the various wavelengths of the pulsed laser light emitted by the LiDAR system 260, a digital 3-D representation of a target or environment may be generated. The pulsed laser light emitted by the LiDAR system 260 includes emissions operated in or near the infrared range of the electromagnetic spectrum, for example, having emitted radiation of about 905 nanometers. Sensors such as the LiDAR system 260 can be used by vehicles to provide detailed 3D spatial information for the identification of objects near the apparatus 202, as well as the use of such information in the service of systems for vehicular mapping, navigation and autonomous operations, especially when used in conjunction with georeferencing devices such as GPS 254 or a gyroscope-based inertial navigation unit (INU, not shown or IMU 258) or related dead-reckoning system. The point cloud data collected by the LiDAR system 260 may be stored in the one or more memories 244.
[0048] The apparatus 202 may also include one or more microphones 262 that are coupled to the communication path 230 and communicatively coupled to the one or more processors 242. The one or more microphones 262 may be any device capable of transforming a mechanical vibration associated with sound into an electrical signal indicative of the sound. The one or more microphones 262 may be used to pick up sound waves such as ultrasonic frequencies, voices, and other sounds within the environment.
[0049] The apparatus 202 may also have one or more speakers 264. The one or more speakers 264 (an example of an audio output device) are coupled to the communication path 230 and communicatively coupled to the one or more processors 242. The one or more speakers 264 transform electrical signals from the speaker driver circuit and the one or more processors 242 of the computing device 240 into mechanical vibrations producing one or more sound signals. It should be understood that some embodiments may not include the one or more speakers 264.
[0050] The apparatus 202 may also include a display 266. The display 266 may be configured to provide a visual output. The display 266 is coupled to the communicationD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO13 / 51path 230. Accordingly, the communication path 230 communicatively couples the display 266 with other components of the apparatus 202. The display 266 may include any medium capable of transmitting an optical output such as, for example, a cathode ray tube, light emitting diodes, a liquid crystal display, a plasma display, or the like. Additionally, the display 266 may be the display 266 of a portable personal device such as a smart phone, tablet, laptop or other electronic device. Additionally, it is noted that the display 266 can include one or more processors 242 and one or more memories 244.
[0051] Still referring to FIG. 2, apparatuses, such as vehicles are now more commonly equipped with communication systems, such as vehicle-to-vehicle communication systems. Some of the systems rely on network interface hardware 270. The network interface hardware 270 may be coupled to the communication path 230 and communicatively coupled to the computing device 240. The network interface hardware 270 may be any device capable of transmitting and / or receiving data with a network 280 or directly with another vehicle, such as equipped with a vehicle-to-vehicle communication system. Accordingly, network interface hardware 270 can include a communication transceiver for sending and / or receiving any wired or wireless communication. For example, the network interface hardware 270 may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card, mobile communications hardware, near-field communication hardware, satellite communication hardware and / or any wired or wireless hardware for communicating with other networks and / or devices. In some aspects, network interface hardware 270 includes hardware configured to operate in accordance with the Bluetooth wireless communication protocol. In some aspects, network interface hardware 270 may include a Bluetooth send / receive module for sending and receiving Bluetooth communications to / from a network 280 and / or another vehicle (e.g., the second vehicle 103).Introduction to Gaussian Splatting
[0052] The techniques and methods described herein may implement and use various computer graphic technologies including, but not limited to, Gaussian splatting.
[0053] Gaussian splatting, which may also be referred to as 3D Gaussian splatting, generates an explicit representation of a 3D scene. The 3D scene comprises a collection of ellipsoidal splats called Gaussians or Gaussian elements. Rendering the 3D scene with the Gaussian splatting can be performed in real-time or near real-time, unlike NeuralD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO14 / 51Radiance Field (NeRF) and Photogrammetry. Additionally, unlike photogrammetry, Gaussian splatting can provide all of the photorealistic effects because Gaussian Splatting uses spherical harmonics to represent the view-dependent effects and lighting. Spherical harmonics are a set of functions used to represent functions on the surface of the sphere, S2. Spherical harmonics are defined as the eigenfunctions of the angular part of the Laplacian in three dimensions. Spherical harmonics can be used to model color changes over the surface of the Gaussian to represent the effects of different lighting and changes in viewing angle of the features and objects making up the environment. For example, surfaces of the Gaussian elements can change color when they are viewed from different angles.
[0054] In general, the process of Gaussian splatting includes obtaining an image from multiple viewpoints of an environment and, optionally, other sensor data, such as a point cloud. The inclusion of a point cloud is not required because a process such as structure from motion (SfM) or other technique can be utilized to produce a sparse 3D point cloud of the environment. To do so, SfM may be run on the obtained images to reconstruct the camera poses. As a byproduct, SfM also produces a sparse 3D point cloud of the environment. The process further includes initializing Gaussians. Gaussians may be “splat” at each point of the 3D point cloud and centered at each point. “Splat” refers to a translucent ellipsoid, or blob, used to represent a point or points in 3D space. These splats are based on Gaussian functions and carry information like position, size, orientation, color, and transparency. When viewed together, the many splats blend together to create a photorealistic 3D scene. The parameters of the Gaussians may be optimized. The process of optimizing the Gaussians may include using gradients, such as a gradient descent process to optimize Gaussian position, size, orientation, and the like to match information from the images. Gaussians may be added, removed, split into more than one Gaussian, etc. during optimization to increase detail where needed. For example, objects in a scene, such as vehicles, people, animals, signs, buildings, and the like may have more Gaussians and smaller sized Gaussians compared to features like the sky, distant landscape, or the ground. Color associated with the Gaussians is represented with spherical harmonics to model view-dependent appearances. In other words, spherical harmonics model color changes over the surface of the Gaussian to represent the effects of different lighting and changes in viewing angle of the features and objects making up the environment. Gaussian splatting further includes rasterization and final renderingD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO15 / 51where the optimized 3D Gaussians are projected into an image plane and alpha-blending (ot-blending) is used to combine Gaussians to create the appearance of partial or full transparency. Each Gaussian or pixels associated with the Gaussian may be weighted such that the weights are considered when alpha-blending along a ray or rays corresponding to a viewing angle that an image is being generated for from the 3D Gaussian reconstruction of the environment.
[0055] Based on an orientation of the camera (e.g., a viewpoint), the 3D Gaussian elements are splatted onto a 2D plane and presented to a user. 3D Gaussian splatting can be effectively combined with modern 3D scanning technologies and accelerates the capture-to-rendering pipeline for real-world scenes. For example, 3D Gaussian splatting can be used in conjunction with LiDAR sensors, structured light sensors, and depth sensors to generate a visualization of the environment 100 in 2D space from an arbitrary viewpoint. For example, with dense point cloud data or depth maps acquired from a LiDAR, 3D Gaussian splatting can initialize a set of Gaussian elements directly from real-world geometry and may avoid time-consuming training or reconstruction stages common in other techniques. 3D Gaussian splatting also represents both spatial and radiometric information to provide realistic rendering under varying lighting conditions and is well-suited for applications in virtual production, real-time VR and XR experiences, and other real-time applications, such as vehicle navigation and operator alert or warning systems within a vehicle or other apparatus.
[0056] Gaussian splatting will now be discussed in more detail with reference to at least FIGS. 3A-4C.
[0057] FIG. 3A is an illustration of 3D Gaussian primitives within a scene 300 of a 3D coordinate system and rendered into 2D image planes based on camera orientation in accordance with some aspects of the disclosure. FIG. 3B is an illustration of projecting the 3D Gaussian elements in accordance with some aspects of the disclosure. FIG. 3C is an illustration of alpha-blending the 3D Gaussian elements to render a tile in connection with a volume grouping function in accordance with some aspects of the disclosure. FIGS.3A, 3B, and 3C will be discussed in conjunction to describe the projection of the 3D Gaussian elements and blending into a 2D image plane.
[0058] 3D Gaussian splatting is a technique for representing and rendering volumetric scenes using 3D information. 3D Gaussian splatting involves the use of a set of 3DD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO16 / 51Gaussian elements (also referred to as Gaussians or primitives) that are each defined by a position in space, a covariance matrix (e.g., which determines shape and orientation), color, opacity, and in some cases other parameters. In certain aspects, a 3D Gaussian element is a volume that can be represented by sparse information, such as in approximately 300 bits. In the context of 3D Gaussian splatting, a non-limiting example of a primitive is a unit sphere with a covariance matrix. As noted, each Gaussian element can be described by a number of parameters, which can include position (e.g., an X, Y, Z coordinate of the location), covariance (e.g., how much the Gaussian element is stretched / scaled along the three orthogonal directions, namely the eigenvectors of the covariance), color (e.g., red, green, blue and its variability depending on the viewing angle using Spherical Harmonic coefficients), and a level of transparency (referred to as opacity). For example, the covariance matrix represents the transformation, scale, and orientation of the primitive in the 3D coordinate system. The covariance matrix may transform the unit sphere into 3D ellipsoids (e.g., 3D Gaussians) that represent a volume within the 3D coordinate system. 3D Gaussian elements represent fuzzy volumes that smoothly blend into each other, differently than solid geometric approaches of NeRF and conventional 3D rendering techniques. 3D Gaussian splatting can be used to generate or reconstruct a 3D scene by projecting the Gaussian elements onto a 2D space (e.g., rendering the Gaussian elements to a 2D image), using Gaussian functions for each splat.3D Gaussian splatting allows for real-time rendering of photorealistic scenes learned from small samples of images.
[0059] In some aspects, the scene 300 includes a first Gaussian element 302, a second Gaussian element 304, and a third Gaussian element 306 that are represented in a 3D coordinate system. In particular, each of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 include a position (e.g., location within the 3D coordinate system), a covariance matrix, a color, and an opacity. The covariance matrix represents the spread and orientation of a multivariate distribution, and transforms a standard shape (e.g., a unit sphere) into a 3D ellipsoid that captures the scale and correlation of the data along different axes. 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other, which is different from other rendering techniques that represent solid geometry. The term “fuzzy volumes” refers to splats (e.g., 3D Gaussians) which define parameters like position, size, orientation, and color that contributes to the final image from a respective viewpoint. The term “smoothly blend”D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO17 / 51refers to viewing multiple 3D Gaussians together when rendered through alpha-blending, thereby creating a seamless and photorealistic representation of an environment.
[0060] The scene 300 includes a first image plane 310 and a second image plane 320 in an arbitrarily different position (e.g., having a location, viewing direction, and orientation) than the first image plane 310. In this example, the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are rendered onto tiles for purposes of illustration. In some aspects, a 2D image may be represented as tiles, which are groups of unit pixels (e.g., 2x2 pixels, 4x4 pixels, etc.). Tiles can be used to parallelize mathematical computations within a graphical processing unit (GPU) to reduce computational latency. In this case, the tiles in FIGS. 3A, 3B, and 3C are 2x2 pixels for purposes of illustration.
[0061] In some aspects, rendering a 3D Gaussian splatting comprises various stages, such as a projection stage, a visibility stage, a rasterization stage, and an image composition stage, to render the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 onto a 2D image plane (e.g., the first image plane 310 or the second image plane 320). In the projection stage, each 3D Gaussian of the scene 300 is projected onto a 2D image plane based on intrinsic and extrinsic parameters associated with a camera (e.g., position, depth of field, focus, etc.). Projecting a 3D Gaussian onto a 2D image plane may include the process of converting the 3D coordinates of the 3D Gaussian into a 2D representation on the 2D image plane, such as creating an image or drawing of the 3D Gaussian from the perspective of the 2D image plane (e.g., a camera viewing the 2D image plane from the camera viewpoint). For example, each of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are projected as ellipses into the first image plane 310. The first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are projected as differently shaped and oriented ellipses into a second image plane 320 based on a different view of their volumetric representations.
[0062] The projection stage may also include other computation reduction strategies such as adaptive density control to prune features that cannot be rendered into the first image plane 310. The projection stage can include other types of transformations, such as ray tracing. Ray tracing is a technique that simulates the path of light rays from the camera through the scene. It determines how these rays interact with objects (e.g., reflection,D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO18 / 51refraction, shadows) to produce highly realistic images. An example of ray tracing includes, but is not limited to, the technique referred to as 3D Gaussian Ray Tracing.
[0063] In another example stage, a visibility stage performs a visibility check for each Gaussian to ensure that Gaussian elements that are visible from the camera viewpoint (e.g., a camera viewpoint that corresponds to the image plane 310) contribute to the final image. For example, from a given camera viewpoint, a portion of an ellipse (e.g., a first Gaussian) may be occluded from view as it is behind another ellipse (e.g., a second Gaussian. Accordingly, the portion of the ellipse (e.g., the first Gaussian) that is occluded may be excluded from rendering computations as they do not contribute to the rendered view. For example, if a scene includes 107Gaussian elements, and each Gaussian includes a covariance matrix, computation complexity can be reduced significantly if 50% of the Gaussian elements are removed.
[0064] The rasterization stage, which can also be referred to as a render function, is the process of rasterizing the colors the visible 3D Gaussian elements onto the first image plane 310 and the second image plane 320. As used herein, “rasterizing” refers to the process of projecting 3D Gaussian spheres or ellipsoids (splats) onto a 2D image plane, creating a realistic representation of a 3D scene. The render function accumulates color that follows a front-to-back compositing approach with closer 3D Gaussian elements being rendered before distant 3D Gaussian elements to ensure correct depth representation. This is more generally referred to as alpha-blending. The color and opacity of the current Gaussian are weighted by the transparency of the scene that has already been rendered in front of it. This blending process is akin to the “over” operator used in alpha compositing, where the color of the current element is placed “over” the existing color, taking into account the transparency of the elements already rendered. For example, FIG. 3B illustrates the first image plane 310 as a planar 4x4 pixel including tiles 311-314. As shown in FIG. 3B, the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are mapped into the 2D space of the tiles 311-314 based on the orientation of a viewpoint with overlapping volumes. During the render function, the color and opacity values of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are accumulated in a manner that accounts for overlapping regions. For example, the accumulation follows a front-to-back compositing approach with closer Gaussian elements being rendered before distant Gaussian elements to ensure correct depth representation. In the example of the firstD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO19 / 51image plane 310, the third Gaussian element 306 is closest to the viewpoint and may occlude the first Gaussian element 302 and the second Gaussian element 304 (e.g., based on the opacity of the third Gaussian element 306). The opacity of the third Gaussian element 306 may also affect the rendered color splatted onto a corresponding tile. For example, if the third Gaussian element 306 has no opacity, the resulting color of a corresponding tile may be strongly correlated with the color of the third Gaussian element 306 based on the rendering function. In another example, if the third Gaussian element 306 has significant opacity, the resulting color may be strongly correlated with other colors behind the third Gaussian element 306 (e.g., the first Gaussian element 302 and the third Gaussian element 306). The rendering of the complete scene (not shown) can include additional volumetric features such as specular highlights, shadows, and other volumetric effects based on volumetric effects associated with the 3D Gaussian primitives.
[0065] In contrast to conventional solid geometry rendering functions, 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other. For example, when a virtual camera observes the scene, the 3D Gaussian elements are projected into 2D screen space using standard graphics pipeline transformations, which results in 2D ellipses represented on a 2D image plane that can be efficiently composed onto the image plane. 3D Gaussian splatting can also capture fine detail with soft lighting effects to generate smooth transitions and prevent aliasing artifacts. Moreover, transparency and depth blending allow for semi-transparent effects, accurate specular highlighting, and accurate occlusions without sorting or complex data structures.
[0066] The rasterization stage can include other stages to further increase rendering efficiency. In one example, a replication and sorting operation can be performed. In a replication and sorting operation, 3D Gaussian elements can be mapped to tiles and then sorted, which can enable various features to reduce computation complexity. For example, depth can affect rendering operation. An example illustrates Table 1 below illustrates the mapping of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 into the tiles 311-314. That is, Table 1 identifies whether a 3D Gaussian overlaps the tile and its corresponding depth (e.g., relative to a camera).Gaussian Tile DepthD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO20 / 513D Gaussian element 304 Tile 311 Dl3D Gaussian element 304 Tile 312 DI3D Gaussian element 306 Tile 311 D23D Gaussian element 306 Tile 312 D23D Gaussian element 306 Tile 313 D23D Gaussian element 306 Tile 314 D23D Gaussian element 302 Tile 313 D3able 1
[0067] In some aspects, the Gaussian elements are stored based on tile and splatting of each tile can be handled in parallel within a GPU. For example, each 3D Gaussian of a corresponding title is splatted onto the tile as a 2D footprint and colors are alpha blended to approximate visibility and transparency. Table 2 illustrates a resulting sorted result and simplifies the splatting and blending operations by limiting contributing Gaussian elements based on the overlap of the corresponding tile.Gaussian Tile Depth3D Gaussian element 304 Tile 311 Dl3D Gaussian element 306 Tile 311 D23D Gaussian element 304 Tile 312 Dl3D Gaussian element 306 Tile 312 D23D Gaussian element 306 Tile 313 D23D Gaussian element 302 Tile 313 D33D Gaussian element 306 Tile 314 D2able 2
[0068] FIG. 3C illustrates the alpha blending operation associated with tile 311. Because the tile 311 is 2x2 pixels, the value of four pixels (C1-C4) of tile 311 can be calculated in parallel based on the equations below:Ci = ai’ci+ cci’c2(l - 011’)C2 = 012’01+ 012’02(1 - 012’)C3 = 013’01+ 013’02(1 - 013’)D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO21 / 51C4 = 0t4’ci+ 0.4’02(1 - 014’)
[0069] In this case, the area of the ellipse overlapping the pixel and the opacity (e.g., a) generates a high-fidelity representation of the 3D scene that includes shading and other volumetric effects.
[0070] 3D Gaussian splatting is an efficient rendering technique that is advantageous over other view synthesis rendering techniques because a neural network does not perform inference at every point in space along a ray. In some cases, the entire scene can be precomputed or optimized into a sparse set of Gaussian primitives to allow efficient rendering. 3D Gaussian splatting can reduce computational overhead and memory usage to allow modest processors to render complex scenes in real-time.
[0071] In contrast to conventional solid geometry rendering functions, 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other. For example, when a virtual camera observes the scene, the 3D Gaussian elements are projected into 2D screen space using standard graphics pipeline transformations, which results in 2D ellipses represented on a 2D image plane that can be efficiently composed onto the image plane.
[0072] 3D Gaussian elements are splatted into a 2D image space and use alpha blending to approximate visibility and transparency. The 3D Gaussian splatting rendering process is similar to forward rasterization and is highly parallelizable, making it suitable for rendering in GPU. In some cases, GPU can render the 3D Gaussian splatting scene at full frame rates (e.g., 30 Hz) for interactive applications like VR, 3D visualization, and other media consumption.
[0073] FIG. 4 A illustrates a training system 400 for a 3D Gaussian splatting model in accordance with some aspects of the disclosure. The training system 400 is configured to receive the 3D information 402 such as SfM points, apoint cloud, etc. The 3D information 402 represents the positions of key features in the scene, such as depths to each point. In some aspects, the 3D information 402 can be a sparse reconstruction of the scene's geometry that is derived from multiple 2D images.
[0074] The 3D information 402 is provided to a Gaussian model engine 404 to initialize a set of 3D Gaussian elements, or a Gaussian model, forming a 3D scene. The Gaussian model engine 404 converts the 3D information 402 in various forms (e.g., points, vertexes, etc.) into a corresponding 3D Gaussian element that includes a positionD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO22 / 51in 3D space, an orientation and scale (e.g., the covariance matrix), and color and opacity (alpha). The Gaussian model engine 404 may also include other trainable properties, such as density fall-off, or feature embeddings. Density fall-off refers to how the Gaussian falls off from its center point. In other words, as you move away from the center of the Gaussian, its influence on the scene decreases. This creates a smooth transition between overlapping Gaussians and helps avoid harsh edges in the rendered image. Feature embeddings refers to additional information or features, often learned from foundation models, into the 3D Gaussian representation of a scene. This enhances the scene representation beyond just geometric and appearance (color) properties, allowing for more advanced scene understanding and manipulation. The initialized 3D Gaussian elements collectively serve as the initial volumetric scene representation.
[0075] The Gaussian model engine 404 provides the Gaussian model to a projection engine 406 for a camera (e.g., viewpoint) to image plane projection. Each 3D Gaussian is projected into 2D space from a viewpoint of a camera 410 and simulates how a Gaussian would appear on the image plane based on the perspective of the camera 410 and intrinsics of the camera 410 (e.g., focus, depth, exposure, etc.).
[0076] The projection engine 406 provides the projected images to a tile rasterizer 412 (e.g., a render function) to rasterize the 3D Gaussian elements onto a 2D grid (image tiles) based on differentiable rendering. Each Gaussian of a corresponding tile contributes a splat to the final image that is modulated by its opacity, shape, and distance to the camera. The intrinsics of the camera 410 can also affect the rendering process such as focal length that can affect appearance of objects at different depths, shutter speed, etc. The contributions of all Gaussian elements are composited into a predicted image by the tile rasterizer 412.
[0077] The training engine 414 is configured to calculate a loss and then initiate an optimization flow to cause parameters of the Gaussian model to change. The training engine 414 compares the predicted image against ground-truth images from the training set (e.g., from corresponding camera viewpoints) to determine a loss. The training engine 414 includes atraining loop that minimizes a composite loss L = (1~ )L]+ LD-SSIM, where Li loss is apixel-wise absolute difference between the rendered and ground-truth images, LD-SSIM is a perceptual difference in image structure and appearance using a Dissimilarity Structural Similarity Index, and the scalar X that balances the pixel-wise absolute difference and the perceptual difference.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO23 / 51
[0078] Based on the loss function, the training engine 414 initiates an optimization flow to optimize the Gaussian model. The training engine 414 computes gradients of the loss from the predicted image and then backpropagates the gradients through the tile rasterizer 412 to influence the parameters of each Gaussian (e.g., position, scale, orientation, color, and alpha). Each gradient indicates information to reduce the image error.
[0079] The tile rasterizer 412 also backpropagates errors to the projection engine 406 and an adaptive density control engine 416 to influence generation of the Gaussian model. Backpropagation of errors refers a feedback loop or a way to use information from the rasterization stage to refine or improve the projection or subsequent stages. The adaptive density control engine 416 uses the gradients to refine the spatial configuration of Gaussian elements and adjust the position and quantity of the Gaussian elements. For example, the adaptive density control engine 416 may add new Gaussian elements and remove redundant Gaussian elements. The adaptive density control engine 416 determines whether the Gaussian elements are too sparse, redundant, or irrelevant, and can control the Gaussian model engine 404 to prune or merge Gaussian elements that contribute little to the rendered image and duplicate or refine Gaussian elements in regions where more detail is needed. The adaptive density control engine 416 controls the detail of the model and improves rendering quality based on the training loop by adjusting the density and distribution of Gaussian elements based on optimization feedback.
[0080] The resulting Gaussian model is a compressed, expressive, and differentiable 3D scene model that can enable real-time and photo-realistic rendering based on a hybrid of point-based rendering and volumetric rendering. Gaussian models can be used in many different contexts, such as media presentation and other entertainment, where real-time rendering is necessary for the user experience.
[0081] The training system 400 is more efficient than other 3D planar reconstruction techniques and novel view synthesis techniques. The training system 400 uses differentiable rendering techniques that adjust the parameters (e.g., position, shape, opacity) of each 3D Gaussian through gradient descent, which reduces the computation and time for scene reconstruction and convergence. Other techniques perform thousands of iterations to train a neural network over dense samples and can take many hours to converge.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO24 / 51
[0082] FIG. 4B is a diagram illustrating an example of a process 420 for 3D Gaussian splatting. The process 420 may utilize SfM using the COLMAP algorithm to generate a 3D point cloud for 3DGS initialization. During operation of the process 420, one or more image sensors can obtain a plurality of 2D images 422 (or image frames), from a plurality of different views (e.g., different image sensor poses), of a scene. The 2D images 422 are used to train the 3DGS model. Once training of the 3DGS model is completed, the trained 3DGS model can be used to render generated images from other views (the same or different views as those in the 2D images 422). For example, the generated images may include a frontal view image 492, an above view image 494, and an end view image 496.
[0083] SfM 424 (e.g., which can use the COLMAP algorithm) can estimate a 3D point cloud from the plurality of 2D images 422 of the scene. The 3D point cloud can be used for initialization 430 of the Gaussian splats. For example, each point in the 3D point cloud can be converted into a Gaussian splat to produce a plurality of 3D Gaussian splats 440 (e.g., also referred to as a Gaussian scene representation). The conversion of the points into Gaussian splats should be sufficient to provide rasterization for the scene. However, only the position and color parameters for the Gaussian splats may be inferred from the SfM data. Training may be performed, for example, on parameters, projections, and adaptive density control 470 of the Gaussian splats according to the different views (e.g., poses) of the image sensor (e.g., camera parameters 450) when capturing the 2D images 422 to learn a representation that yields high quality results.
[0084] In some aspects, the training procedure for 3DGS can employ a stochastic gradient descent technique (e.g., which is similar to a neural network, but without the layers) based on the 2D images 422. During the training, the Gaussian splats can be rasterized to one or more images 490 (e.g., each corresponding to a different view of the scene depicted in the input 2D images 422) using a differentiable tile rasterizer 480 (e.g., using a differentiable rendering algorithm). A loss can then be determined based on a difference between the rasterized images 490 of the scene and ground truth images (e.g., the 2D images 422) of the scene. The parameters of the Gaussian splats may be adjusted according to the loss. Automated densification (e.g., adaptive density control 470) and pruning of the Gaussian splats can also applied.
[0085] When the 2D images 422 have a sufficient variance in different views, the training can allow for the 3D scene to be generated from various different viewpoints withD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO25 / 51realistic details (e.g., generated images, such as the frontal view image 492, the above view image 494, and the end view image 496). As shown in FIG. 4B, since the frontal view image 492 provides realistic results for the scene, but the above view image 494 and the end view image 496 do not provide realistic results for the scene, it is evident that the 2D images 422 did not contain a sufficient amount of views (e.g., the above view and the side view) of the scene.
[0086] Tables 1 and 2 below illustrate algorithms that can be implemented in connection with various aspects of the training system 400 of FIG. 4A and / or the process 420 of FIG. 4B. / / Optimization and densif ication / / w, h : width and height of the training imagesM SfM PointsS, C, A InitAttributes ( ) / / Covariances , Colors , Opacitiesi 0 / / Iteration Countwhile not converged doV, I SampleTrainingView ( ) / / Camera V and ImageI Rasterize (M, S, C, A, V) / / Alg . 2L Loss ( I , I ) / / LossM, S, C, A <- Adam (VL) / / Backprop & Stepif IsRef inementlteration ( i ) thenfor all Gaussians (p, S , c, a) in (M, S, C, A) doif a < s or IsTooLarge (p, S ) thenRemoveGaussian ( ) / / Pruningend ifif VPL > TPthen / / Densif icationif ||S|| > TSthen / / Over-reconstructionSplitGaussian (p, S , c, a)else / / Under-reconstructionCloneGaussian (p, S , c, a)end ifend ifend forend ifi i + 1end whileTable 1
[0087] Table 1 provides an example of a core training loop for optimizing a 3D Gaussian Splatting scene representation by initializing the 3D Gaussian elements’ properties (positions, covariances, colors, opacities) using SfM point data. In each iteration, a training view is sampled, and a differentiable rendering pass is performed. TheD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO26 / 51loss is computed between the rendered and ground-truth image, and backpropagation is used to update all Gaussian parameters through an optimizer (e.g., Adam). On designated refinement steps, the model performs pruning, removing Gaussian elements with low opacity or excessively large spread, and densification, where Gaussian elements with high loss gradients are either cloned (to increase coverage in sparse areas) or split (to resolve over-dense regions). / / GPU Software Rasterization of 3D Gaussians / / w, h : width and height of the image to rasterize / / M, S : Gaussian means and covariances in world space / / C, A: Gaussian colors and opacities / / V: view configuration of current camerafunction Rasterize (w, h, M, S, C, A, V)CullGaussian (p, V) / / Frustum CullingM ' , S ' ScreenspaceGaussians (M, S, V) / / TransformT CreateTiles (w, h)L, K DuplicateWithKeys (M ' , T) / / Indices and KeysSortByKeys (K, L) / / Globally SortR Identif yTileRanges (T, K)1 ^ 0 / / Init Canvasfor all Tiles t in I dofor all Pixels i in t dor GetTileRange (R, t )I [i] BlendlnOrder ( i , L, r, K, M ' , S ' , C, A)end forend forreturn Iend functionTable 2
[0088] Table 2 describes a rasterization function (e.g., a render function) implemented by Table 1 and outlines how the 3D Gaussian elements are rasterized into a 2D image using a GPU-optimized pipeline. Gaussian elements outside the current view are culled for efficiency. The visible Gaussian elements are projected into screen space based on the current camera view, and a tiled representation of the image is created to enable parallel rendering. Each Gaussian is then duplicated across the tiles it affects, and globally sorted by rendering order. Tile ranges are computed, and an empty image canvas is initialized. For each tile and each pixel in the tile, a blend operation is performed in depth or alpha order to composite the final image.
[0089] FIG. 4C depicts the projector 460 and the differentiable tile rasterizer 480 of FIG. 4B in more detail. The projector 460 identifies the gaussians 461 that are visible to the virtual camera 464 (e.g., 464A-464K) for a pose 463. A “pose” refers to the positionD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO27 / 51and orientation of the virtual camera in a world coordinate system, with respect to six degrees of freedom (6D0F), using different representations, e.g., a transformation matrix. The projector 460 performs gaussian splatting, which entails projecting the visible gaussians 461 (e.g., 461 A-461K), which are three dimensional, onto a two-dimensional (2D) image plane 465 (e.g., 465A-465K) based on the pose 463 (e.g., 463A-463K) of the virtual camera 464 and the intrinsic parameters of the virtual camera 464. Each gaussian 461 is represented as an ellipse 466 (e.g., 466A-466K)in the 2D image plane 465. The ellipses 466 can overlap one another and can have various sizes and orientations depending on parameters of the gaussians 461 that are projected and the pose 463 of the virtual camera 464. For example, the gaussians 461 A that are visible to the virtual camera 464A in the pose 463 A are projected on to a 2D image plane 465 A as ellipses 466A, and the gaussians 46 IK that are visible to the virtual camera 464K in the pose 463 K are projected on to a 2D image plane 465K as ellipses 466K.
[0090] Data 467 describing the ellipses 466 on the 2D image plane 465 is provided to a differentiable tile rasterizer 480. For example, data 467A describes the ellipses 466A associated with the pose 463 A, and data 467K describes the ellipses 466K associated with the pose 463 K. The differentiable tile rasterizer 480 is configured to perform operations to blend the ellipses 466 to generate pixel data representing a 2D image that would be captured by the virtual camera 464 for the particular selected pose 463. To generate the pixel data, the differentiable tile rasterizer 480 divides the 2D image plane 465 into tiles (represented in FIG. 3B). For purposes of this description, each tile is treated as corresponding to a single pixel; however, the exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel. Additionally, to simplify the drawings, in FIG. 3B, the 2D image plane 310 includes some regions that are not associated with any gaussian; however, it will generally be the case that each tile is associated with at least one Gaussian 302, 304, 306 (FIG. 3B) to provide color information for the tile.
[0091] The differentiable tile rasterizer 480 obtains depth information 481 for the visible gaussians 461 of the pose 463, where depth refers to the distance in the scene of a gaussian from the virtual camera 464. For example, the depth information 481 can be calculated for each of the visible gaussians 461 based on the location of the virtual camera 464 in the scene (as indicated by data describing the pose 463) and the location of the center of the Gaussian in the scene (e.g., as indicated by the X-axis center positionD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO28 / 51coordinate, the Y-axis center position coordinate, and the Z-axis center position coordinate of each Gaussian). For example, depth information 481 A indicates distances in the scene between each of the gaussians 461 A and the virtual camera 464 A, and depth information 48 IK indicates distances in the scene between each of the gaussians 46 IK and the virtual camera 464K.
[0092] The differentiable tile rasterizer 480 can perform tile-depth association 484 to, for each tile of the 2D image plane 465, sort the ellipses 466 that at least partially overlap the tile based on the depth information 481. Since the Gaussians are generally not opaque, a first gaussian that is more distant from the virtual camera 464 in the scene can be partially visible through a second gaussian that is closer to the virtual camera 464. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 480 accounts for the colors of each gaussian of a set of gaussians that influence the particular pixel, the opacities of the gaussians of the set, and the order of the gaussians of the set as viewed from the virtual camera 464.
[0093] To parallelize generation of the pixel data, the differentiable tile rasterizer 480 can perform thread association 486 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources, such computing resources can include one or more cores of one or more central processing unit (CPU), one or more cores of one or more graphics processing unit (GPU), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colors of the gaussians (as determined based on the pose 463 of the virtual camera 464 and color parameters specified by spherical harmonic color coefficients, opacity of the gaussians, and the depth information 481).
[0094] Blending the contributions of the gaussians results in pixel data 489 representing an image 490 (e.g., a view) captured by the virtual camera 464 for the particular pose 463. The pixel data 489 may be provided to a loss calculator 495, which is configured to compare the pixel data 489 to the ground-truth image 452 corresponding to the pose 463. The loss calculator 495 determines a loss value for each pixel (e.g., a photometric L2 loss) to quantify how the pixel data 489 differs from the ground-truth image 452. Gradient values 497 based on the loss values may be backpropagated throughD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO29 / 51the differentiable tile rasterizer 480 and the projector 460 to estimate updated parameter values for the visible gaussians 461 of the respective pose 463.
[0095] In some implementations, a Gaussian scene representation is updated selectively such that only a subset of Gaussians is modified in response to changes in an environment.
[0096] Use of a gaussian scene representation in a situation in which the gaussian scene representation is periodically or occasionally updated to capture new scene information presents significant challenges due to the computational complexity of updating gaussian scene representations. Aspects disclosed herein reduce the computational complexity associated with updating a gaussian scene representation by using different update strategies to update the gaussian scene representation in different situations. For example, one update strategy can include performing a full update of the gaussian scene representation, and another update strategy is to perform a partial update of the gaussian scene representation. As used herein, a “partial update” refers to updating a specified subset of the gaussians of the gaussian scene representation while holding other gaussians of the gaussian scene representation constant.
[0097] The specified subset of the gaussians updated during a partial update are selected based, at least in part, on changes detected in the scene. In some implementations, changes in the scene are identified based on input data associated with the environment, and mapping data is used to identify Gaussians corresponding to the detected changes.
[0098] The one or more processors are configured to perform pixel-to-gaussian mapping of at least a portion of the input pixel data to identify gaussians of the gaussian scene representation to be updated based on the input pixel data. The processor(s) can identify a region of interest associated with differences between the input pixel data and the reference pixel data, and use mapping data to identify gaussians of the gaussian scene representation that are associated with the region of interest.
[0099] After identifying Gaussians to be updated, some implementations further determine which portions of the input data are used to update those Gaussians.
[0100] The one or more processors are configured to perform gaussian-to-pixel mapping to identify pixels of the input pixel data to be used to update the gaussian scene representation. Mapping data can be used to determine which pixels of the input pixelD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO30 / 51data are to be used as training data to update the selected gaussians. The mapping data can be stored for subsequent reference.
[0101] During a partial update, Gaussians that are not included in the specified subset are held constant. Based on the selected subset of Gaussians and corresponding input data, the Gaussian scene representation is updated without modifying Gaussians that are not selected.
[0102] In some implementations, the Gaussian scene representation is updated according to an update cycle in which input data associated with the environment is obtained and changes in the environment are detected.
[0103] For a given update cycle, only Gaussians associated with detected changes in the environment are selected for update, while remaining Gaussians are held constant. The selection of Gaussians for update may be performed using pixel-to-gaussian mapping, region-of-interest identification, gaussian-to-pixel mapping, or combinations thereof. Mapping data generated during a prior update cycle may be reused to identify Gaussians or pixels for subsequent update cycles.
[0104] By limiting updates to the selected subset of Gaussians, computational complexity associated with maintaining the Gaussian scene representation is reduced. Selective updating of Gaussians enables efficient maintenance of the Gaussian scene representation without recomputing the entire representation for each update cycle.Example Resource Constrained 3DGS and NVS Processes
[0105] Aspects of the present disclosure relate to implementing and rendering 3DGS and NVS models on resource constrained devices, such as autonomous vehicles or robots. Some devices and systems, such as some autonomous vehicles, semi-autonomous vehicles, robots, and the like may have a fixed set of resources including computation capacity, memory, network bandwidth, storage, and power to carry out 3DGS and NVS processes. Accordingly, for devices with a fixed set of available resources, the 3DGS and NVS processes need to be adapted to be effectively implemented. For example, there may be instances where the image quality for certain features in the environment can be reduced (e.g., by reducing the number of Gaussians utilized to represent the feature), so that implementation of the 3DGS and / or NVS processes can be performed by the device.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO31 / 51
[0106] The techniques described herein may control the number of Gaussians utilized for rendering the 3DGS model of an environment. In certain aspects, other parameters may also be adjusted and or otherwise controlled, such as parameters of the spherical harmonics utilized in the Gaussian splatting process, for example the spherical harmonic order or the spherical harmonic degree. Implementations of the techniques described herein solve technical problems associated with implementing 3DGS and NVS processes on resource constrained devices and also enabling those devices to carry out real-time or near real-time operations that depend on the models rendered from the 3DGS and NVS processes.
[0107] FIGS. 5A and 5B depict the environment 100 illustrated in FIG. 1 from the point of view of the driver of the first vehicle 102 at a first time and a second time, respectively. The environment depicted in FIG. 5A may be the content of the environment at the first time (e.g., first environment 500) and may be the content that is used to generate a corresponding base 3DGS model. The environment depicted in FIG.5B may be the content of the environment at the second time (e.g., second environment 502). The second environment 502 may not be fully rendered into a 3DGS model. Instead, following the techniques described herein, foreground features may be identified in the image of the second environment 502. The foreground features may be rendered into 3DGS models and compared with foreground features of the base 3DGS model to carry out an update of the base 3DGS model and thereby generate an updated 3DGS model corresponding to the second environment.
[0108] For example, foreground features depicted in the first environment 500 may include the second vehicle 103, a third vehicle 104, a fourth vehicle 105, and a sixth vehicle 106. The foreground features depicted in the second environment 502, which is the same location as the first environment 500 but at a time later than the first environment 500, does not include the third vehicle 104, the fourth vehicle 105, and the sixth vehicle 106 anymore, but now includes a person 124. Once the foreground features are identified they can be compared and those from the second environment 502 can replace the foreground features of the first environment 500 by incorporating them into the background of the first environment 500. Accordingly, a Gaussian splatting process for rendering a 3DGS model for the second environment can be streamlined as only updates to the foreground features may be rendered and the previously rendered background fromD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO32 / 51the base 3DGS model corresponding to the first environment 500 serves as the background for the updated 3DGS model corresponding to the second environment 502.
[0109] Reference will be made to FIGS. 5A and 5B as the techniques for utilizing a base 3DGS model for a new Gaussian instance of the same environment are discussed in more detail with reference to FIGS.6A-6B which depict block diagrams of the technique.
[0110] At block 610A in FIG. 6A, a base 3DGS model of an environment corresponding to a first time is obtained, for example from a data storage device that is either local or remote to the apparatus. At block 610B in FIG. 6B, a base 3DGS model of an environment corresponding to a first time may be generated from a plurality of image captured of the environment. The base 3DGS model may be generated from a plurality of images of the environment obtained from block 602 or from a video of the environment obtained from block 602. When a video of the environment is used as the image input to the Gaussian splatting process (e.g., process 420 depicted and described with reference to FIG.4B), the video may be decomposed into individual frames. If block 610 does not obtain a previously generated base 3DGS model, then it may be configured to generate the base 3DGS model based on the Gaussian splatting process, for example, process 420 depicted and described with reference to at least FIG. 4B.
[0111] At block 620, foreground features and background features in the base 3DGS model are separated. The process, at block 620, of separating the foreground features and the background features, may be performed in the image space domain or the 3D Gaussian space, from block 625. The process of separation in the image space domain will be discussed first. As an initial step, one or more viewpoints (e.g., viewing location and field of view angle) are defined to generate corresponding image renderings from the base 3DGS model. The one or more viewpoints may be defined on the basis of uniform distribution by location, a random distribution, or another method for determining viewpoints. The one or more viewpoints may be input to the pose selector 324 of FIG.4C, such that the projector 460 (FIG. 4C) may generate corresponding 2D image planes. The 2D image planes may be rasterized with the differentiable tile rasterizer 480 to generate images corresponding to each of the one or more viewpoints. The generated images may be processed using a semantic segmentation tool to identify features that correspond to a background and features that correspond to a foreground. As discussed herein, the background features may be static (e.g., not currently moving) and / or immovable features (e.g., features that a human cannot readily move like a table or chair).D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO33 / 51Foreground features may be features that are dynamic (e.g., in motion objects) and / or movable features, such as a table, a chair, parked vehicle, or the like. Background and foreground features may be classified into one or more classes, such as sky, building, road, sign, human, animal, furniture, vehicle, and / or the like. In some aspects, block 620 may also include processing the images to identify objects and classify the object with a class label.
[0112] In certain aspects, the process of separation may be performed in the 3D Gaussian space. In such instances, a 3D segmentation method may segment (e.g., label) the scene represented by the base 3DGS model into background and foreground Gaussians (e.g., annotate the Gaussians as belonging to a background or foreground segmentation). In some aspects, an object detection and classification process may be used to segment (e.g., label) the scene represented by the base 3DGS model into background and foreground Gaussians (e.g., annotate the Gaussians as belonging to a background or foreground segmentation).
[0113] The Gaussians forming the Gaussian scene representation of the environment may be structured and stored as structured data in a database that is queryable, at block 630. FIG.7 depicts an illustrative structured data representation of the Gaussians forming the Gaussian scene representation. A database 700 may store one or more 3DGS models corresponding to an environment. The 3DGS models can be delimitated by location and optionally by time. That is, a set of first 3DGS models may correspond to Location 1. The set of first 3DGS models may be further stored with a time stamp referring to the time at which the model corresponds to. The database can store 3DGS models from a plurality of locations (e.g., Location 1, Location 2, Location n).
[0114] A location may be viewed from multiple perspectives. Accordingly, a 3DGS model may further be categorized by the field of view (FOV) of the location that is represented by the Gaussians that form the 3DGS model. For example, the FOV may be corresponding to a viewing direction, a viewing angle (e.g., 90 degrees, 180 degrees, 270 degrees, 360 degrees, and optionally an azimuth angle). Each of the FOVs may be processed as discussed with respect to block 620. That is, the base 3DGS model corresponding to Location 2, Time 2, and FOV 2 may be semantically segmented into foreground and background segmentations. Each of the foreground and background segmentations may be classified into one or more different classification categories, such as sky, road, ground, vehicle, human, animal, structure, or the like. The classificationsD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO34 / 51may be assigned to groups of Gaussians that represent the corresponding feature (e.g., sky, road, ground, vehicle, human, animal, structure, or the like).
[0115] As further depicted in FIG. 7, the Gaussian scene representation for each classified feature (e.g., Class 1, Class 2, Class n, Class 1 ’, Class 2’, Class n’) is composed of a plurality of Gaussians. Each Gaussian scene representation 754, 754’ includes a plurality of gaussians 756, such as: a gaussian 756A, a gaussian 756B, and a gaussian 756Q (where Q is an integer greater than 2). Each gaussian 756 is characterized by a plurality of parameters describing various properties of the gaussian 756. For example, in FIG. 7, the gaussian 756 A is characterized by an X-axis center position coordinate (“center pos. (X)”) 780, Y-axis center position coordinate (“center pos. (Y)”) 781, and Z-axis center position coordinate (“center pos. (Z)”) 782. Together, the X-axis center position coordinate 780, the Y-axis center position coordinate 781, and the Z-axis center position coordinate 782 indicate a center position of the gaussian 756 A and correspond to the position vector, p i. In FIG. 7, the gaussian 756A is also characterized by an X-axis scale parameter 783, a Y-axis scale parameter 784, and a Z-axis scale parameter 785. Together the X-axis scale parameter 783, the Y-axis scale parameter 784, and the Z-axis scale parameter 785 indicate a size and shape of the ellipsoid of the gaussian 756A and correspond to the scale vector, s i. The gaussian 756A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO) 786, rotation (ql) 787, rotation (q2) 788, rotation (q3) 789) specified as quaternions in the example illustrated in FIG. 7. Together the rotation parameters 786-789 indicate a spatial orientation of the ellipsoid of the gaussian 756A and correspond to the orientation parameters, q_i. The gaussian 756A is also characterized by an opacity parameter 790, which indicates a maximum opacity of the gaussian 756A. The opacity parameter 790 corresponds to the opacity parameter, t_i. The gaussian 756A is also characterized by a set of color parameters, e.g., color parameters 791, 792, and 793 in FIG. 7. The color parameter(s) 791-793 can be specified as a set of spherical harmonics (SH) coefficients, H i. Although FIG. 7 illustrates the gaussian 756A as characterized by a set of fourteen parameters, in other examples, the gaussians 756 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters 791-793 can be used, depending on the order of the SH coefficients used to represent the colors of the gaussians 756.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO35 / 51
[0116] In some implementations, not all of the parameters described above are used to specify a gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacity parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a viewindependent color) rather than higher-order spherical harmonics that capture viewdependent color variations. Using a reduced parameter set can decrease memory usage and computational complexity associated with storing, updating, and rendering the gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.
[0117] In some implementations, the gaussian scene representation may use two-dimensional (2D) gaussians rather than three-dimensional (3D) gaussians. A 2D gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D gaussians, which distribute density throughout a model the complete volumetric ellipsoid, 2D gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.
[0118] A 2D gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parameters (e.g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D gaussian primitive may have fewer parameters than a corresponding 3D gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO36 / 51
[0119] Using 2D gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D gaussian primitives can be more tightly aligned to object surfaces than 3D gaussians. Additionally, 2D gaussians may provide more accurate geometry representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.
[0120] In some implementations, the gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each gaussian of the gaussian scene representation can be used to represent a triangular face of a mesh, enabling conversion between gaussian representations and mesh representations. This approach provides a one-to-one mapping between gaussians and triangular mesh faces that is differentiable, enabling the gaussian scene representation to be modified in a manner similar to modifying a mesh.
[0121] To convert a 2D gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D gaussian may be performed. The covariance matrix L can be decomposed as L = RART, where R is a matrix of eigenvectors (el, e2, e3) and A is a diagonal matrix of eigenvalues (XI, X2, X3). The two largest eigenvalues ( I, X2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (X3) corresponds to the normal direction. A triangular face can then be constructed using the mean of the gaussian as a center point and defining triangle vertices along the tangent directions. For example, triangle vertices may be defined as vl = p + alel, v2 = p + a2e2, and v3 = p - oil el, where al and a2 are scale factors proportional to square roots of their respective eigenvalues XI and X2 and p is the center point corresponding to the mean of the 2D gaussian. This creates a triangle aligned with the gaussian's orientation and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the gaussian.
[0122] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the gaussian. When the mesh is modified, the gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the gaussians when the mesh is animated or otherwise modified. During rendering, the gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of gaussians of the gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between gaussians and triangular faces.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO37 / 51
[0123] Throughout this disclosure, references to gaussians, gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D gaussians, 2D gaussians, mesh-integrated gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the type of gaussian representation used, as described herein.
[0124] Block 640 corresponds to a device seeking to build a 3DGS model corresponding to its current environment. Block 640 may obtain image or video data 606 corresponding to the environment at time (t) which may be a current time or a more current time than the time (t-n) at which the base 3DGS model was rendered. Similar to the image space process described with reference to block 620, block 640 may process the image and / or video data to separate (e.g., classify) foreground and background features. In certain aspects, the foreground features may be rendered into Gaussians through process 420 for Gaussian splatting depicted and described with reference to at least FIG. 4B. That is, Gaussian representations for the foreground features may be rendered while the background features may be omitted from the rendering process, thereby reducing computational resources and time for processing.
[0125] Additionally, at block 640, the device may query the database to determine whether a base 3DGS model for the environment can be obtained and updated. In such instances, at block 640 a query of the database may be generated and implemented with the database. The query may include query parameters such as one or more of location, a time, a FOV, and / or optionally one or more classes (e.g., classification categories discussed herein). The database may return Gaussians and Gaussian parameters for Gaussians that make up the base 3DGS model of environment corresponding to the query. For example, the Gaussians 756 and Gaussian parameters illustratively depicted in FIG.7.
[0126] In certain aspects, the Gaussians (e.g., first Gaussians) that correspond to foreground features representing the environment at time (t) have corresponding coordinate locations within the 3D Gaussian representation. The first Gaussians can be configured to replace the corresponding Gaussians from the base 3DGS model so that the base 3DGS model is updated with current foreground feature information based on the corresponding coordinate locations of each of the first Gaussians and the Gaussians from the base 3DGS model.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO38 / 51
[0127] At block 650, the updated set of Gaussians and Gaussian scene representations for each of the foreground features based on the image and / or video data from the time (t) and the background features from the base 3DGS model from time t-n, may be fed into the projector 460 (FIG. 4C) at block 660. The projector 460 processes may be implemented at block 660 to render 2D image planes based on the updated set of Gaussians and Gaussian scene representations (e.g., the updated 3DGS model).
[0128] Block 670 may then receive the 2D image planes generated by the projector 460 implemented at block 660. Block 670 implements a differentiable tile rasterizer 480 to generate one or more images 490 of the environment from time t.
[0129] The one or more images 490 may be used by any number of post process tasks such as autonomous or semi-autonomous vehicle control, robotic manipulations, or the like.
[0130] It should be understood that techniques for utilizing a base 3DGS model for a new Gaussian instance of the same environment are provided herein. The techniques described herein reduce computational resources and time for processing a 3DGS model of an environment by utilizing base 3DGS models and updating the foreground information with current foreground features while maintaining the background features.
[0131] It should be noted that the process depicted in the block diagrams in FIGS.6A-6B are only two example processes. Variations or alternative process utilizing similar technology can be utilized.Example Operations
[0132] FIG. 8 shows an example method 800 for updating preexisting 3DGS models with an apparatus, the system 900 of FIG. 9. Aspects of the method 800 can be implemented by a computing device comprising one or more processors and a non-transitory computer readable memory and / or the system 900 of FIG. 9.
[0133] Method 800 begins at block 805 with obtaining a base 3D Gaussian splat (3DGS) model of an environment corresponding to a first time, wherein the base 3DGS model comprises Gaussians classified as either background or foreground.
[0134] Method 800 proceeds to block 810 with obtaining foreground features of the environment.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO39 / 51
[0135] Method 800 proceeds to block 815 with updating base model Gaussians corresponding to the base 3DGS model with one or more Gaussians for each of the foreground features corresponding to a second time after the first time.
[0136] Method 800 proceeds to block 820 with outputting an updated 3DGS model comprising Gaussians corresponding to background features from the base 3DGS model and the one or more Gaussians for each of the foreground features.
[0137] In some aspects, method 800 includes obtaining the one or more images of the environment corresponding to the second time after the first time; segmenting the one or more images of the environment into background features and the foreground features; and rendering the one or more Gaussians for each of the foreground features.
[0138] In some aspects, segmenting the one or more images of the environment into the background features and the foreground features comprises: determining that one or more features in the one or more images is immovable or static; and classifying the one or more features as one of the background features
[0139] In some aspects, segmenting the one or more images of the environment into the background features and the foreground features comprises: determining that one or more features in the one or more images is moveable or dynamic; and classifying the one or more features as one of the foreground features.
[0140] In some aspects, obtaining the base 3DGS model comprises querying a database of 3DGS models based on at least one of a location, a time, or a field of view.
[0141] In some aspects, the Gaussians of the base 3DGS model are classified as either background or foreground based on a 3D segmentation process.
[0142] In some aspects, method 800 includes further comprising implementing a task based on the updated 3DGS model.
[0143] In some aspects, the task is at least one of: a vehicle control operation; or a robotic manipulation.
[0144] Note that FIG.8 is just one example of a method, and other methods including fewer, additional, or alternative operations are possible consistent with this disclosure.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO40 / 51Example System
[0145] FIG. 9 is a block diagram illustrating an example system 900 according to techniques of this disclosure. In various examples, system 900 may be part of a computing device, video gaming console, smartphone, mobile device, computing device, vehicle, robot, or other type of device. System 900 includes a computing system 902. Additionally, in the example of FIG. 9, system 900 may include one or more image sources 904. Computing system 902 may include one or more computing devices, such as personal computers, chipsets, mobile devices, or other types of devices.
[0146] Image sources 904 are configured to generate image data, such as Red-Green-Blue (RGB) images or images in other color spaces. Image sources 904 may be positioned at various locations around system 900. For instance, in an example where system 900 is a vehicle, image sources 904 may include two or more forward- facing image cameras, two or more rear-facing image cameras, and so on. In some examples, image sources 904 include depth cameras that are configured to generate depth images. Depth images represent the depths of objects. In some examples, there is a depth camera for each of image sources 904. For instance, in an example where image sources 904 include a left image camera and a right image camera, the depth cameras may include a left depth camera corresponding to the left image camera and a right depth camera corresponding to the right image camera. In other examples, there are multiple image sources 904 and a single depth camera. Depth images generated by a depth camera may represent the depths of objects shown in images generated by an image camera corresponding to the depth camera.
[0147] In the example of FIG. 9, computing system 902 includes at least one central processing unit (CPU) 906, at least one graphic processing unit 908, a display device 910, and a storage system 912. CPU 906, GPU 908, display device 910, and storage system 912 may be communicatively coupled. CPU 906 includes one or more processors 914. Processors 914 may be implemented in circuitry. Example types of processors 914 may include microprocessors, digital signal processors, application-specific integrated circuits (ASICs), and so on. Display device 910 may include display screens, extended reality display devices, and other devices for displaying output. Storage system 912 may include one or more non-transitory computer-readable storage media. Example types of non-transitory computer-readable storage media may include random access memory (RAM) units, disk drives, and so on. CPU 906, GPU 908, display device 910, and the computer-D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO41 / 51readable storage media of storage system 912 may be distributed among two or devices of computing system 902 or may be consolidated within a single device of computing system 902.
[0148] Storage system 912 may be configured to store various types of data and computer-readable instructions. In the example of FIG. 9, storage system 912 stores data and instructions associated with an application 916. Application 916 may be one of a variety of different types of application, such as a video game application, a graphics design application, a 3D rendering application, and so on. Processors 914 of CPU 906 may execute application 916 to generate geometry data.
[0149] In certain aspects, the application 916 and / or the computer-readable storage media of storage system 912 is configured to store instructions (e.g., computer-executable code), that when executed by the one or more processors 914, cause the one or more processors 914 to perform method 800 described with respect to FIG. 8, or any aspect related to it, including any operations described in relation to FIG. 8 Note that reference to a processor performing a function of system 900 may include one or more processors 914 performing that function of the system 900, such as in a distributed fashion.
[0150] In the example of FIG. 9, GPU 908 includes a geometry processing system 918 and a renderer 920. Geometry processing system 918 may include one or more processors that perform a geometry processing step of a graphics pipeline in order to generate a set of primitives based on the geometry data. Renderer 920 may include one or more processors that perform a rasterizing step of the graphics pipeline in order to generate one or more rendered images. A rendered image is a 2-dimensional (2D) image, such as a 2D image suitable for display on a display screen. In some examples, display device 910 displays one or more of the rendered images. In some examples, storage system 912 stores one or more of the rendered images.
[0151] The system 900 may include a transceiver 950 (e.g., a transmitter and / or a receiver) coupled to the CPU 906. The transceiver 950 is configured to transmit and receive signals for the system 900 via an antenna 952, such as the various signals as described herein. The CPU 906 may be configured to perform processing functions for the system 900, including processing signals received and / or to be transmitted by the system 900.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO42 / 51
[0152] More generally, means for communicating, transmitting, sending or outputting for transmission may include the one or more transceivers 950 and / or antenna 952 of the system 900 in FIG. 9, and / or one or more processors 914 of the system 900 in FIG. 9. Means for communicating, receiving or obtaining may include the one or more transceivers 950 and / or antenna 952 of the system 900 in FIG. 9, and / or one or more processors 914 of the system 900 in FIG. 9.Example Clauses
[0153] Implementation examples are described in the following numbered clauses:
[0154] Clause 1: A method for updating preexisting 3DGS models comprises: obtaining a base 3D Gaussian splat (3DGS) model of an environment corresponding to a first time, wherein the base 3DGS model comprises Gaussians classified as either background or foreground; obtaining foreground features of the environment; updating base model Gaussians corresponding to the base 3DGS model with one or more Gaussians for each of the foreground features corresponding to a second time after the first time; and outputting an updated 3DGS model comprising Gaussians corresponding to background features from the base 3DGS model and the one or more Gaussians for each of the foreground features.
[0155] Clause 2: The method of Clause 1, further comprising: obtaining the one or more images of the environment corresponding to the second time after the first time; segmenting the one or more images of the environment into background features and the foreground features; and rendering the one or more Gaussians for each of the foreground features.
[0156] Clause 3: The method of any one of Clauses 1-2, wherein segmenting the one or more images of the environment into the background features and the foreground features comprises: determining that one or more features in the one or more images is immovable or static; and classifying the one or more features as one of the background features.
[0157] Clause 4: The method of any one of Clauses 1-2, wherein segmenting the one or more images of the environment into the background features and the foreground features comprises: determining that one or more features in the one or more images is moveable or dynamic; and classifying the one or more features as one of the foreground features.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO43 / 51
[0158] Clause 5: The method of any one of Clauses 1-4, wherein obtaining the base 3DGS model comprises querying a database of 3DGS models based on at least one of a location, a time, or a field of view.
[0159] Clause 6: The method of any one of Clauses 1-5, wherein the Gaussians of the base 3DGS model are classified as either background or foreground based on a 3D segmentation process.
[0160] Clause 7: The method of any one of Clauses 1-6, further comprising implementing a task based on the updated 3DGS model.
[0161] Clause 8: The method of any one of Clauses 1-6, wherein the task is at least one of: a vehicle control operation; or a robotic manipulation.
[0162] Clause 9: One or more apparatuses, comprising: one or more memories comprising executable instructions; and one or more processors configured to execute the executable instructions and cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1-8.
[0163] Clause 10: One or more apparatuses configured for feature detection and extraction, comprising: one or more memories; and one or more processors, coupled to the one or more memories, configured to cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1-8.
[0164] Clause 11: One or more apparatuses configured for feature detection and extraction, comprising: one or more memories; and one or more processors, coupled to the one or more memories, configured to perform a method in accordance with any one of Clauses 1-8.
[0165] Clause 12: One or more apparatuses, comprising means for performing a method in accordance with any one of Clauses 1-8.
[0166] Clause 13 : One or more non- transitory computer-readable media comprising executable instructions that, when executed by one or more processors of one or more apparatuses, cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1-8.
[0167] Clause 14: One or more computer program products embodied on one or more computer-readable storage media comprising code for performing a method in accordance with any one of Clauses 1-8.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO44 / 51
[0168] Clause 15: One or more apparatuses configured for feature detection and extraction, comprising: a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1-8.Additional Considerations
[0169] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limiting of the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For instance, the methods described may be performed in an order different from that described, and various actions may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.
[0170] The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, an Al processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO45 / 51microprocessors, one or more microprocessors in conjunction with a DSP core, a SoC, a SiP, or any other such configuration.
[0171] As used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).
[0172] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.
[0173] As used herein, “coupled to” and “coupled with” generally encompass direct coupling and indirect coupling (e.g., including intermediary coupled aspects) unless stated otherwise. For example, stating that a processor is coupled to a memory allows for a direct coupling or a coupling via an intermediary aspect, such as a bus.
[0174] The methods disclosed herein comprise one or more actions for achieving the methods. The method actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims. Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an ASIC, or processor.
[0175] The following claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims. Reference to an element in the singular is not intended to mean only one unless specifically so stated, but rather “one or more.” The subsequent use of a definite article (e.g., “the” or “said”) with an element (e.g., “the processor”) is not intended to invoke a singular meaning (e.g., “only one”) on the element unless otherwise specifically stated. For example, reference to an element (e.g., “a processor,” “the processor,” etc.), unlessD&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO46 / 51otherwise specifically stated, should be understood to refer to one or more elements (e.g., “one or more processors,” or the like). The terms “set” and “group” are intended to include one or more elements, and may be used interchangeably with “one or more.” Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g., an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions. Unless specifically stated otherwise, the term “some” refers to one or more. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.D&S Ref. No.: QCM2503356WO
Claims
Qualcomm Ref. No.: 2503356WO47 / 51CLAIMS1. An apparatus, comprising:a processing system that includes one or more processors and one or more memories coupled with the one or more processors, the processing system configured to cause the apparatus to:obtain a base 3D Gaussian splat (3DGS) model of an environment corresponding to a first time, wherein the base 3DGS model comprises Gaussians classified as either background or foreground;obtain foreground features of the environment;update base model Gaussians corresponding to the base 3DGS model with one or more Gaussians for each of the foreground features corresponding to a second time after the first time; andoutput an updated 3DGS model comprising Gaussians corresponding to background features from the base 3DGS model and the one or more Gaussians for each of the foreground features.
2. The apparatus of claim 1, wherein the processing system configured to cause the apparatus to:obtain one or more images of the environment corresponding to the second time after the first time;segment the one or more images of the environment into background features and the foreground features; andrender the one or more Gaussians for each of the foreground features.
3. The apparatus of claim 2, wherein to segment the one or more images of the environment into the background features and the foreground features the processing system is configured to cause the apparatus to:determine that one or more features in the one or more images is immovable or static; andclassify the one or more features as one of the background features.
4. The apparatus of claim 2, wherein to segment the one or more images of the environment into the background features and the foreground features the processing system is configured to cause the apparatus to:D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO48 / 51determine that one or more features in the one or more images is moveable or dynamic; andclassify the one or more features as one of the foreground features.
5. The apparatus of claim 1, wherein to obtain the base 3DGS model, the processing system is configured to cause the apparatus to query a database of 3DGS models based on at least one of a location, a time, or a field of view.
6. The apparatus of claim 1, wherein the Gaussians of the base 3DGS model are classified as either background or foreground based on a 3D segmentation process.
7. The apparatus of claim 1, wherein the processing system is configured to cause the apparatus to implement a task based on the updated 3DGS model.
8. The apparatus of claim 7, wherein the task is at least one of:a vehicle control operation; ora robotic manipulation.
9. The apparatus of claim 1, further comprising one or more cameras configured to capture image data or video data of the environment.
10. The apparatus of claim 1, further comprising a navigation module of the apparatus configured to control the apparatus based on the updated 3DGS model.
11. The apparatus of claim 1 , further comprising a transceiver configured to transmit the updated 3DGS model to another apparatus.
12. The apparatus of claim 1, further comprising one or more sensors configured to capture image data or video data of the environment.
13. A method, comprising:obtaining a base 3D Gaussian splat (3DGS) model of an environment corresponding to a first time, wherein the base 3DGS model comprises Gaussians classified as either background or foreground;obtaining foreground features of the environment;D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO49 / 51updating base model Gaussians corresponding to the base 3DGS model with one or more Gaussians for each of the foreground features corresponding to a second time after the first time; andoutputting an updated 3DGS model comprising Gaussians corresponding to background features from the base 3DGS model and the one or more Gaussians for each of the foreground features.
14. The method of claim 13, further comprising:obtaining one or more images of the environment corresponding to the second time after the first time;segmenting the one or more images of the environment into background features and the foreground features; andrendering the one or more Gaussians for each of the foreground features.
15. The method of claim 14, wherein segmenting the one or more images of the environment into the background features and the foreground features comprises:determining that one or more features in the one or more images is immovable or static; andclassifying the one or more features as one of the background features.
16. The method of claim 14, wherein segmenting the one or more images of the environment into the background features and the foreground features comprises:determining that one or more features in the one or more images is moveable or dynamic; andclassifying the one or more features as one of the foreground features.
17. The method of claim 13, wherein obtaining the base 3DGS model comprises querying a database of 3DGS models based on at least one of a location, a time, or a field of view.
18. The method of claim 13, wherein the Gaussians of the base 3DGS model are classified as either background or foreground based on a 3D segmentation process.
19. The method of claim 13, further comprising implementing a task based on the updated 3DGS model.D&S Ref. No.: QCM2503356WOQualcomm Ref. No.: 2503356WO50 / 5120. The method of claim 19, wherein the task is at least one of:a vehicle control operation; ora robotic manipulation.D&S Ref. No.: QCM2503356WO
