Three-dimensional (3D) reconstruction assisted 3D gaussian splatting with real time object replacement

The novel view synthesis system using 3D Gaussian splatting with real-time object replacement addresses the computational and memory challenges of existing 3D model construction, enabling efficient rendering on resource-constrained devices by replacing common objects with templates, thus reducing memory and processing demands.

WO2026240969A1PCT designated stage Publication Date: 2026-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/024989
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-07-31
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing 3D model construction systems are computationally intensive, memory-consuming, and generate significant heat, making them unsuitable for resource-constrained devices like mobile GPUs and XR headsets, and traditional methods like NeRF are slow and require extensive computational resources.

Method used

Implement a novel view synthesis system using 3D Gaussian splatting with real-time object replacement, where common objects are identified and replaced with object templates, reducing memory and processing demands by approximating less semantically important features with fewer Gaussian elements.

Benefits of technology

This approach allows for efficient, real-time rendering of 3D scenes with reduced memory and computational requirements, enabling 3D content generation on resource-constrained devices without thermal throttling or memory overflow.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026024989_19112026_PF_FP_ABST
    Figure US2026024989_19112026_PF_FP_ABST
Patent Text Reader

Abstract

Systems and techniques are described for novel view synthesis. For example, a computing device can identify a plurality of objects within a three-dimensional (3D) representation of a scene; determine that a first object of the plurality of objects corresponds to a first object template in a group (e.g., library) of object templates; and render the 3D representation based on the first object template.
Need to check novelty before this filing date? Find Prior Art

Description

Qualcomm Ref. 2503354WO1THREE-DIMENSIONAL (3D) RECONSTRUCTION ASSISTED 3D GAUSSIAN SPLATTING WITH REAL TIME OBJECT REPLACEMENTCROSS REFERENCE TO RELATED PATENT APPLICATION

[0001] This application claims priority to U. S. Provisional Patent Application Ser. No. 63 / 855,343 titled ‘ THREE-DIMENSIONAL (3D) RECONSTRUCTION ASSISTED 3D GAUSSIAN SPLATTING WITH SELECTABLE VIEWS,” filed on July 31, 2025, U. S. Provisional Patent Application Ser. No. 63 / 855,350 titled “THREE-DIMENSIONAL (3D) RECONSTRUCTION ASSISTED 3D GAUSSIAN SPLATTING WITH HIERARCHICAL STRUCTURE,” filed on July 31, 2025, U. S. Provisional Patent Application Ser. No. 63 / 855,359 titled “THREE-DIMENSIONAL (3D) RECONSTRUCTION ASSISTED 3D GAUSSIAN SPLATTING WITH REAL TIME OBJECT REPLACEMENT,” filed on July 31, 2025, U. S. Provisional Patent Application Ser. No. 63 / 804,910 titled “UPDATE STRATEGY SELECTION FOR GAUSSIAN SCENE REPRESENTATION,” filed on May 13, 2025, U. S. Provisional Patent Application Ser. No. 63 / 831,815 titled “SELECTIVE UPDATE OF GAUSSIAN SCENE REPRESENTATION BASED DIFFERENCE PIXELS AND UPDATE TARGETS,” filed on June 27. 2025, U. S. Provisional Patent Application Ser. No. 63 / 810,880 titled “GAUSSIAN SCENE REPRESENTATION UPDATE BASED ON PIXEL-TO-PARAMETER MAPPING,” filed on May 23, 2025, U. S. Provisional Patent Application Ser. No. 63 / 825,138 titled “GAUSSIAN SCENE REPRESENTATION UPDATE BASED ON GAUSSIAN-TO-PIXEL MAPPING,” filed on June 17, 2025, and U. S. Provisional Patent Application Ser. No. 63 / 820.238 titled “OBJECT MAPPING DATA FOR GAUSSIAN SCENE REPRESENTATION UPDATE,” filed on June 9, 2025, all of which are incorporated herein by reference in their entirety for all purposes.TECHNICAL FIELD

[0002] The present disclosure generally relates to image processing. For example, aspects of this disclosure relate to a three-dimensional (3D) reconstruction assisted 3D Gaussian splatting with real time object replacement.BACKGROUND

[0003] The increasing versatility of digital camera products has allowed digital cameras to be integrated into a wide array of devices and has expanded their use toQualcomm Ref. 2503354WO2different applications. For example, phones, drones, vehicles, computers, televisions, and many other devices today are often equipped with camera devices. The camera devices allow users to capture images and / or video (e.g., including frames of images) from any system equipped with a camera device. The images and / or videos can be captured for recreational use, professional photography, surveillance, and automation, among other applications. Moreover, camera devices are increasingly equipped with specific functionalities for modifying images or creating artistic effects on the images. For example, many camera devices are equipped with image processing capabilities for generating different effects on captured images.

[0004] Traditional systems for constructing 3D models use a significant amount of computational resources, memorv. and bandwidth, and in some cases generate significant heat in the process. In recent decades, there has been a demand for 3D content for computer graphics, virtual reality, communications, among other fields. There is also a demand for performing more computing tasks on portable computing devices (e.g., mobile devices) rather than stationary computing systems.SUMMARY

[0005] The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary has the sole purpose to present certain concepts relating to one or more aspects relating to the mechanisms disclosed herein in a simplified form to precede the detailed description presented below.

[0006] Systems and techniques are described herein for three-dimensional (3D) reconstruction assisted 3D Gaussian splatting. In some aspects, an apparatus for image processing is provided. The apparatus includes one or more memories and one or more processors coupled to the one or more memories and configured to: identify a plurality of objects within a three-dimensional (3D) representation of a scene; determine that a first obj ect of the plurality of obj ects corresponds to a first obj ect template in a group of obj ect templates; and render the 3D representation based on the first object template.Qualcomm Ref. 2503354WO3

[0007] In some aspects, a method for rendering one or more images is provided. The method includes: identifying a plurality of objects within a 3D representation of a scene; determining that a first object of the plurality of objects corresponds to a first object template in a group of object templates; and rendering the 3D representation based on the first object template.

[0008] In some aspects, a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: identify a plurality of objects within a 3D representation of a scene; determine that a first object of the plurality of objects corresponds to a first object template in a group of object templates; and render the 3D representation based on the first object template.

[0009] In some aspects, an apparatus for image processing is provided. The apparatus includes: means for identifying a plurality of objects within a 3D representation of a scene; means for determining that a first object of the plurality of objects corresponds to a first object template in a group of object templates; and means for rendering the 3D representation based on the first object template.

[0010] In some aspects, each of the apparatuses described above is, can be part of, or can include an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle or system or component of the vehicle, a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a smart or connected device, a camera system, a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the apparatus includes an image sensor (e.g., a camera) or multiple image sensors (e.g.. multiple cameras) for capturing one or more images. In some aspects, the apparatus includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the apparatuses described above can include one or more sensors. In some cases, the one or more sensors can be used for determining a location of the apparatuses, a state of theQualcomm Ref. 2503354WO4apparatuses (e.g., a tracking state, an operating state, a temperature, a humidity level, and / or other state), and / or for other purposes.

[0011] The foregoing has outlined rather broadly the features and technical advantages of examples according to the disclosure in order that the detailed description that follows may be better understood. Additional features and advantages will be described hereinafter. The conception and specific examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. Characteristics of the concepts disclosed herein, both their organization and method of operation, together with associated advantages, will be better understood from the following description when considered in connection with the accompanying figures. Each of the figures is provided for the purposes of illustration and description, and not as a definition of the limits of the claims.

[0012] While aspects are described in the present disclosure by illustration to some examples, those skilled in the art will understand that such aspects may be implemented in many different arrangements and scenarios. Techniques described herein may be implemented using different platform types, devices, systems, shapes, sizes, and / or packaging arrangements. For example, some aspects may be implemented via integrated chip implementations or other non-module-component based devices (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchasing devices, medical devices, and / or artificial intelligence devices). Aspects may be implemented in chip-level components, modular components, non-modular components, non-chip-level components, device-level components, and / or system-level components. Devices incorporating described aspects and features may include additional components and features for implementation and practice of claimed and described aspects. For example, transmission and reception of wireless signals may include one or more components for analog and digital purposes (e.g., hardware components including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders, and / or summers). It is intended that aspects described herein may be practiced in a wide variety of devices, components, systems, distributed arrangements, and / or end-user devices of varying size, shape, and constitution.Qualcomm Ref. 2503354WO5

[0013] Other obj ects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0014] The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Details of one or more aspects of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. However, the accompanying drawings illustrate only some typical aspects of this disclosure and are therefore not to be considered limiting of its scope. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims.

[0016] FIG. 1 is a block diagram illustrating an example architecture of an image capture and processing system, in accordance with some aspects of the disclosure;

[0017] FIG. 2 is an illustration of three-dimensional (3D) Gaussians that are rendered using multiple views based on 3D Gaussian splatting in accordance with some aspects of the disclosure;

[0018] FIG. 3A is an illustration of 3D Gaussian primitives that are rendered into a two-dimensional (2D) plane based on camera orientation in accordance with some aspects of the disclosure;

[0019] FIG. 3B is an illustration of alpha-blending the 3D Gaussians primitives to render a tile in connection with a volume grouping function in accordance with some aspects of the disclosure;Qualcomm Ref. 2503354WO6

[0020] FIG. 3C is an illustration of alpha-blending the 3D Gaussians to render a tile in connection with a volume grouping function in accordance with some aspects of the disclosure;

[0021] FIG. 4A illustrates a training system for a 3D Gaussian splatting model in accordance with some aspects of the disclosure;

[0022] FIG. 4B is a diagram illustrating an example of a process for 3D Gaussian splatting, in accordance with some aspects of the disclosure;

[0023] FIGs. 4C and 4D illustrate a render system for rendering images using a 3D Gaussian Model in accordance with some aspects of the disclosure;

[0024] FIG. 5 A is a block diagram of a novel view synthesis (NVS) system that is configured to render 3D representation of a scene in accordance with some aspects of the disclosure;

[0025] FIG. 5B illustrates a conceptual illustration 550 of an alignment associated with a 3D Gaussian template in accordance with some aspects of the disclosure;

[0026] FIG. 6 illustrates a conceptual flowchart of a process of an object template engine for replacing features within a 3D representation in accordance with some aspects of the disclosure;

[0027] FIG. 7 illustrates a conceptual flowchart of a process of a render engine for rendering an object template within a 3D representation in accordance with some aspects of the disclosure;

[0028] FIG. 8 illustrates another conceptual flowchart of a process of an object template engine for replacing features within a 3D representation in accordance with some aspects of the disclosure;

[0029] FIG. 9 is a flowchart illustrating a method of replacing identified objects with template objects in real time in accordance with some aspects of the disclosure;Qualcomm Ref. 2503354WO7

[0030] FIG. 10 shows an example of a computing system that can implement various aspects of the present disclosure and any engine illustrated in FIG. 1 or any component thereof.DETAILED DESCRIPTION

[0031] Certain aspects of this disclosure are provided below for illustration purposes. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure. Some of the aspects described herein can be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0032] The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example aspects will provide those skilled in the art with an enabling description for implementing an example aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0033] The terms ‘“exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage or mode of operation.

[0034] A camera is a device that receives light and captures image frames, such as still images or video frames, using an image sensor. The terms “image,” “image frame,” and “frame” are used interchangeably herein. Cameras may include processors, such as image signal processors (ISPs), that can receive one or more image frames and process the one or more image frames. For example, a raw image frame captured by a cameraQualcomm Ref. 2503354WO8sensor can be processed by an ISP to generate a final image. Processing by the ISP can be performed by a plurality of filters or processing blocks being applied to the captured image frame, such as denoising or noise filtering, edge enhancement, color balancing, contrast, intensity' adjustment (such as darkening or lightening), tone adjustment, among others. Image processing blocks or modules may include lens / sensor noise correction, Bayer filters, de-mosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, among others.

[0035] Cameras can be configured with a variety of image capture and image processing operations and settings. The different settings result in images with different appearances. Some camera operations are determined and applied before or during capture of the image, such as automatic exposure control (AEC) and automatic white balance (AWB) processing. Additional camera operations applied before, during, or after capture of an image include operations involving zoom (e.g., zooming in or out). ISO, aperture size, f / stop, shutter speed, and gain. Other camera operations can configure postprocessing of an image, such as alterations to contrast, brightness, saturation, sharpness, levels, curves, or colors.

[0036] As previously mentioned, there is a demand for three-dimensional (3D) content in various fields (e.g., for computer graphics, communications, extended reality (XR) such as virtual reality and / or augmented reality, etc.), triggering a change in emphasis for the requirements needed to generate 3D content. For example, existing systems for constructing 3D models can be built around specialized hardware, resulting in a high cost, and often cannot satisfy the requirements of these new applications. Such requirements have stimulated the use of digital imaging (e.g., using images from cameras) for 3D reconstruction.

[0037] In some cases, volume blocks (e.g., voxel blocks) can be utilized to reconstruct a 3D scene from two-dimensional (2D) images, such as stereo images obtained from a stereo camera. A voxel block represents a value on a regular grid in 3D space. As with pixels in a 2D bitmap, voxel blocks do not have their position (e.g., coordinates) explicitly encoded within their values. Instead, rendering systems infer the position of a voxel block based upon its position relative to other voxel blocks (e.g., its position in the data structure that makes up a single volumetric image).Qualcomm Ref. 2503354WO9

[0038] In some examples, a system can perform 3D reconstruction (3DR) using depth frames and corresponding poses (e.g., a pose of a camera when a depth frame is captured by the camera) as input for 3D scene reconstruction. For instance, the pose of the camera may be a six (6) degrees of freedom (6 DoF) pose. In some cases, the system may use a sequence of depth frames (or depth maps) of a scene and corresponding poses as an input. The depth frames can be generated using deep learning (DL) algorithms. non-DL algorithms, and / or other depth estimation methods. A 3D space of the scene can be uniformly sampled along the horizontal (X), vertical (Y), and depth (Z) directions. The 3D space can be divided into fixed size volumes (e g., block volumes with a fixed number of samples). For example, in some cases, when performing 3D surface reconstruction, the system can model the scene as a 3D sparse volumetric representation (e.g., referred to as a volume grid). The volume grid can contain a set of volume blocks, such as voxel blocks (or voxels). In some cases, the volume blocks are each indexed by their position in space with a sparse data representation (e.g., only storing blocks that surround an object and / or obstacle). In some aspects, the scene can be divided into a dense volumetric representation (as opposed to a sparse volumetric representation).

[0039] In some aspects, a system can perform 3DR to reconstruct a 3D scene from 2D depth frames (also referred to as depth maps) and 2D image frames (e.g., color frames, such as RGB frames having red (R), green (G), and blue (B) color values per pixel). The system can divide the scene into 3D blocks (e.g., voxel blocks or volume blocks, as noted previously). In some examples, a pose of a camera used to capture each 2D depth frame and / or 2D image frame can also be used to perform the 3DR. In some cases, the system may project each voxel block onto a 2D depth frame and a 2D image frame to determine the depth and / or color of the voxel block. For example, the system may project a 2D image frame into the 3D voxel space using the depth information of a 2D depth frame (indicating depth values for pixels of the 2D image) and may select voxel blocks in which a pixel may be located based on the depth information. Once all of the voxel blocks are updated based on the input depth and image frames, the process can repeat for a new depth frame and image frame pair or set. In some cases, color integration may not be needed. For instance, some 3DR systems may operate on depth frames only (and not image frames). The systems and techniques described herein can apply to depth only 3DR systems and to 3DR systems that operate on depth and image frames (e.g., color frames).Qualcomm Ref. 2503354WO10

[0040] As previously mentioned, in 3DR, 3D scenes are represented using a 3D volume of blocks which can be referred to as voxels (or voxel blocks or volume blocks). Each voxel may include (or is associated with) implicit surface information, such as in the form of an implicit surface representation (e.g., a truncated Signed Distance Function (TSDF)) value and a weight for depth integration. For instance, a TSDF value is a measure of distance of the voxel block from a surface, and the weight is a measure of the reliability’ of the TSDF value. A TSDF weight can be estimated using various approaches, such as a simple counter (e.g., a binary' weight of 1 or 0), based on a depth range, or from a confidence of the depth predictions. For instance, the TSDF may be a function which measures a distance d for each 3D point or voxel to the closest point on a surface of an object. Positive TSDF values may indicate that a voxel of a volume block is in front of a surface, negative TSDF values may indicate that a voxel of the volume block is within (e.g., behind, inside, etc.) the surface, and a 0 TSDF value may indicate the surface. In some cases, the distance d may be truncated to -1, 1 using a threshold. In some cases, a block selection algorithm can select a block if at least one depth pixel is determined to be located in the block. In such cases, there may be no need for a counter and thresholding, or a block can be selected if a counter is equal to 1.

[0041] In some aspects, a 3DR system may include three stages, including block selection, depth integration, and surface extraction. During block selection, blocks that have surfaces or are located close to a surface can be selected. These blocks can then be allocated into memory. In depth integration (also referred to as block integration), all voxel blocks within a block volume can be iterated over and an updated TSDF value and weight can be calculated. In surface extraction, a marching cubes algorithm can be used to determine triangular surfaces in the blocks.

[0042] Two approaches are generally employed to generate novel views of a 3D environment (e.g., views not included in any of the input images), including Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS). NeRF is an implicit representation and 3D Gaussian Splatting (3DGS) is an explicit representation.

[0043] NeRF is a machine learning (ML) model (e.g., neural network) that takes 2D images representing a scene as an input and interpolates between the images to render a complete 3D scene. NeRF models can generate photorealistic rendering of complex 3DQualcomm Ref. 2503354WO11scenes from a sparse set of 2D images. NeRF can overcome some limitations of traditional 3DR techniques using point clouds and mesh-based rendering that often present fine detail, soft shadows, and view-dependent effects such as specular highlights. NeRF can generate a 3D scene representation by encoding volumetric density and color in a fully- connected neural network, which is queried using 3D coordinates and viewing directions to allow the model to learn a continuous, differentiable function that captures both geometry and appearance. NeRF can provide high fidelity output, but the computation complexity- of such models can limit its applicability. For example, training a single NeRF model can take hours to days. Further, rendering novel views using a NeRF model can also be slow due to the render function performing thousands of queries to the neural network per pixel. Despite optimizations such as fast NeRF variants, the reliance on dense sampling and per-pixel computation consumes too much computation power for broad use.

[0044] 3D Gaussian splatting is another technique for representing and rendering volumetric scenes using 3D information. 3D Gaussian splatting involves the use of a set of 3D Gaussian elements (or primitives) that are each defined by a position in space, a covariance matrix (e.g., which determines shape and orientation), color, opacity, and in some cases other parameters. A 3D Gaussian element is a volume that can be represented by sparse information, such as in approximately 300 bits. In the context of 3D Gaussian splatting, a non-limiting example of a primitive is a unit sphere with a covariance matrix. As noted, each Gaussian element can be described by a number of parameters, which can include position (e.g., an X, Y. Z coordinate of the location), covariance (e.g., how much the Gaussian element is stretched / scaled along the three orthogonal directions, namely the eigenvectors of the covariance), color (e.g., red, green, blue and its variability depending on the viewing angle using Spherical Harmonic coefficients), and a level of transparency (referred to as opacity). For example, the covariance matrix represents the transformation, scale, and orientation of the primitive in the 3D coordinate system. The covariance matrix may transform the unit sphere into 3D ellipsoids (e.g., 3D Gaussians or splats) that represent a volume within the 3D coordinate system. 3D Gaussian elements represent fuzzy volumes that smoothly blend into each other, differently than solid geometric approaches of NeRF and conventional 3D rendering techniques. 3D Gaussian splatting can be used to generate or reconstruct a 3D scene by projecting the GaussianQualcomm Ref. 2503354WO12elements onto a 2D space (e.g., rendering the Gaussian elements to a 2D image), using Gaussian functions for each splat. 3D Gaussian splatting allows for real-time rendering of photorealistic scenes learned from small samples of images.

[0045] The process for 3DGS can begin by generating or estimating (e.g., using a structure from motion (SfM) or other technique) a 3D point cloud from a set of 2D images of the scene. For instance, the SfM technique may be performed by using the COLMAP algorithm. A Gaussian splat can then be added for each point in the point cloud (e.g., each point of the point cloud can be converted into a Gaussian splat). Such conversion of points into Gaussian splats is sufficient to provide rasterization for the scene. However, only position and color can be inferred from the SfM data. Training is performed to learn a representation that yields high quality results.

[0046] The training procedure for 3DGS can use a stochastic gradient descent technique (e.g., similar to a neural network training using backpropagation, but without the layers of the neural network). During the training, the Gaussian splats are rasterized to an image (e.g., corresponding to a view of the scene) using differentiable Gaussian rasterization (e.g., using a differentiable rendering algorithm). A loss may then be determined based on a difference between the rasterized image of the scene and a ground truth image of the scene. The parameters of the Gaussian splats are adjusted according to the loss. Automated densification and pruning of the Gaussian splats are applied. The training allows for the 3D scene to be generated with photorealistic details.

[0047] Since 3DGS can be rendered quickly due to a differentiable rendering algorithm and can generate arbitrary 2D projections, 3DGS has gained a significant amount of attention, which has pushed NeRF to the sidelines. 3DGS relies on a 3D point cloud generated from SfM using the COLMAP algorithm followed by training to control the density of the Gaussian splats and to tune the parameters (e.g., including location, orientation, extent (scale), opacity, and view-dependent color in terms of spherical harmonics). After the training session has completed, an explicit representation of the scene is obtained that can be rendered for arbitrary viewpoints of the scene.

[0048] The quality of this explicit representation may rely on the initial point cloud and the 2D ground truth images captured from the scene to tune the Gaussian splats. InQualcomm Ref. 2503354WO13cases where the point cloud is reliable (e.g.. and, as such, the 3DGS initialization is reliable) and there is sufficient variability in the collected 2D images of the scene, 3DGS should produce a reasonable 3D model of the scene given a sufficient number of training iterations.

[0049] A complex 3D model representing a scene using Gaussian splatting uses a large number of Gaussian elements (e.g., 106- 107). Each Gaussian element also includes parameters such as position, scale, orientation, opacity, and color, which are potentially used for multiple splat operations per frame. Gaussian elements may also include features such as linking and identification of scores, which further increases memory consumption. Modifying the Gaussian model to update the positions, opacities, or other attributes of the Gaussian element primitives includes computational processes such as reading and writing large volumes of parameter data and can be a computationally expensive process. Rendering images using 3D Gaussian elements splatting can also lead to large memory consumption as scene complexity increases. Gaussian splatting can thus increase the memory footprint of a system into the gigabyte range. On resource-constrained devices such as mobile GPUs, XR headsets, etc., the memory bus may become constrained, thermal issues due to memory operation can throttle performance, or memory may overflow and crash.

[0050] In one or more aspects, systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as “systems and techniques’") are described herein that provide a novel view synthesis (NVS) system with real time object replacement. In some cases, the NVS system uses techniques such as NeRF or 3D Gaussian splatting to render a frame from any position within (e.g., with any view of) the 3D representation or 3D model. Not all objects within the 3D representation have equal priority and multiple objects are common. In some aspects, objects within the scene can be replaced by a templated object that represents a common object or feature. For example, in the case of an NVS system used in connection with driving a vehicle (e.g., for an autonomous driving system), the NVS system may identify common objects such as signs, traffic lights, vehicles, trees, people, and so forth.

[0051] In some aspects, the NVS system is configured to identify objects and replace the object with an object template that approximates the identified object with lessQualcomm Ref. 2503354WO14graphical fidelity. For example, a traffic light may be represented by 10,000 Gaussian elements in a 3D Gaussian splatting system. In such an example, using the NVS system, the traffic light may be replaced by an object template of that traffic light having 1,000 Gaussian elements. In this way, the NVS system can reduce features that are not as semantically important to the 3D representation to reduce memory, processing, and / or splatting operations. In some cases, the NVS system may determine a semantic importance and elect to not replace an object within the scene based on the semantic importance.

[0052] The NVS system may include various techniques to render the object template. In some cases, the Gaussian elements may be deleted from memory, and the object template may be rendered using a separate data structure. In another case, the Gaussian elements may be replaced by a replacement object template. A replacement object template includes features corresponding to the original object, such as scale, orientation, and so forth, to maintain a high-fidelity view of the 3D scene.

[0053] The systems and techniques described herein may be implemented by any type of system or device. Examples of systems that can be used to implement the systems and techniques described herein include a vehicle or a computing device or system of the vehicle, an XR device or system, a mobile device, a server device, or any other system, device, or apparatus (e.g., a computing device or a system or component of the computing device).

[0055] In the present discussion, a “Gaussian’’ refers to a graphical element of a Gaussian scene representation. Each Gaussian is specified by a set of parameters indicating characteristics, described further below, of the Gaussian in scene space. In some examples, Gaussians have a generally ellipsoidal 3D shape in the scene space, with a maximum opacity at a center position of the Gaussian. In some examples, the opacity diminishes following an equation similar to a Gaussian probability distribution.

[0056] In a particular aspect, the parameters specifying a set of Gaussians may include:• a position vector, pi, which indicates the center position of a Gaussian with index z;Qualcomm Ref. 2503354WO15• an opacity parameter, t, E [0,1], which indicates a maximum opacity of Gaussian;• orientation parameters, qi, which can include a quaternion defining the spatial orientation (rotation) of the Gaussian;• a scale vector, si, which defines a size and shape of the ellipsoid of the Gaussian;anda set of spherical harmonics (SH) coefficients, Hi, which can include a matrix with columns composed of 3D color vectors (in portions of this disclosure H, is referred to as an SH coefficient array).

[0057] In some implementations, not all of the parameters described above are used to specify a Gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacity parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a viewindependent color) rather than higher-order spherical harmonics that capture viewdependent color variations. Using a reduced parameter set can decrease memory usage and computational complexity associated with storing, updating, and rendering the Gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.

[0058] In some implementations, the Gaussian scene representation may use two-dimensional (2D) Gaussians rather than three-dimensional (3D) Gaussians. A 2D Gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D Gaussians, which distribute density throughout the complete volumetric ellipsoid, 2D Gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.Qualcomm Ref. 2503354WO16

[0059] A 2D Gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D Gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parameters (e g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D Gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D Gaussian primitive may have fewer parameters than a corresponding 3D Gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D Gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.

[0060] Using 2D Gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D Gaussian primitives can be more tightly aligned to object surfaces than 3D Gaussians. Additionally, 2D Gaussians may provide more accurate geometry representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.

[0061] In some implementations, the Gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each Gaussian of the Gaussian scene representation can be used to represent a triangular face of a mesh, enabling conversion between Gaussian representations and mesh representations. This approach provides a one-to-one mapping between Gaussians and triangular mesh faces that is differentiable, enabling the Gaussian scene representation to be modified in a manner similar to modifying a mesh.

[0062] To convert a 2D Gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D Gaussian may be performed. The covariance matrix X can be decomposed as S = RART, where R is a matrix of eigenvectors (el, e2, e3) and A is a diagonal matrix of eigenvalues ( I, X2, X3). The two largest eigenvalues ( I, X2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (X3) corresponds to the normal direction. A triangular face can then be constructed using the mean of the Gaussian as a center point and defining triangle vertices along the tangentQualcomm Ref. 2503354WO17directions. For example, triangle vertices may be defined as v1 = p + α1e1, v2 = p + α2e2, and v3 = p - α1e1, where α1 and α2 are scale factors proportional to square roots of their respective eigenvalues λ1 and λ2 and p is the center point corresponding to the mean of the 2D Gaussian. This creates a triangle aligned with the Gaussian's orientation and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the Gaussian.

[0063] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the Gaussian. When the mesh is modified, the Gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the Gaussians when the mesh is animated or otherwise modified. During rendering, the Gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of Gaussians of the Gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between Gaussians and triangular faces.

[0064] Throughout this disclosure, references to Gaussians, Gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D Gaussians, 2D Gaussians, mesh-integrated Gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the type of Gaussian representation used, as described herein.

[0065] The following elements may be determined based on the parameters above:• S is a 3x3 matrix similar to the covariance matrix in Gaussian probability distributions.• c is a 3D view-dependent color vector for each Gaussian, computed using the set of spherical harmonics coefficients of the Gaussian.

[0066] The three dimensions of the color vector c may correspond to three different color component values, such as a red value, a green value, and a blue value, that specify a color for the Gaussian. The color vector c is view-dependent in the sense that the Gaussian may appear to have different colors depending on a position from which theQualcomm Ref. 2503354WO18Gaussian is viewed, e.g., due to reflections. The view-dependent color vector c for the Gaussian may be derived from a camera position f, the center position p, of the Gaussian, and the SH coefficient array H, of the Gaussian. The camera position f and the center position p, may each be a vector of spherical coordinates (θ, φ). Thus, a viewing direction vector comprising a θ and a φ coordinate value that defines a viewing direction from the camera position f to the center position p, may be described as (f - pi). Therefore, the view-dependent color vector c for a Gaussian i having central position p, from camera position f may be written as ci(f - pi, Hi) or simply c(f - p, H).

[0067] In certain 3D Gaussian Splatting (3DGS) processes, H, includes up to 16 spherical harmonics coefficients per color vector component (e.g., 16 spherical harmonics components for red, 16 spherical harmonics components for green, and 16 spherical harmonics components for blue). A renderer may calculate a value of a color component in the view-dependent color vector c based on a linear combination LC of a set of terms Ti... Ymaxscaled by the spherical harmonics coefficients in H, for the color component. That is, the renderer may calculate the linear combination LCij for Gaussian i and color component j as:

[0054] Equation (1) may be rewritten as follows:Ci = H; y, (2)

[0055] etc., are spherical harmonics coefficients for Gaussian i in H, for the color component j. Each of the terms in vector y may be defined by the following spherical harmonics equation:

[0056] In equation (3) above, the values (θ, φ) specify the viewing direction vector derived from f and pi. The values I and m are the degree and order of the spherical harmonics equation, where - I <m < I. Thus, in an example where there are 16 spherical harmonics coefficients, the maximum value of I is 3, and when I is 0, m is 0; when I is 1,Qualcomm Ref. 2503354WO19m is 1, 0, -1; when I is 2, m is 2, 1, 0, -1, -2. and so on. Thus, Y₀⁰ may correspond to l being 0 and m being 0, Y₁¹ may correspond to l being 1 and m being 1, Y₁⁰ may correspond to l being 1 and m being 0, Y₁⁻¹ may correspond to l being 1 and m being -1, Y₂² may correspond to l being 2 and m being 2, and so on. With respect to conventional RGB values, the renderer may calculate the value of a color component in the view-dependent color vector c by applying a sigmoid function to LC and multiplying by a dynamic range scaling factor, such as 255.

[0068] To generate a rendered image based on a set of Gaussians, a renderer may determine color values of pixels in the rendered image. A rendered image is a type of digital image that is composed of a grid of individual pixels, each with its own color vector. Each pixel is associated with a camera ray extending from the pixel through a virtual aperture of a virtual camera. In conventional 3DGS rendering, for each pixel of the rendered image, the renderer may identify Gaussians that intersect the camera ray associated with the pixel and sort the identified Gaussians based on a depth from the virtual camera. For each of the identified Gaussians, the renderer calculates an opacity value for a location within the Gaussian based on the position vector of the Gaussian, the maximum opacity of the Gaussian, the orientation vector of the Gaussian, and the scale vector of the Gaussian. The renderer calculates the opacity value (α) for 3D location x along the camera ray within a Gaussian i (i.e., αi(x)) as follows:t( (x- Pi)T[S(q Si)]-1(x-pi)\αi(x) = τi exp{ - ½ (x-pi)ᵀ[Σ(qi, si)]⁻¹(x-pi) }, i = 1,2, …, N, (4)

[0069] In equation (4) and throughout this disclosure, the exp function is e' where e is Euler’s number, (,.)Tindicates matrix transposition, [... ]-1indicates a matrix inversion, and N indicates the total quantify of identified Gaussians that intersect the camera ray. In general terms, equation (4) scales the maximum opacify’ value tt) of the Gaussian based on a Gaussian distribution function according to a distance of the location x from a center pi of the Gaussian. The location x may be defined as a vector of coordinates, e.g., spherical coordinates, Cartesian coordinates, etc.

[0070] When processing a location x within a Gaussian, the Tenderer also calculates a view-dependent color vector c for the Gaussian. The view -dependent color vector c may include color component values that define a color for the Gaussian, such as red, green.Qualcomm Ref. 2503354WO20and blue values. Considering a virtual camera with a focal point (i.e.. virtual aperture) at position f, the Tenderer may calculate an alpha-scaled color vector r, for Gaussian i. The alpha-scaled color vector r, specifies a contribution of a location x within the Gaussian i to a color of a pixel. The alpha-scaled color vector may be calculated using the following formula:ri(x,f) = αi(x) c(f - pi, Hi), i = 1,2, …, N. (5)

[0071] Thus, equation (5) determines an alpha-scaled color vector r for a location x viewed from position f by scaling the view-dependent color vector c of a Gaussian i that contains the location x based on the previously determined opacity value for the location x (i.e., αi(x)). The view-dependent color vector c may be determined using a spherical harmonics equation based on the SH coefficient matrix Hi of the Gaussian and the differences between coordinates the center position of the Gaussian (f - pi) as described above. The view-dependent color vector of a Gaussian may be dependent on the viewing position f because of the directions of light sources, which can cause shadows, reflections, and other optical effects.

[0072] Depending on the size and orientation of a Gaussian and a discretization of the 3D space into discrete locations, there may be multiple locations along a camera ray that intersect the same Gaussian. Hence, the Tenderer may calculate multiple opacity values and color vectors for the same Gaussian. The Tenderer may use these opacity values and color values in a rendering equation for determining a color vector of a pixel of a rendered image. However, for ease of explanation, this disclosure and the equations thereof assume that there is only one location per Gaussian.

[0073] An advantage of using the formula of equation (4) is that mapping Gaussians to the 2D rendered image can be approximated by a 2D Gaussian with a 2x2 covariance matrix that can be computed asΣ2D= J W Σ3DWTJT, (6)Qualcomm Ref. 2503354WO21where W is a matrix defined by the camera’s image-generation transformation, and J is the Jacobian matrix defined by an affine approximation of the projective camera transformation.

[0074] For ease of explanation, this disclosure omits some details about how rendering is discretized for image pixels, and considers that, for all Gaussians, there are opacity values a^m, n] and color vectors c m, n] defined for each image pixel with integer image coordinates [m, n]. The image coordinates of a pixel indicate a 2D location of the pixel within a rendered image. In other words, each pixel of a rendered image is identified by a set of coordinates m and n. Due to the transparency of Gaussians, before a scene is rendered, the Gaussians to be rendered are sorted according to their distance from the camera position f. Formally, a sorting sequence s[k] of indexes of Gaussians is determined according to increasing distance, such that1,2, -, N. (7)

[0075] In equation (7), the double straight brackets (i.e., ||... ||) indicate Euclidean distance.

[0076] Thus, the Tenderer processes locations (e.g., Gaussians) along the camera ray associated with a pixel in an order from closest to the virtual camera to further from the virtual camera. Hence, if the opacity values of locations within the Gaussians intersected by the camera ray associated with the pixel reach a level corresponding to complete opacity, later Gaussians that intersect the camera ray do not contribute to the color vector of the pixel. Hence, the Tenderer can skip processing of locations along the camera ray associated with the pixel that intersect the later Gaussians.

[0077] The Tenderer may compute a final color vector for a pixel based on the opacity values and the view-dependent color values for the locations along the camera ray associated with the pixel. For instance, after sorting the Gaussians based on depth, the Tenderer may compute a rendered color vector r for the pixel according to the following equation, which corresponds to the computer graphics technique of alpha-blending:Qualcomm Ref. 2503354WO22

[0078] In equation (8), N indicates a total quantity of graphics elements (e.g.. Gaussians) along the camera ray associated with the pixel at coordinates [m, n]. The term cs[k][m, n] indicates a color vector for a distance-sorted graphics element having index value k in a sorting order of the graphics elements. In an example where the graphics element is a Gaussian, cs[k][m,n] may be a view-dependent color vector for the Gaussian. The term aS[k] [m, n] indicates an opacity value for the distance-sorted graphics element having an index value k in the sorting order. Similarly, asp][m, n] indicates an opacity value for the distance-sorted graphics element having an index value I in the sorting order. Note that in equation (8), the term ∏l=1k-1(1 − αs[l][m, n]) is calculated individually for each of index values k=l to N so that the scaled color vector for the graphics element with index value k (cs[k] [m, n] crs[k] [m, n]) is further scaled based on the opacity values of graphics elements closer to the camera position than the graphics element with index value k. Thus, a Tenderer using equation (8) sorts the graphics elements based on depth from the camera position.

[0079] In some implementations, a Gaussian scene representation of an environment is not globally updated each time new input data is obtained. Instead, the Gaussian scene representation may be maintained using different update strategies, as described below.

[0080] Use of a Gaussian scene representation in a situation in which the Gaussian scene representation is periodically or occasionally updated to capture new scene information presents significant challenges due to the computational complexity of updating Gaussian scene representations. Aspects disclosed herein reduce the computational complexity associated with updating a Gaussian scene representation by using different update strategies to update the Gaussian scene representation in different situations. For example, one update strategy can include performing a full update of the Gaussian scene representation, and another update strategy is to perform a partial update of the Gaussian scene representation. As used herein, a “partial update” refers to updating a specified subset of the Gaussians of the Gaussian scene representation while holding other Gaussians of the Gaussian scene representation constant.

[0081] The available update strategies that can be selected among can include at least a full update, a partial update, and no update. In addition to selecting between no update,Qualcomm Ref. 2503354WO23a partial update, and a full update, in some cases, the processor(s) can select among several available partial update strategies.

[0082] In some implementations, the selection of which Gaussians are eligible for update during a partial update is driven by changes detected in input data, such as changes in pixel data associated with the environment.

[0083] The one or more processors are configured to perform pixel-to-Gaussian mapping of at least a portion of the input pixel data to identify Gaussians of the Gaussian scene representation to be updated based on the input pixel data. The specified subset of the Gaussians updated during a partial update are selected based, at least in part, on changes detected in the scene.

[0084] The processor(s) can identify a region of interest associated with differences between the input pixel data and the reference pixel data. In this example, the processor(s) can use mapping data to identify Gaussians of the Gaussian scene representation that are associated with the region of interest.

[0085] After identifying Gaussians to be updated, some implementations further determine which portions of the input data are used to update those Gaussians.

[0086] The one or more processors are configured to perform Gaussian-to-pixel mapping to identify pixels of the input pixel data to be used to update the Gaussian scene representation. Mapping data can be used to determine which pixels of the input pixel data are to be used as training data to update the selected Gaussians. The mapping data can be stored for subsequent reference.

[0087] In some implementations, the identification of Gaussians eligible for update is additionally or alternatively based on objects detected in the environment, rather than solely on pixel-level changes.

[0088] The one or more processors are configured to compare the object data to mapping data that identifies objects represented in the Gaussian scene representation to identify Gaussians of the Gaussian scene representation to be updated based on the input pixel data.Qualcomm Ref. 2503354WO24

[0089] Objects within the scene can be identified and saliency mapping can be used to associate one or more of the objects with Gaussians of the Gaussian scene representation. The region of interest used in the staged approach can be associated with a particular object or a portion of an object identified in the scene.

[0090] In some implementations, the selection among update strategies and the identification of Gaussians to be updated is influenced by operating conditions of a device or apparatus. The device condition(s) can include or indicate a battery level, a processor load, a bus load, a cache load, a memory usage, an available network bandwidth, a memory access latency, any combination thereof, and / or other condition(s). The update strategy for the Gaussian scene representation can be selected based on levels of available computing resources.

[0057] Various aspects of the present disclosure will be described with respect to the figures.

[0058] FIG. 1 is a block diagram illustrating an architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components that are used to capture and process images of scenes (e.g., an image of a scene 110). The image capture and processing system 100 can capture standalone images (or photographs) and / or can capture videos that include multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces a scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by an image sensor 130.

[0059] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for instance, the control mechanisms 120 may include one or more exposure control mechanisms 125 A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms besides those that areQualcomm Ref. 2503354WO25illustrated, such as control mechanisms controlling analog gain, flash, HDR. depth of field, and / or other image capture properties.

[0060] The focus control mechanism 125B of the control mechanisms 120 can obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to the image sensor 130 or farther from the image sensor 130 by actuating a motor or servo, thereby adjusting focus. In some cases, additional lenses may be included in the device 105A, such as one or more microlenses over each photodiode of the image sensor 130, which each bend the light received from the lens 115 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0061] The exposure control mechanism 125 A of the control mechanisms 120 can obtain an exposure setting. In some cases, the exposure control mechanism 125 A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control a size of the aperture (e.g., aperture size or f / stop), a duration of time for which the aperture is open (e.g.. exposure time or shutter speed), a sensitivity of the image sensor 130 (e.g., ISO speed or film speed), analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0062] The zoom control mechanism 125C of the control mechanisms 120 can obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control a focal length of an assembly of lens elements (lens assembly) that includes the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to one another. TheQualcomm Ref. 2503354WO26zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which can be lens 115 in some cases) that receives the light from the scene 110 first, with the light then passing through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses of equal or similar focal length (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0063] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures an amount of light that eventually corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and may thus measure light matching the color of the filter covering the photodiode. For instance, Bayer color filters include red color filters, blue color filters, and green color filters, with each pixel of the image generated based on red light data from at least one photodiode covered in a red color filter, blue light data from at least one photodiode covered in a blue color filter, and green light data from at least one photodiode covered in a green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also referred to as "emerald") color filters instead of or in addition to red, blue, and / or green color filters. Some image sensors may lack color filters altogether, and may instead use different photodiodes throughout the pixel array (in some cases vertically stacked). The different photodiodes throughout the pixel array can have different spectral sensitivity curves, therefore responding to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0064] In some cases, the image sensor 130 may alternately or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes, or portions of certain photodiodes, at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier to amplify the analog signals output by the photodiodes and / orQualcomm Ref. 2503354WO27an analog to digital converter (ADC) to convert the analog signals output of the photodiodes (and / or amplified by the analog gain amplifier) into digital signals. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 120 may be included instead or additionally in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0065] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor.

[0066] The host processor 152 can be a digital signal processor (DSP) and / or other type of processor. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip can also include one or more input / output ports (e.g., input / output (I / O) ports 156), central processing units (CPUs), graphics processing units (GPUs), broadband modems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 can include any suitable input / output ports or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI), such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output port. In one illustrative example, the host processor 152 can communicate with the image sensor 130 using an I2C port, and the ISP 154 can communicate with the image sensor 130 using an MIPI port.

[0067] The image processor 150 may perform a number of tasks, such as de-mosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF,Qualcomm Ref. 2503354WO28automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, receipt of inputs, managing outputs, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 / 1025, readonly memory (ROM) 145 / 1018, a cache 1012. system memory 1015, another storage device 1030, or some combination thereof.

[0068] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 can include a display screen, a keyboard, a keypad, a touchscreen, a trackpad, a touch-sensitive surface, a printer, any other output devices 1035, any other input devices 1045, or some combination thereof. In some cases, a caption may be input into the image processing device 105B through a physical keyboard or keypad of the I / O devices 160, or through a virtual keyboard or keypad of a touchscreen of the I / O devices 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the device 105B and one or more peripheral devices, over which the device 105B may receive data from the one or more peripheral devices and / or transmit data to the one or more peripheral devices. The I / O devices 160 may include one or more wireless transceivers that enable a wireless connection between the device 105B and one or more peripheral devices, over which the device 105B may receive data from the one or more peripheral devices and / or transmit data to the one or more peripheral devices. The peripheral devices may include any of the previously-discussed types of I / O devices 160 and may themselves be considered I / O devices 160 once they are coupled to the ports, jacks, wireless transceivers, or other wired and / or wireless connectors.

[0069] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105 A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105 A and the image processing device 105B may be coupled together, for example via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image capture device 105 A and the image processing device 105B may be disconnected from one another.Qualcomm Ref. 2503354WO29

[0070] As shown in FIG. 1, a vertical dashed line divides the image capture and processing system 100 of FIG. 1 into two portions that represent the image capture device 105 A and the image processing device 105B, respectively. The image capture device 105A includes the lens 115, control mechanisms 120, and the image sensor 130. The image processing device 105B includes the image processor 150 (including the ISP 154 and the host processor 152), the RAM 140, the ROM 145, and the I / O devices 160. In some cases, certain components illustrated in the image capture device 105 A, such as the ISP 154 and / or the host processor 152, may be included in the image capture device 105 A.

[0071] The image capture and processing system 100 can include an electronic device, such as a mobile or stationary telephone handset (e g., smartphone, cellular telephone, or the like), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 can include one or more wireless transceivers for wireless communications, such as cellular network communications, 802.11 Wi-Fi communications, wireless local area network (WLAN) communications, or some combination thereof. In some implementations, the image capture device 105 A and the image processing device 105B can be different devices. For instance, the image capture device 105 A can include a camera device and the image processing device 105B can include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0072] While the image capture and processing system 100 is shown to include certain components, one of ordinary skill will appreciate that the image capture and processing system 100 can include more components than those shown in FIG. 1. The components of the image capture and processing system 100 can include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs. CPUs, and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The software and / or firmware canQualcomm Ref. 2503354WO30include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image capture and processing system 100.

[0073] The host processor 152 can configure the image sensor 130 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interface). In one illustrative example, the host processor 152 can update exposure settings used by the image sensor 130 based on internal processing results of an exposure control algorithm from past image frames.

[0074] In some aspects, the host processor 152 can also dynamically configure the parameter settings of the internal pipelines or modules of the ISP 154 to match the settings of one or more input image frames from the image sensor 130 so that the image data is correctly processed by the ISP 154. Processing (or pipeline) blocks or modules of the ISP 154 can include modules for lens / sensor noise correction, de-mosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, among others. The settings of different modules of the ISP 154 can be configured by the host processor 152. Each module may include a large number of tunable parameter settings. Additionally, modules may be co-dependent as different modules may affect similar aspects of an image. For example, denoising and texture correction or enhancement may both affect high frequency aspects of an image. As a result, a large number of parameters are used by an ISP to generate a final image from a captured raw image.

[0075] In some cases, the image capture and processing system 100 may perform one or more of the image processing functionalities described above automatically. For instance, one or more of the control mechanisms 120 may be configured to perform autofocus operations, auto-exposure operations, and / or auto-white-balance operations. In some embodiments, an auto-focus functionality allows the image capture device 105 A to focus automatically prior to capturing the desired image. Various auto-focus technologies exist. For instance, active autofocus technologies determine a range between a camera and a subject of the image via a range sensor of the camera, typically by emitting infrared lasers or ultrasound signals and receiving reflections of those signals. In addition, passive auto-focus technologies use a camera’s own image sensor to focus the camera, and thusQualcomm Ref. 2503354WO31do not use additional sensors to be integrated into the camera. Passive AF techniques include Contrast Detection Auto Focus (CDAF), Phase Detection Auto Focus (PDAF), and in some cases hybrid systems that use both. The image capture and processing system 100 may be equipped with these or any additional type of auto-focus technology.

[0076] FIG. 2 is an illustration of scene 200 including 3D Gaussian elements that can be rendered from an arbitrary viewpoint in accordance with some aspects of the disclosure. In some aspects, the scene 200 includes a large volume of 3D Gaussian elements that each include a position, a covariance matrix, a color, and an opacity. The scene 200 may be referred to as a Gaussian model having high fidelity based on a large number (e.g., 107) of 3D Gaussian elements to ensure every viewpoint is represented accurately.

[0077] Based on an orientation of the camera (e.g., a viewpoint), the 3D Gaussian elements are splatted onto a 2D plane and presented to a user. 3D Gaussian splatting can be effectively combined with modem 3D scanning technologies and accelerates the capture-to-rendering pipeline for real-world scenes. For example, 3D Gaussian splatting can be used in conjunction with LiDAR sensors, structured light sensors, and depth sensors to visualize the scene 200 in 2D space from an arbitrary viewpoint. For example, with dense point cloud data or depth maps acquired from a LiDAR, 3D Gaussian splatting can initialize a set of Gaussian elements directly from real -world geometry and may avoid time-consuming training or reconstruction stages common in other techniques. 3D Gaussian splatting also represents both spatial and radiometric information to provide realistic rendering under varying lighting conditions and is well -suited for applications in virtual production, real-time VR and XR experiences, and other real-time applications.

[0078] FIG. 3A is an illustration of 3D Gaussian primitives within a scene 300 of a 3D coordinate system and rendered into 2D image planes based on camera orientation in accordance with some aspects of the disclosure. FIG. 3B is an illustration of projecting the 3D Gaussian elements in accordance with some aspects of the disclosure. FIG. 3C is an illustration of alpha-blending the 3D Gaussian elements to render a tile in connection with a volume grouping function in accordance with some aspects of the disclosure. FIGS.3A, 3B, and 3C will be discussed in conjunction to describe the projection of the 3D Gaussian elements and blending into a 2D image plane.Qualcomm Ref. 2503354WO32

[0079] In some aspects, the scene 300 includes a first Gaussian element 302, a second Gaussian element 304, and a third Gaussian element 306 that are represented in a 3D coordinate system. In particular, the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 include a position (e.g., location within the 3D coordinate system), a covariance matrix, a color, and an opacity’. The covariance matrix represents the spread and orientation of a multivariate distribution, and transforms a standard shape (e.g., a unit sphere) into a 3D ellipsoid that captures the scale and correlation of the data along different axes. 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other, which is different from other rendering techniques that represent solid geometry.

[0080] The scene 300 includes a first image plane 310 and a second image plane 320 in an arbitrarily different direction than the first image plane 310. In this example, the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are rendered onto tiles for purposes of illustration. In some aspects, a 2D image may be represented as tiles, which are groups of unit pixels (e.g., 2x2 pixels, 4x4 pixels, etc.). Tiles can be used to parallelize mathematical computations within a graphical processing unit (GPU) to reduce computational latency. In this case, the tiles in FIGS. 3 A, 3B, and 3C are 2x2 pixels for purposes of illustration.

[0081] In some aspects, rendering a 3D Gaussian splatting comprises various stages, such as a projection stage, a visibility stage, a rasterization stage, and an image composition stage, to render the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 onto a 2D image plane (e.g., the first image plane 310 or the second image plane 320). In the projection stage, each 3D Gaussian of the scene 300 is projected onto a 2D image plane based on intrinsic and extrinsic parameters associated with a camera (e.g.. position, depth of field, focus, etc.). For example, each of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are projected as ellipses into the first image plane 310. The first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are projected as differently shaped and oriented ellipses into a second image plane 320 based on a different view of their volumetric representations.Qualcomm Ref. 2503354WO33

[0082] The projection stage may also include computation reduction strategies such as adaptive density control to prune features that cannot be rendered into the first image plane 310. The projection stage can include other types of transformations that reduce complexity. Another example projection includes an Affine transformation represented by O(x) = Wa + P. where W is a 3x3 rotation matrix and P is a 1x3 translation vector, and produces an intermediate 3D representation. The affine transformation maps a 3D point from object space to camera space. A Ray Space transformation then transforms the intermediate 3D representation to align the rays parallel to a coordinate axis and simplify mathematical operations associated with the ray (e.g., integrations).

[0083] In another example stage, a visibility stage performs a visibility' check for each Gaussian to ensure that Gaussian elements that are visible from the camera viewpoint contribute to the final image. Gaussian elements that are determined to be occluded are excluded from rendering computations as they do not contribute to the rendered view. For example, if a scene includes 107Gaussian elements, and each Gaussian includes a covariance matrix, computation complexity can be reduced significantly if 50% of the Gaussian elements are removed.

[0084] The rasterization phase, which can also be referred to as a render function, is the process of rasterizing the colors of the visible 3D Gaussian elements onto the first image plane 310 and the second image plane 320. The render function accumulates color that follows a front-to-back compositing approach with closer 3D Gaussian elements being rendered before distant 3D Gaussian elements to ensure correct depth representation. For example, FIG. 3B illustrates the first image plane 310 as a planar 4x4 pixel plane and including tiles 311-314. As show n in FIG. 3B, the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are mapped into the 2D space of the tiles 311-314 based on the orientation of a viewpoint with overlapping volumes. During the render function, the color and opacity values of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 are accumulated in a manner that accounts for overlapping regions. For example, the accumulation follows a front-to-back compositing approach with closer Gaussian elements being rendered before distant Gaussian elements to ensure correct depth representation. In the example of the first image plane 310, the third Gaussian element 306 is closest to the viewpoint and may occlude the first Gaussian element 302 and theQualcomm Ref. 2503354WO34second Gaussian element 304 (e.g., based on the opacity of the third Gaussian element 306). The opacity of the third Gaussian element 306 may also affect the rendered color splatted onto a corresponding tile. For example, if the third Gaussian element 306 has no opacity, the resulting color of a corresponding tile may be strongly correlated with the color of the third Gaussian element 306 based on the rendering function. In another example, if the third Gaussian element 306 has significant opacity, the resulting color may be strongly correlated with other colors behind the third Gaussian element 306 (e.g., the first Gaussian element 302 and the second Gaussian element 304). The rendering of the complete scene (not shown) can include additional volumetric features such as specular highlights, shadows, and other volumetric effects based on volumetric effects associated with the 3D Gaussian primitives.

[0085] In contrast to conventional solid geometry rendering functions, 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other. For example, when a virtual camera observes the scene, the 3D Gaussian elements are projected into 2D screen space using standard graphics pipeline transformations, which results in 2D ellipses represented on a 2D image plane that can be efficiently composed onto the image plane. 3D Gaussian splatting can also capture fine detail with soft lighting effects to generate smooth transitions and prevent aliasing artifacts. Moreover, transparency and depth blending allow for semi-transparent effects, accurate specular highlighting, and accurate occlusions without sorting or complex data structures.

[0086] The rasterization phase can include other stages to further increase rendering efficiency. In one example, a replication and sorting operation can be performed. In a replication and sorting operation, 3D Gaussian elements can be mapped to tiles and then sorted, which can enable various features to reduce computation complexity. For example, depth can affect a rendering operation. Table 1 below illustrates an example mapping of the first Gaussian element 302, the second Gaussian element 304, and the third Gaussian element 306 into the tiles 311-314. That is, Table 1 identifies whether a 3D Gaussian element overlaps the tile and its corresponding depth (e.g., relative to a camera).Qualcomm Ref. 2503354WO35Table 1

[0087] In some aspects, the Gaussian elements are stored based on tile and splatting of each tile can be handled in parallel within a GPU. For example, each 3D Gaussian of a corresponding tile is splatted onto the tile as a 2D footprint and colors are alpha blended to approximate visibility and transparency. Table 2 illustrates a resulting sorted result and simplifies the splatting and blending operations by limiting contributing Gaussian elements based on the overlap of the corresponding tile.Table 2

[0088] FIG. 3C illustrates the alpha blending operation associated with tile 311. Because the tile 311 is 2x2 pixels, the value of four pixels (C1-C4) of tile 311 can be calculated in parallel based on the equations below:Qualcomm Ref. 2503354WO36Ci = ai’ci+ afc2(l- a )C2 = (X2 C1+ 0.2’02(1 - 02’)C3 = Cl3’ci+ (X3’C2(1- 013’)C4 = Ct-l'Cl 1 0u’C2(l- 04')

[0089] In this case, the area of the ellipse overlapping the pixel and the opacity (e.g., a) generates a high-fidelity representation of the 3D scene that includes shading and other volumetric effects.

[0090] 3D Gaussian splatting is an efficient rendering technique that is advantageous over other view synthesis rendering techniques because a neural network does not perform inference at every point in space along a ray. In some cases, the entire scene can be precomputed or optimized into a sparse set of Gaussian primitives to allow efficient rendering. 3D Gaussian splatting can reduce computational overhead and memory usage to allow modest processors to render complex scenes in real-time.

[0091] In contrast to conventional solid geometry rendering functions, 3D Gaussian elements encode fuzzy volumes that smoothly blend into each other. For example, when a virtual camera observes the scene, the 3D Gaussian elements are projected into 2D screen space using standard graphics pipeline transformations, which results in 2D ellipses represented on a 2D image plane that can be efficiently composed onto the image plane.

[0092] 3D Gaussian elements are splatted into a 2D image space and use alpha blending to approximate visibility and transparency. The 3D Gaussian splatting rendering process is similar to forward rasterization and is highly parallelizable, making it suitable for rendering on a GPU. In some cases, a GPU can render the 3D Gaussian splatting scene at full frame rates (e.g., 30 Hz) for interactive applications like VR, 3D visualization, and other media consumption.

[0093] In some aspects, a 3D Gaussian splatting rendering process may use a mapping table that establishes the relationship between pixels in image space and Gaussians in three-dimensional space. For example, a forward pass of the model generates a mapping table by analyzing the scene content and determining which Gaussians contribute to which pixels. In this case, each pixel can be represented by a list of Gaussians thatQualcomm Ref. 2503354WO37contribute to the pixel result (e.g., [{ pixel 1: [Gaussians 3. 5, and 6] [. pixel 2: [Gaussians 1, 2, 6, and 10]}). Each entry in the table links a pixel index to one or more Gaussian parameter sets that influence the appearance of that pixel when the scene is rendered or reconstructed. The mapping table provides a structured reference for downstream tasks such as rendering novel views, updating scene content, or performing selective transmission of Gaussian data.

[0094] A mapping table can have multiple representations, such as a one-to-one mapping, or a many-to-many mapping. In a one-to-one mapping, each pixel is mapped to one Gaussian, which can be used in specialized functions, such as learning. In a many-to-many mapping, many pixels can be mapped to many different Gaussians. For example, a many-to-many mapping can be identified as a single pixel mapped to a list of Gaussians (e.g., a list of Gaussians described above).

[0095] An image-to-latent mapping process may refine how pixel data in image space is translated into latent Gaussian parameters for downstream tasks such as selective rendering or model updates. For example, an image-to-latent mapping process begins by identifying an object or region of interest within an image, which yields a specific subset of pixel indices corresponding to that object. The system queries a previously derived pixel-to-Gaussian mapping table to retrieve the set of Gaussians linked to each pixel based on the pixel indices. For example, when the mapping is many-to-many, a single pixel may map to multiple Gaussians and, conversely, a single Gaussian may contribute to multiple pixels. This overlap is typical in complex scenes with transparency, soft edges, or splat blending. As a result, the initial list of retrieved Gaussians often includes duplicate entries, reflecting shared contributions across adjacent pixels. In contrast, in a one-to-many scenario, each pixel maps to multiple Gaussians, but each Gaussian may serve a single pixel, which still uses deduplication if multiple pixels select overlapping subsets of Gaussians due to partial object boundaries. In the one-to-one case, each pixel is linked uniquely to a single Gaussian with no overlap, so the retrieval step yields a direct, non-redundant latent parameter list. After the initial query, a deduplication operation ensures that unique Gaussian parameter sets are retained by removing any redundant mappings to the same Gaussian.Qualcomm Ref. 2503354WO38

[0096] FIG. 4A illustrates a training system 400 for a 3D Gaussian splatting model in accordance with some aspects of the disclosure. The training system 400 is configured to receive the 3D information 402 such as SfM points, a point cloud, etc. The 3D information 402 represents the positions of key features in the scene, such as depths to each point. In some aspects, the 3D information 402 can be a sparse reconstruction of the scene's geometry that is denved from multiple 2D images.

[0097] The 3D information 402 is provided to a Gaussian model engine 404 to initialize a set of 3D Gaussian elements, or a Gaussian model, forming a 3D scene. The Gaussian model engine 404 converts the 3D information 402 in various forms (e.g., points, vertices, etc.) into a corresponding 3D Gaussian element that includes a position in 3D space, an orientation and scale (e.g., the covariance matrix), and color and opacity (alpha). The Gaussian model engine 404 may also initialize other trainable properties, such as density fall-off or feature embeddings. The initialized 3D Gaussian elements collectively serve as the initial volumetric scene representation.

[0098] The Gaussian model engine 404 provides the Gaussian model to a projection engine 406 for a camera (e.g., viewpoint) to image plane projection. Each 3D Gaussian is projected into 2D space from a viewpoint of a camera 410 and simulates how a Gaussian would appear on the image plane based on the perspective of the camera 410 and intrinsics of the camera 410 (e.g., focal length, depth, exposure, etc.).

[0099] The projection engine 406 provides the projected images to a tile rasterizer 412 (e.g., a render function) to rasterize the 3D Gaussian elements onto a 2D grid (image tiles) based on differentiable rendering. Each Gaussian of a corresponding tile contributes a soft splat to the final image that is modulated by its opacity, shape, and distance to the camera. The intrinsics of the camera 410 can also affect the rendering process, such as focal length, that can affect the appearance of objects at different depths, shutter speed, etc. The contributions of all Gaussian elements are composited into a predicted image by the tile rasterizer 412.

[0100] The training engine 414 is configured to calculate a loss and then initiate an optimization flow to cause parameters of the Gaussian model to change. The training engine 414 compares the predicted image against ground- truth images from the trainingQualcomm Ref. 2503354WO39set (e.g., from corresponding camera viewpoints) to determine a loss. The training engine 414 includes a training loop that minimizes a composite loss L = (l-A)Li+ALo..«™, where Li loss is a pixel-wise absolute difference between the rendered and ground-truth images, LD-SSIM is a perceptual difference in image structure and appearance using a Dissimilarity Structural Similarity Index, and the scalar X that balances the pixel-wise absolute difference and the perceptual difference.

[0101] Based on the loss function, the training engine 414 initiates an optimization flow to optimize the Gaussian model. The training engine 414 computes gradients of the loss from the predicted image and then backpropagates the gradients through the tile rasterizer 412 to influence the parameters of each Gaussian (e.g., position, scale, orientation, color, and alpha). Each gradient indicates information to reduce the image error.

[0102] The tile rasterizer 412 also backpropagates errors to the projection engine 406 and an adaptive density control engine 416 to influence generation of the Gaussian model. The adaptive density control engine 416 uses the gradients to refine the spatial configuration of Gaussian elements and adjust the position and quantity of the Gaussian elements. For example, the adaptive density control engine 416 may add new Gaussian elements and remove redundant Gaussian elements. The adaptive density control engine 416 determines whether the Gaussian elements are too sparse, redundant, or irrelevant, and can control the Gaussian model engine 404 to prune or merge Gaussian elements that contribute little to the rendered image and duplicate or refine Gaussian elements in regions where more detail is needed. The adaptive density control engine 416 controls the details of the model, keeps the model compact, and improves rendering quality based on the training loop by adjusting the density' and distribution of Gaussian elements based on optimization feedback.

[0103] The resulting Gaussian model is a compressed, expressive, and differentiable 4D scene model that can enable real-time and photorealistic rendering based on a hybrid of point-based rendering and volumetric rendering. Gaussian models can be used in many different contexts, such as media presentation and other entertainment, where real-time rendering is necessary for the user experience.Qualcomm Ref. 2503354WO40

[0104] The training system 400 is more efficient than other 3D planar reconstruction techniques and novel view synthesis techniques. The training system 400 uses differentiable rendering techniques that adjust the parameters (e.g., position, shape, opacity) of each 3D Gaussian through gradient descent, which reduces the computation and time for scene reconstruction and convergence. Other techniques perform thousands of iterations to train a neural network over dense samples and can take many hours to converge.

[0105] FIG. 4B is a diagram illustrating an example of a process 420 for 3D Gaussian splatting. The process 420 utilizes SfM using the COLMAP algorithm 425 to generate a 3D point cloud for 3DGS initialization. During operation of the process 420, one or more image sensors 422 can obtain a plurality of 2D images 424 (or image frames), from a plurality of different views (e.g., different image sensor poses), of a scene. The 2D images 424 are used to train the 3DGS model. Once training of the 3DGS model is completed, the trained 3DGS model can be used to render generated images from other views (the same or different views as those in the 2D images). For example, the generated images may include a frontal view image 438 A, an above view image 438B, and a side view image 438C.

[0106] The SfM (e.g., which can use the COLMAP algorithm 425) can estimate a 3D point cloud from the plurality of 2D images 424 of the scene. The 3D point cloud can be used for initialization 426 of the Gaussian splats. For example, each point in the 3D point cloud can be converted into a Gaussian splat to produce a plurality of 3D Gaussian splats 428. The conversion of the points into Gaussian splats should be sufficient to provide rasterization for the scene. However, in some cases, only the position and color parameters for the Gaussian splats may be inferred from the SfM data. Training should be performed (e.g., on parameters, projections 430, and adaptive density7control 432 of the Gaussian splats according to the pose information 427 of the image sensor 422 (e.g., camera) when capturing the 2D images 424 to learn a representation that yields high quality results.

[0107] In some aspects, the training procedure for 3DGS can employ a stochastic gradient descent technique (e.g., which is similar to a neural network, but without the layers) based on the 2D images 424. During the training, the Gaussian splats can be rasterized to one or more images 436 (e.g., each corresponding to a different view of theQualcomm Ref. 2503354WO41scene depicted in the input 2D images 424) using a differentiable tile rasterizer 434 (e.g., using a differentiable rendering algorithm). A loss can then be determined based on a difference between the rasterized images 436 of the scene and ground truth images (e.g., the 2D images 424) of the scene. The parameters of the Gaussian splats may be adjusted according to the loss. Automated densification (e.g., adaptive density control 432) and pruning of the Gaussian splats can also be applied.

[0108] When the 2D images 424 have a sufficient variance in different views, the training can allow for the 3D scene to be generated from various different viewpoints with realistic details (e.g., generated images, such as the frontal view image 438A, the above view image 438B, and the side view image 438C). As shown in FIG. 4B, since the frontal view image 438A provides realistic results for the scene, but the above view image 438B and the side view image 438C do not provide realistic results for the scene, it is evident that the 2D images 424 did not contain a sufficient amount of views (e.g., the above view and the side view) of the scene.

[0109] Tables 3 and 4 below illustrate algorithms that can be implemented in connection with various aspects of the training system 400 of FIG. 4 A and / or the process 420 of FIG. 4B.Qualcomm Ref. 2503354WO42Table 3

[0110] Table 3 provides an example of a core training loop for optimizing a 3D Gaussian Splatting scene representation by initializing the 3D Gaussian elements’ properties (positions, covariances, colors, opacities) using SfM point data. In each iteration, a training view is sampled, and a differentiable rendering pass is performed. The loss is computed between the rendered and ground-truth image, and backpropagation is used to update all Gaussian parameters through an optimizer (e.g., Adam). On designated refinement steps, the model performs pruning, removing Gaussian elements with low opacity or excessively large spread, and densification, where Gaussian elements with high loss gradients are either cloned (to increase coverage in sparse areas) or split (to resolve over-dense regions).Table 4Qualcomm Ref. 2503354WO43

[0111] Table 4 describes a rasterization function (e.g., a render function) implemented by Table 3 and outlines how the 3D Gaussian elements are rasterized into a 2D image using a GPU-optimized pipeline. Gaussian elements outside the current view are culled for efficiency. The visible Gaussian elements are projected into screen space based on the current camera view, and a tiled representation of the image is created to enable parallel rendering. Each Gaussian is then duplicated across the tiles it affects, and globally sorted by rendering order. Tile ranges are computed, and an empty image canvas is initialized. For each tile and each pixel in the tile, a blend operation is performed in depth or alpha order to composite the final image.

[0112] FIGs. 4C and 4D illustrate a render system 440 for rendering images using a 3D Gaussian Model in accordance with some aspects of the disclosure. The render system440 includes sensors 441, such as a first sensor 441A, a second sensor 441B, and a Pth sensor 441P. At least one of the sensors 441 is an image sensor (e.g., the image sensors 422) configured to generate image data. One or more others of the sensors 441 can include or correspond to motion sensors, light sensors, device condition sensors, etc. The sensors441 are configured to generate sensor data 444. The render system 440 optionally includes a sensor controller 443 that is configured to control operation of the sensor 441 and / or to select a subset of the sensor data 442 to generate sensor data 444, which is used for further processing. If the sensor controller 443 is omitted, the sensors 441 can be externally controlled (e.g., responsive to user input), can operate according to a schedule, or can generate the sensor data 444 responsive to detecting a condition (e.g., in response to detecting movement).

[0113] In FIG. 4C, the render system 440 includes a pose estimator 445 that is configured to determine pose data (e.g., a position and / or orientation of one of the sensors 441 within a scene), which can be provided along with the sensor data 444 as input 446 to an image subspace selector, which can be configured to perform partial geometry and scene updates. In some cases, the sensor data 444 can include pose information (e.g., as data or metadata). In such cases, the pose estimator 445 is optional. If the pose estimator 445 is omitted, the sensor controller 443 (or the sensors 441 if the sensor controller 443 is also omitted) provides the input 446 including the sensor data 444 and corresponding pose data to the image subspace selector.Qualcomm Ref. 2503354WO44

[0114] In some aspects, the sensor data 442 (or sensor data 444) is used by a model initializer 447 to generate an initial Gaussian scene representation 448. For example, the model initializer 447 uses similar operations for the full update to generate initial parameters for the Gaussians 449 of the Gaussian scene representation 448. After initial generation of the Gaussian scene representation 448, the model initializer 447 can be bypassed and partial updates can be used to update a portion of the Gaussian scene representation 448.

[0115] In the example illustrated in FIG. 4C, the Gaussian scene representation 448 includes a plurality of Gaussians 449, such as: Gaussians 449 A through 449Q (where Q is an integer). Each Gaussian 449 is characterized by a plurality of parameters describing various properties of the Gaussian 449. For example, in FIG. 4C, the Gaussian 449A is characterized by an X-axis center position coordinate, Y-axis center position coordinate, and Z-axis center position coordinate. Together, the X-axis center position coordinate, the Y-axis center position coordinate, and the Z-axis center position coordinate indicate a center position of the Gaussian 449A and correspond to the position vector, p i. In FIG.4C, the Gaussian 449A includes an X-axis scale parameter, a Y-axis scale parameter, and a Z-axis scale parameter. Together the X-axis scale parameter, the Y -axis scale parameter, and the Z-axis scale parameter indicate a size and shape of the ellipsoid of the Gaussian 449A and correspond to the scale vector, s i. The Gaussian 449A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO), rotation (ql), rotation (q2), rotation (q3)) specified as quaternions C. Together the rotation parameters indicate a spatial orientation of the ellipsoid of the Gaussian 449A and correspond to the orientation parameters, q_i. The Gaussian 449A is also characterized by an opacity parameter, which indicates a maximum opacity of the Gaussian 449 A. The opacity parameter corresponds to the opacity parameter, t_i. The Gaussian 449A is also characterized by various color parameters (e.g. different color spaces such red green blue, oklab color space, hue saturation light, etc). The color parameters can be specified as a set of spherical harmonics (SH) coefficients. H i. Although FIG. 4C illustrates the Gaussian 449A as characterized by a set of fourteen parameters, in other examples, the Gaussians 449 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters can be used, depending on the order of the SH coefficients used to represent the colors of the Gaussians 449.Qualcomm Ref. 2503354WO45

[0116] In some aspects, an image subselector can generate partial updates and is configured to, based on the input 446, identify one or more patches of pixels (e.g., image patches) for a region of interest (ROI), or more than one ROI, associated with the sensor data 444 to generate ROI data. For example, the image subspace selector can identify one or more image patches (from one or more images) that correspond to an object that is in motion in the scene (e.g., based on motion data from a motion sensor, optical flow data from two or more images, etc.). In this example, the ROI can correspond to the image patch (or patches) associated with the object that is in motion. In other examples, the image subspace selector can identify one or more image patches (from one or more images) that correspond to pixels of input pixel data of the sensor data 442 or 444 that differ (by at least a threshold amount) from pixels of reference pixel data. In this example, the ROI can be defined by the location of the image patch (or patches) associated with the pixels that differ between the input pixel data and the reference pixel data. The ROI data identifies one or more sets of pixels of one or more images. As another example, the image subspace selector can identify one or more image patches (from one or more images) based on user input that designates or identifies a region of interest.

[0117] The render system 440 can include an update strategy selector (not shown) that is configured to select an update strategy based on strategy selection data. The strategy selection data can include the sensor data 442 or 444, the ROI data, other data, or a combination thereof. The update strategy selector is configured to provide a selection indication to the update engines, where the selection indication causes the update engines to initiate operations associated with the selected update strategy. For example, if the selection indication indicates that a full update strategy is selected, the update engines initiate the full update, and if the selection indication indicates that a partial update strategy is selected, the update engines initiate the partial update. If no update is selected, the update strategy selector can omit generation of the selection indication or can generate a '‘no update’’ selection indication.

[0118] When the update engines select the full update, the update engines cause a camera pose selector 450 (as indicated by off-page reference A) to begin operations associated with the full update. The pose selector 450 is configured to select a pose 452 from training data 470 that includes a set of training images (e.g., images 471A-471K) and pose data (e.g., poses 472A-472K) associated with each image 471 A-471K (where KQualcomm Ref. 2503354WO46is an integer greater than two). An image associated with the selected pose 452 is used as a ground-truth image 489 for loss calculation after a view associated with the pose 452 is rendered.

[0119] The pose 452 describes the position and orientation of a virtual camera 460 (including virtual cameras 460A through 460K) relative to the Gaussians 449 of the Gaussian scene representation 448 to be rendered as a view. Due to the pose 452 and intrinsic parameters (e.g., focal length, principal point, etc.) of the virtual camera 460, some of the Gaussians (e.g.. Gaussians 461, including Gaussians 461A through 461K) are visible to the virtual camera 460, and some of the Gaussians (e.g., Gaussians 463, including Gaussians 463A through Gaussians 463K) are not visible to the virtual camera 460. For example, for a pose 452A, a virtual camera 460A can see Gaussians 461A; however, Gaussians 463A are not visible to the virtual camera 460A in the pose 452A. As another example, for a pose 452K, a virtual camera 460K can see Gaussians 461K; however, Gaussians 463K are not visible to the virtual camera 460K in the pose 452K.

[0120] To ensure that the full update updates each Gaussian 449 based on all relevant data, the full update generally entails rendering a view for every available pose (e.g., poses 472A through 472K) and comparing each rendered view to the ground-truth image (e.g., each of images 471 A to 47 IK) associated with the pose 472 used to render the view. Thus, performing the full update is similar to operations performed by the model initializer 447 to generate an initial version of the Gaussian scene representation 448 based on a set of images except that the full update is initialized from a preexisting Gaussian scene representation 448.

[0121] A projector 454 identifies the Gaussians 461 that are visible to the virtual camera 460 for a pose 452. The projector 454 performs Gaussian splatting, which entails projecting the visible Gaussians 461, which are three dimensional, onto a two-dimensional (2D) image plane 462 (including 2D image plane 462A through 2D image plane 462K) based on the pose 452 of the virtual camera 460 and the intrinsic parameters of the virtual camera 460. Each Gaussian 461 is represented as an ellipse 465 (including ellipses 465A through ellipses 465K) in the 2D image plane 462. The ellipses 465 can overlap one another and can have various sizes and orientations depending on parameters of the Gaussians 461 that are projected and the pose 452 of the virtual camera 460. ForQualcomm Ref. 2503354WO47example, the Gaussians 461 A that are visible to the virtual camera 460A in the pose 452A are projected onto a 2D image plane 462A as ellipses 465 A, and the Gaussians 46 IK that are visible to the virtual camera 460K in the pose 452K are projected onto a 2D image plane 462K as ellipses 465K.

[0122] Data 464 (including data 464A through data 464K) describing the ellipses 465 on the 2D image plane 462 is provided to a differentiable tile rasterizer 434. For example, data 464A describes the ellipses 465A associated with the pose 452A, and data 464K describes the ellipses 465K associated with the pose 452K. The differentiable tile rasterizer 434 is configured to perform operations to blend the ellipses 465 to generate pixel data representing a 2D image that would be captured by the virtual camera 460 for the particular selected pose 452. To generate the pixel data, the differentiable tile rasterizer 434 divides the 2D image plane 462 into tiles (represented in FIG. 3B by bold lines and different shading in the 2D image planes 462A and 462K of the differentiable tile rasterizer 434). For purposes of this description, each tile is treated as corresponding to a single pixel; however, the exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel. Additionally, to simplify the drawings, in FIG. 3B. the 2D image plane 462 includes some regions that are not associated with any Gaussian; however, it will generally be the case that each tile is associated with at least one Gaussian 449 to provide color information for the tile.

[0123] The differentiable tile rasterizer 434 obtains depth information 482 (including depth information 482A through depth information 482K) for the visible Gaussians 461 of the pose 452, where depth refers to the distance in the scene of a Gaussian from the virtual camera 460. For example, the depth information 482 can be calculated for each of the visible Gaussians 461 based on the location of the virtual camera 460 in the scene (as indicated by data describing the pose 452) and the location of the center of the Gaussian in the scene (e.g., as indicated by the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 of each Gaussian). For example, depth information 482A indicates distances in the scene between each of the Gaussians 461 A and the virtual camera 460 A, and depth information 482K indicates distances in the scene between each of the Gaussians 46 IK and the virtual camera 460K.Qualcomm Ref. 2503354WO48

[0124] The differentiable tile rasterizer 434 can perform tile-depth association 484 to, for each tile of the 2D image plane 462. sort the ellipses 465 that at least partially overlap the tile based on the depth information 482. Since the Gaussians 449 are generally not opaque, a first Gaussian that is more distant from the virtual camera 460 in the scene can be partially visible through a second Gaussian that is closer to the virtual camera 460. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 434 accounts for the colors of each Gaussian of a set of Gaussians that influence the particular pixel, the opacities of the Gaussians of the set, and the order of the Gaussians of the set as viewed from the virtual camera 460.

[0125] To parallelize generation of the pixel data, the differentiable tile rasterizer 434 can perform thread association 486 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources. Computing resources can include one or more cores of one or more central processing units (CPUs), one or more cores of one or more graphics processing units (GPU), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of Gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colors of the Gaussians, as determined based on the pose 452 of the virtual camera 460 and color parameters specified by spherical harmonic color coefficients, opacity of the Gaussians, and the depth information 482.

[0126] Blending the contributions of the Gaussians results in pixel data 481 representing an image (e g., a view) captured by the virtual camera 460 for the particular pose 452. The pixel data 481 is provided to a loss calculator 488, which is configured to compare the pixel data 481 to the ground-truth image 489 corresponding to the pose 452. The loss calculator 488 determines a loss value for each pixel (e.g., a photometric L2 loss) to quantify how the pixel data 481 differs from the ground-truth image 489. Gradient values 483, 485 (including gradient values 485A through 485K) based on the loss values are backpropagated through the differentiable tile rasterizer 434 and the projector 454 to estimate updated parameter values 487 for the visible Gaussians 461 of the pose 452.

[0127] The full update continues by iteratively repeating the processes of pose selection, projection, rasterization, loss calculation, and backpropagation of gradientsQualcomm Ref. 2503354WO49until all of the poses 472 have been processed. At one or more points during the full update, an adaptive density control 432 performs operations that change the number of Gaussians 449 of the Gaussian scene representation 448, such as pruning one or more Gaussians 449 (e g., the Gaussian 449B) or splitting one or more Gaussians 449. In some cases, a different Gaussian scene representation 448 exists after the full update than existed before the full update since a different number and arrangement of Gaussians 449 can be present, and any Gaussian 449 that is carried over can have different parameters. For example, in FIG. 3B, the Gaussian scene representation 448 is illustrated as including a first Gaussian 449A, a second Gaussian 449B, and one or more additional Gaussians up to a Qth Gaussian 449Q (where Q is an integer greater than 2). After the full update, the Gaussian scene representation 448 can include more than Q or fewer than Q Gaussians 449. Further, the full update is a feedback-based process since, during each iteration, the visible Gaussians 461 for a pose 452 are updated based on loss values indicating differences between an image (e.g., the pixel data 481) of a view rendered for the pose 452 and a reference image (e.g., the ground-truth image 489 associated with the pose 452).

[0128] In some cases, a Gaussian scene representation can be represented as a mesh Gaussian background to reduce computational volume. For example, Gaussian elements deemed in the background can be converted into corresponding mesh primitives (e.g., triangles) and will not affect scene fidelity by virtual of the mesh being within the background. As described above, each Gaussian can be characterized by a center position, scale, orientation, opacity, and color parameters. In some cases, a background Gaussian can be represented a triangle (e.g., in a mesh) and a computing system can generate a mesh representation from the Gaussian parameters for use during rendering of a background region.

[0129] In some aspects, the processor performs an eigen-decomposition of a covariance matrix associated with the Gaussian to obtain eigenvectors and eigenvalues. For example, a Gaussian has a mean (p) corresponding to the center point and a covariance (V) corresponding to the orientation and spread. The eigen decomposition of is RAR where R is corresponds to the eigenvectors | ei, e2, es ] and A is the diagonal (7.i, 7.2, >.3). The covariance matrix indicates an orientation and spread of the Gaussian, while the mean corresponds to a center point of the Gaussian. The eigenvectors define principalQualcomm Ref. 2503354WO50directions of the Gaussian and the eigenvalues can define magnitudes associated with the principal directions.

[0130] In some aspects, the processor identifies a surface plane associated with the Gaussian based on the eigen-decomposition. For example, two larger eigenvalues can be associated with tangent directions of the surface plane, and a smallest eigenvalue (e.g., ) can be associated with a normal direction of the surface plane. Based on these directions, the processor can generate triangle vertices using the mean of the Gaussian as a center and by offsetting from the center along one or more tangent directions according to a scale factor derived from the eigenvalues. For example, using the mean of the Gaussian and the triangle vertices along tangent directions (e.g., vl=p+aei, V2=p+ae2, V3=p-ae1), where a is a scale factor (e.g., proportional to square root of Xi and Ae). In some implementations, color values for the triangle vertices can be determined from color parameters of the Gaussian, such as the color parameters associated with the Gaussian primitive.

[0131] In some implementations, the generated triangle primitives are rendered in combination with Gaussian splatting operations. For example, Gaussians that remain in Gaussian form can be projected onto a two-dimensional image plane and represented as ellipses for blending during rendering. In this w ay, a rendering pipeline can include both Gaussian-based rendering and mesh-based rendering derived from selected Gaussians, including Gaussians associated with a background portion of the scene (e.g.. based on a field of view, occlusions, etc.). The hybrid Gaussian and mesh rendering technique can provide flexibility in how different portions of the scene are represented and rendered.

[0132] In some other aspects, the Gaussian scene representation can include 2D Gaussians instead of three-dimensional Gaussians for selected portions of the scene. For example, a 2D Gaussian can omit one volumetric dimension and use one dimension as a normal associated with the 2D Gaussian while preserving tangent-plane information for rendering. For example, such an arrangement can provide a geometrically simpler representation for surfaces that are well-approximated as locally planar.

[0133] FIG. 5A is a block diagram of a NVS system 500 that is configured to render a 3D representation of a scene in accordance with some aspects of the disclosure. ForQualcomm Ref. 2503354WO51example, the NVS system 500 may be configured to render 2D images generated by a NeRF model or 2D images using 3D Gaussian splatting. In some aspects, the NVS system 500 includes a templating mechanism to reduce scene complexity without sacrificing visual fidelity or degrading user experience.

[0134] In some aspects, the NVS system 500 includes an object detector 502 that receives a 3D model or image as input. The 3D model may be provided from a real-time sensor (e.g., a LiDAR of a vehicle) or a pre-processed scene representation (e.g., from a 3D Gaussian Splatting system), or a synthetic environment. The object detector 502 is configured to identify discrete objects within the scene and may include various objects in the environment such as trees, vehicles, furniture, or people. In some aspects, obj ect detector 502 includes an ML model that is trained to identify geometric, semantic, or learned features of an environment, and provide output object information that identifies a boundary (e.g.. a box) of the object and an object identifier. In some cases, the object detector 502 may also provide information such as scale and orientation.

[0135] The object template selection engine 504 receives the object information from the object detector 502 and matches each object to a simplified representation, or a template associated with a corresponding object. As a non-limiting example, the template object may reduce complexity by a factor of 10. The templates may be stored in an object template library 506 (including a group or library of templates) and include various features for searching and matching. For example, objects may have different types, such as different types of trees, different types of signs, and so forth. Object templates are predefined models that correspond to commonly encountered object types and may be used to significantly reduce the complexity of the 3D representation. The object template selection engine 504 uses the identified objects from the object detector 502 and then matches a corresponding template in the object template library 506. The object template selection engine 504 may insert the matching object templates into the rendered scene in place of the original obj ects while maintaining spatial alignment. In some cases, the obj ect template selection engine 504 may delete the identified object and maintain a separate data structure for object templates. The object templates may significantly reduce complex operations by reducing geometry and complex data representations such as higher dimensional covariance matrices.Qualcomm Ref. 2503354WO52

[0136] The object template library 506 is a curated collection of geometric and visual models designed for substitution into synthesized views of a 3D representation. Each template is a lightweight representation of a known object type and may consist of simplified geometry, precomputed shading hints, or low-parameter splats. The object templates may not require dynamic features like transparency or view-dependent shading and may reduce computational overhead when rendered. In some aspects, the object template library 506 may interface with the object template selection engine 504 and support both retrieval (e.g., rendering) and potential updates (e.g., continual learning or tuning). In some cases, object templates in the object template library 506 may also include dynamic parameters to allow modifications to a parameter (e.g., height, width, pitch, etc.) to create volumetrically accurate representations of the replaced objects within the scene.

[0137] The NVS system 500 includes a primitive detection engine 507 that is configured to identify primitives in the 3D representation. For example, the primitive detection engine 507 matches the individual elements or features within the template to corresponding elements (e.g., Gaussian primitives) in the 3D representation.

[0138] In some aspects, a feature detection engine 508 is provided to detect alignment information between a normalized object template stored in the object template library 506 and the selected elements of features in the 3D representation. In an example, the feature detection engine 508 is configured to identify distinct features of both the 3D representation and the selected template Gaussian model. For example, the feature detection engine may analyze geometric attributes such as keypoints, edges, or landmark regions corresponding to physical parts (e.g., hands, eyes, or bounding shapes in a human model) that serve as anchors for alignment with respect to elements in the 3D representation. The feature detection engine 508 may further assign priority or confidence scores to the detected features to support ordering, reordering, or partial updates.

[0139] In some cases, the Gaussian selection engine 508 may use a pixel-to-Gaussian mapping to determine which Gaussians contribute to alignment feature vectors. Instead of processing the entire set of Gaussians for an updated scene, the feature detection engine 508 may build a map that links pixels in the image space to Gaussians that are relevant for computing alignment or pose information. This mapping can support one-to-one orQualcomm Ref. 2503354WO53many-to-many relationships and allows the feature detection engine 508 to select the necessary subset of Gaussians with parameters (e.g., position, orientation, or weighting) for calculating updated alignment vectors using the feature alignment engine 510.

[0140] A feature alignment engine 510 is configured to receive the identified features and determine an alignment information dataset of a selected object template. For example, an alignment dataset may include translation vectors, rotation angles, and scale factors to align and transform features of the selected object templates. Non-limiting examples of the feature alignment engine 510 include an Umeyama algorithm and an iterative closest point (ICP) algorithm. Umeyama uses a closed-form least-squares method for estimating the optimal similarity transformation (e.g., rotation, translation, and scaling) between two sets of corresponding points in 2D or 3D. Alternatively, the feature detection engine 508 may use an ICP method that repeatedly refines the alignment by minimizing distances between the template and scene point sets through iterative matching and transformation updates.

[0141] In some aspects, the feature alignment engine 510 may also scale the object template based on translating position, orientation, and scale. For example, the object template may be positioned and represented by Gaussian elements that are translated in 3D. To maintain geometric fidelity, each Gaussian element is scaled by a corresponding scale factor to shape the volumes of the object template and maintain visual fidelity.

[0142] In some aspects, feature detection engine 508 and the feature alignment engine 510 may occur at different times of the render pipeline. In some aspects, the feature detection engine 508 and the feature alignment engine 510 may occur when primitives are being rendered onto an image plane, for example, to replace pixels. In other aspects, the feature detection engine 508 and the feature alignment engine 510 may occur by replacing elements within the 3D representation before rendering into the image plane.

[0143] In some aspects, the render engine 514 receives a 3D representation including the original geometry and substituted object templates. The render engine 514 synthesizes the final image output using techniques consistent with real-time rendering, image-based rendering, or 3D Gaussian Splatting. By relying on the substituted templates, the engine bypasses the need to process high-complexity splats for substituted objects, leading to aQualcomm Ref. 2503354WO54significant reduction in memory use, shader operations, and blending calculations. The final output is a rendered image that maintains visual fidelity for the overall scene while being faster and more efficient to compute. This output can be used for display, recording, or downstream analytics, such as segmentation or viewpoint interpolation.

[0144] The NVS system 500 combines the detail and flexibility of full 3D models with the efficiency and regularity of object templates. The NVS system 500 can reduce computation by preprocessing scenes to detect and replace high-cost objects with simpler, template-based representations.

[0145] In some cases, the NVS system 500 may also be configured to perform template injection based on information from an image plane. For example, the object detector 502 may receive images from an image sensor and identify information within a corresponding 3D representation (e.g., using SfM). In one example, the object detector 502 may perform 2D object detection using computer vision or deep learning techniques (e.g., transformer-based models) to identify object instances and provide a bounding box, class labels, and / or segmentation masks. In some cases, the NVS system 500 may apply 2D-to-3D lifting techniques (e.g.. monocular depth estimation or pose estimation) to approximate 3D position, scale, or orientation of the objects.

[0146] The NV S system 500 may include a semantic engine 520 that is configured to identify a semantic importance of objects within the 3D representation. For example, the semantic engine 520 may include pretrained classifiers or scene graph parsers that recognize objects and assign weights based on task relevance. In some cases, user information (e.g., gaze detection) may also be used by the semantic engine 520 to identify semantic importance. Semantically important features within a scene may not be replaced to maintain visual fidelity. Background objects and objects at the periphery are suitable for replacement because the minimal loss of visual fidelity does not degrade the user experience and may be outside of the user’s fovea region.

[0147] FIG. 5B illustrates a conceptual illustration 550 of an alignment associated with a 3D Gaussian template in accordance with some aspects of the disclosure. In some aspects, Gaussian primitives may be an array or corresponding list of Gaussian primitives represented by a Gaussian Primitive type that maps various properties (e g., the GaussiansQualcomm Ref. 2503354WO55449A). For example, Table 5 below provides an example Gaussian Primitive object type definition in JavaScript Object Notation (JSON).Table 5

[0148] The 3DGS system may also store alignment information (e.g., an array of alignment items) with Gaussian templates to rotate, scale, and translate the Gaussian templates to replace Gaussian primitives. Table 6 illustrates an example type definition of an alignment item and includes an optional list of Gaussian primitives.Qualcomm Ref. 2503354WO56Table 6

[0149] In some aspects, a Gaussian template 570 is identified to replace an object 580 represented by a plurality of Gaussian primitives. For example, the object 580 is a car and an alignment engine 590 is configured to generate alignment information 592 to allow the Gaussian template 570 to accurately replace the object 580 based on scale, rotation, and translation of its features. For example, the primitives of the Gaussian template 570 can have less fidelity and may replace the primitives of the object 580 to reduce computations.

[0150] In some aspects, the alignment engine 590 is configured to identify a plurality of corresponding points between the Gaussian template 570 and the object 580 (e.g., using the feature detection engine 508). For example, the feature detection engine 508 may detect comers 571 and 574 of the Gaussian template 570 correspond to comers 581 and 584 of the object 580, and tires 572 and 573 of the Gaussian template 570 correspond to tires 582 and 583 of the object 580.

[0151] The alignment engine 590 can implement various techniques to perform an alignment that is applied to each primitive to replace the object 580 based on the identified points. A non-limiting technique is the Umeyama Algorithm, which can be used to estimate the similarity transformation between two sets of corresponding points in Rn, to compute an optimal transformation (rotation, translation, and uniform scale) that aligns a source point set to a destination point set using a least-squares technique. For example, given a point set for the Gaussian template 570 X = {xi,..., xn} and a point set for the object 580 Y = {yi,..., yn}, the Umeyama algorithm computes the centroids of X and Y, generates a covariance matrix between the centered sets, and then applies a singular value decomposition (SVD) to the covariance matrix to extract rotation USVT=SVD (XcYcT). A rotation matrix ℛ =UDVT, in which D corrects for rotation, is generated and scale s and translation t are generated. Based on the rotation matrix ℛ, scale s, and translation t, qi= s·ℛ xi+ t. where Xi are a plurality of points associated with the Gaussian template 570, and qi are the replacement primitives.

[0152] FIG. 6 illustrates a conceptual flowchart 600 of a process of an object template engine (e.g., the object template selection engine 504 of FIG. 5 A) for replacing features within a 3D representation in accordance with some aspects of the disclosure. In someQualcomm Ref. 2503354WO57aspects, the object template engine operates on a set of 3D Gaussian parameters to identify, align, and optionally replace or remove objects.

[0153] At block 602, the object template engine identifies Gaussian parameters associated with a target object in the 3D representation. For example, the object template engine may identify a static target object such as a sign or a vehicle. The object template engine identifies Gaussian properties associated with the target object, such as its position and orientation (e.g., Euler angles) within the 3D representation, a scale of the target object, color distributions, and so forth.

[0154] At block 604, the object template engine is configured to align the corresponding Gaussian properties of the identified object template (e.g., the substitute object) and generate corresponding alignment information. The object template engine may align the substitute object based on the Gaussian parameters of the target object from block 602. For example, alignment may include rotation and normalization of scale to ensure consistency of the object template engine with respect to the target object to maintain visual fidelity. The object template engine may also be configured to apply a transformation to at least one parameter of the object template to generate a scene with a higher correlation. For example, an object template of a vehicle that replaces another vehicle in the 3D representation may be changed to match the height and width of the replaced vehicle.

[0155] In some aspects, the alignment of object templates includes an optimization function that minimizes the spatial and feature-based discrepancies between the object in the scene and the corresponding object template. For example, the optimization function operates over transformation parameters (e.g., translation, rotation, and uniform or anisotropic scaling) to align the template’s frame to the centroid and orientation of the Gaussian elements of the original object. The optimization objective may incorporate terms such as the Euclidean distance between matched Gaussian centroids, covariance alignment, and similarity in intensity or color distribution. In some cases, the optimization can use weighted least squares to give higher priority to Gaussians with stronger opacity or larger spatial extent. The result of the optimization is a transformation matrix that aligns the object template with the original object instance and preserves the object template for consistent reuse across multiple views.Qualcomm Ref. 2503354WO58

[0156] At block 606. the object template engine is configured to remove the identified Gaussian elements of the target object from the 3D representation of the scene. For example, the target object is removed from the 3D representation, which may be a linked list of Gaussian elements. Other types of data structures can be used, such as a contiguous array of elements in memory, hashes, etc.

[0157] At block 608, the object template engine is configured to save the alignment information (e.g., from block 604) into a separate data structure. The alignment information includes information sufficient to render the template objects to have a corresponding pose. The alignment information includes a canonical pose of the template object, a transformation matrix, and any associated identifiers that link the aligned object to a template or classification label. In this case, the replacement template object is not inserted into the 3D representation directly, which can allow for conditional rendering of the template object. For example, if the camera of the 3D representation does not view the template object, compute associated with alignment and projection into the 3D representation can be reduced. The object template engine may save the alignment information and the replacement template object into a data structure such as a hash or dictionary, a separate list, and so forth.

[0158] FIG. 7 illustrates a conceptual flowchart 700 of a process of a render engine (e.g., the render engine 514 of FIG. 5A) for rendering an object template within a 3D representation in accordance with some aspects of the disclosure. For example, the conceptual flowchart 700 may be executed when a deleted object within a 3D representation of a scene is detected while rendering a frame. For example, while reading a list of Gaussian elements (e.g., the 3D representation), the render engine may read an element that explicitly indicates a template object is to be rendered.

[0159] At block 702, the render engine may read the alignment information from a corresponding memory location. In the case of a hash, the alignment information can be identified from a key, for example, in the list of Gaussian elements. The render engine may also read the object template at block 702.

[0160] At block 704, the render engine is configured to modify the object template based on the alignment information. The alignment information may include scale,Qualcomm Ref. 250335 WO59position, and orientation and may be applied to the object template in a transformation (e.g., a matrix transformation) to yield a substitute object. For example, the modification can include rotating, moving, scaling, and other transformations. The modification of the object template is to mutate the 3D properties (e.g., positions of 3D Gaussian elements) to be consistent with the scene.

[0161] In some aspects, at block 706, the render engine may be configured to render the object template based on a property associated with the object template. For example, a hat may have a configurable height that dynamically scales a single property. Modifying a single parameter, or even multiple parameters, can render spatially accurate substitute objects to maintain a high fidelity user experience. For example, reducing the visual fidelity can lead to distractions and reduce the quality of the user experience.

[0162] At block 708, the render engine is configured to select a render technique based on the view. For example, the NVS system 500 may include a Gaussian render engine 710 and a separate template render engine 712 to handle different rendering workflows. The Gaussian render engine 710 is configured to sort and alpha blend operations to accurately composite multiple overlapping Gaussians in the scene to preserve realistic shading and transparency effects. By contrast, the template render engine 712 can be configured to output a fixed color without executing sorting or alpha blending if the rendered tile is covered solely by the template Gaussians. For example, if a region is entirely replaced by a template object from the library, the template render engine 712 may skip sorting operations and alpha blending entirely to improve rendering efficiency. Bypassing prevents redundant blending computations and simplifies rendering.

[0163] FIG. 8 illustrates another conceptual flowchart 800 of a process of an object template engine (e.g., the object template selection engine 504 of FIG. 5A) for replacing features within a 3D representation in accordance with some aspects of the disclosure.

[0164] At block 802, the process identifies Gaussian parameters associated with a target object in the 3D representation. For example, the object template engine may identify a static target object such as a sign or a vehicle. The object template engine identifies Gaussian properties and associated with the target object, such as its positionQualcomm Ref. 2503354WO60and orientation (e.g., Euler angles) within the 3D representation, a scale of the target object, color distributions, and so forth.

[0165] At block 804, the object template engine aligns the corresponding Gaussian properties of the identified object template (e.g., the substitute object) and generates corresponding alignment information. The object template engine may align the substitute object based on the Gaussian parameters of the target object from block 802. For example, alignment may include rotation and normalization of scale to ensure consistency of the object template engine with respect to the target object to maintain visual fidelity. The object template engine may also be configured to apply a transformation to at least one parameter of the object template to generate a scene with a higher correlation. For example, an object template of a vehicle that replaces another vehicle in the 3D representation may be changed to match a height and width of the replaced vehicle. The alignment of object templates includes an optimization function described above that minimizes the spatial and feature-based discrepancies between the object in the scene and the corresponding object template.

[0166] At block 806. the object template engine is configured to apply the transformation to the object template to yield the substitute object and replace elements of the target object within the substitute object in the 3D representation (e.g., a linked list of 3D Gaussian elements). As noted above, the object template includes less detail, and the integrity’ of the data structure of the 3D representation can be maintained. For example, if the data structure is a linked list, the 3D representation can replace the elements in the list with the elements of the substitute object, which has fewer elements.

[0167] FIG. 9 is a flowchart illustrating a method of replacing identified objects with template objects in real time in accordance with some aspects of the disclosure. The process 900 can be performed by a computing device (or apparatus) or a component (e.g., one or more chipsets, an SoC, one or more processors such as one or more central processing units (CPUs), GPUs, digital signal processors (DSPs), neural processing units (NPUs), neural signal processors (NSPs), microcontrollers, ASICs, FPGAs, programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc., an ML system such as a neural network model, any combination thereof, and / or other component or system) of the computing device. TheQualcomm Ref. 2503354WO61operations of the process 900 may be implemented as software components that are executed and run on one or more processors (e.g., CPU, GPU, DSP, NPU or neural engine, SoC, the processor 1010, and / or other processor(s)).

[0168] At block 902. the computing device may identify a plurality of objects within a 3D representation of a scene. For example, the 3D representation of the scene comprises a plurality of Gaussian primitives, and each Gaussian primitive includes a covariance matrix defining a volume within the 3D representation of the scene.

[0169] At block 904, the computing device may determine that a first object of the plurality of objects corresponds to a first object template in a group of object templates (e.g., a library of object templates). The first object template may have less graphical complexity than the first object.

[0170] In some cases, a semantic score can be used to replace features based on value. For example, the computing device may determine a semantic score associated with an importance of a second object within the 3D representation and skip identification of the second object within the group of object templates based on the semantic score. For example, if the object template is associated with a fovea region (e.g., focal region), the object template may not be used to preserve visual fidelity.

[0171] In some aspects, the computing device may determine alignment information associated with the first object template. The alignment information includes orientation, position, and scale of the first object template and can be used to reduce complexify of the rendering process. In some cases, the computing device may store the alignment information associated with the first object template separately from the 3D representation. For example, the computing device may determine whether the first object template will be rendered into an image frame and render the first object template into corresponding elements associated with the 3D representation using the alignment information based on whether the first object template will be rendered.

[0172] In another aspect, the computing device may determine alignment information associated with the first object template, modify the first object template based on the alignment information into a substitute object, and replace the first object with the substitute object.Qualcomm Ref. 2503354WO62

[0173] The object template may be modified to match corresponding features in the 3D representation. For example, the computing device may identify at least one parameter associated with the first object template to modify based on a parameter of the first object and modify the at least one parameter of the first obj ect template to correspond to the parameter of the first object.

[0174] At block 906, the computing device may render the 3D representation based on the first object template. In some aspects, the computing device may include one or more displays configured to display the rendered 3D representation of the scene.

[0175] In some aspects, the computing device executing process 900 may additionally determine whether parameters of Gaussian primitives associated with the 3D representation should be updated prior to identifying replacement object templates. For example, the computing device may detect differences between a current scene input and a previously stored representation and select an update strategy for the Gaussian primitives comprising the scene, including a full update, a partial update, or no update. When a partial update strategy is selected, the computing device may identify a subset of Gaussian primitives associated with a region of interest corresponding to one or more objects identified (e.g., at block 902) and selectively update parameters of those Gaussian primitives while leaving other primitives unchanged. Mapping data linking scene regions, objects, or pixels to corresponding Gaussian primitives may be accessed, where the mapping data is organized into staged structures prioritized according to recency or frequency of access, to enable rapid retrieval of the Gaussian primitives associated with objects undergoing motion or lighting changes. The computing device may then update selected parameters such as primitive location, orientation, color, or opacify without performing a full rendering pass, loss calculation, or backpropagation across the entire scene representation.

[0176] In some aspects, the computing device may determine the update strategy' for Gaussian primitives by comparing input pixel data corresponding to a captured scene view with reference pixel data associated with a previously rendered image frame of the 3D representation. Difference regions may be identified based on metrics including region size, count, magnitude of pixel deviation, and image location, and these regions may define one or more regions of interest used to identify affected Gaussian primitives.Qualcomm Ref. 2503354WO63Pixel-to-Gaussian mapping data may be used to determine a first subset of Gaussian primitives corresponding to the changed pixels, and Gaussian-to-pixel mapping may further identify additional pixels influenced by those primitives. The computing device may then update parameters of the identified primitives using the expanded pixel set while maintaining unchanged primitives outside the affected regions. Because a subset of primitives are updated, the rendering pipeline used (e.g., at block 906) may omit full scene rendering and training operations, instead rendering a limited number of views sufficient to adjust the parameters of the affected primitives before object-template replacement is performed.

[0177] In certain aspects, the computing device may integrate device-condition inputs into the update-selection logic for the Gaussian primitives that define the 3D representation used in process 900. For example, differences between incoming pixel data and reference pixel data may be evaluated using metrics describing the number, spatial distribution, magnitude, and scene location of difference regions, and these metrics may be evaluated together with device-resource conditions such as processor utilization, thermal state, battery level, or available compute cycles of the GPU or NPU executing the rendering pipeline. Based on these inputs, the computing device may select among no update, partial update, or full update of Gaussian primitives before performing template substitution at block 904. When partial updates are selected, pixel-to-Gaussian mapping may identify the primitives corresponding to the changed image regions, and only targeted parameters of those primitives may be modified, including positional or orientation parameters associated with object motion and color or opacity parameters associated with lighting variation, thereby preserving the remainder of the scene representation unchanged while maintaining real-time rendering performance.

[0178] The computing device may determine which Gaussian primitives are eligible for update using gradient-based mapping techniques applied to rendered image views of the 3D representation. For example, saliency mapping or gradient analysis may be used to generate pixel-to-Gaussian and Gaussian-to-pixel associations that indicate which primitives contribute most strongly to differences between input pixel data and reference pixel data. The computing device may use these associations to determine a set of difference Gaussians corresponding to one or more regions of interest and evaluate a size metric based on the proportion of such difference Gaussians relative to the full set ofQualcomm Ref. 2503354WO64primitives defining the scene. Thresholds applied to this size metric may determine whether no update, partial update, or full update should be performed prior to objecttemplate substitution. The thresholds may be dynamically adjusted based on available computing resources or device state so that the system preferentially performs partial or no updates when operating under constrained compute conditions while still maintaining visual consistency in the rendered output.

[0179] In another aspect, the computing device may perform parameter refinement for selected Gaussian primitives identified as corresponding to scene changes prior to rendering the 3D representation (e.g., at block 906). For example, pixel-to-Gaussian mapping may identify primitives associated with pixels belonging to changed regions of interest within captured scene imagery, and Gaussian-to-pixel mapping may optionally identify additional training pixels influenced by those primitives. The computing device may update parameters of the selected primitives, including spatial parameters such as position and orientation to reflect object motion and appearance parameters such as color coefficients, spherical harmonics values, or opacity to reflect lighting changes within the scene. Because the updates are limited to primitives corresponding to the detected changes, the computing device may avoid performing full rendering, loss computation, or backpropagation across the entire Gaussian scene representation and instead apply localized feedforward parameter adjustments, thereby maintaining consistency between the updated primitives and the object templates used for real-time replacement within the rendering pipeline.

[0180] FIG. 10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology7. In particular, FIG. 10 illustrates an example of computing system 1000, which may be for example any computing device making up an internal computing system, a remote computing system, a camera, or any component thereof in which the components of the system are in communication with each other using connection 1005. Connection 1005 may be a physical connection using a bus, or a direct connection into processor 1010, such as in a chipset architecture. Connection 1005 may also be a virtual connection, networked connection, or logical connection.

[0181] In some embodiments, computing system 1000 is a distributed system in which the functions described in this disclosure may be distributed within a datacenter,Qualcomm Ref. 2503354WO65multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represent many such components each performing some or all of the function for which the component is described. In some embodiments, the components may be physical or virtual devices.

[0182] Example system 1000 includes at least one processing unit (CPU or processor) 1010 and connection 1005 that communicatively couples various system components including system memory71015, such as ROM 1020 and RAM 1025, to processor 1010. Computing system 1000 may include a cache 1012 of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1010.

[0183] Processor 1010 may include any general purpose processor and a hardware service or software service, such as services 1032, 1034, and 1036 stored in storage device 1030. configured to control processor 1010, as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory7controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0184] To enable user interaction, computing system 1000 includes an input device 1045, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, a motion input, speech, etc. Computing system 1000 may also include output device 1035, which may be one or more of a number of output mechanisms. In some instances, multimodal systems may enable a user to provide multiple types of input / output to communicate with computing system 1000.

[0185] Computing system 1000 may include communication interface 1040, which may generally govern and manage the user input and system output. The communication interface may perform or facilitate receipt and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those making use of an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an AppleTM LightningTM port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, 3G, 4G, 5G and / or other cellular data network wireless signalQualcomm Ref. 2503354WO66transfer, a BluetoothTM wireless signal transfer, a BluetoothTM low energy (BLE) wireless signal transfer, an IBEACONTM wireless signal transfer, a radio-frequency identification (RFID) wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, WLAN signal transfer, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communications interface 1040 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system 1000 based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based GPS, the Russian-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0186] Storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other types of computer readable media which may store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip / stripe. any other magnetic storage medium, flash memory, memristor memory’, any other solid-state memory-, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, RAM, staticQualcomm Ref. 2503354WO67RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory' (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory7(EEPROM), flash EPROM (FLASHEPROM), cache memory (e.g., Level 1 (LI) cache, Level 2 (L2) cache, Level 3 (L3) cache, Level 4 (L4) cache, Level 5 (L5) cache, or other (L#) cache), resistive random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0187] The storage device 1030 may include software services, servers, services, etc., that, when the code that defines such software is executed by the processor 1010, causes the system to perform a function. In some embodiments, a hardware service that performs a particular function may include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1010, connection 1005, output device 1035, etc., to carry out the function. The term '‘computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data may be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly' or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory' sharing, message passing, token passing, network transmission, or the like.

[0188] Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein, but those skilled in theQualcomm Ref. 2503354WO68art will recognize that the application is not limited thereto. Thus, while illustrative embodiments of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate embodiments, the methods may be performed in a different order than that described.

[0189] For clarity' of explanation, in some instances the present technology may be presented as including individual functional blocks including devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary' detail in order to avoid obscuring the embodiments.

[0190] Further, those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardw are or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.Qualcomm Ref. 2503354WO69

[0191] Individual embodiments may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0192] Processes and methods according to the above-described examples may be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions may include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used may be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memorw USB devices provided with non-volatile memory, networked storage devices, and so on.

[0193] In some embodiments the computer-readable storage devices, media, and memories may include a cable or wireless signal containing a bitstream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0194] Those of skill in the art will appreciate that information and signals may be represented using any of a variety' of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic weaves, magnetic fields or particles, optical fields or particles, or anyQualcomm Ref. 2503354WO70combination thereof, in some cases depending in part on the particular application, in part on the desired design, in part on the corresponding technology, etc.

[0195] The various illustrative logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also may be embodied in peripherals or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0196] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0197] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety' of devices such as general purpose computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions that, when executed, performs one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) such asQualcomm Ref. 2503354WO71synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory', magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that may be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0198] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry'. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0199] One of ordinary skill will appreciate that the less than (<) and greater than (>) symbols or terminology used herein may be replaced with less than or equal to (“<”) and greater than or equal to (“>”) symbols, respectively, without departing from the scope of this description.

[0200] Where components are described as being “configured to” perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g.. microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.Qualcomm Ref. 2503354WO72

[0201] The phrase "coupled to" or "‘communicatively coupled to" refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.

[0202] Claim language or other language reciting “at least one of’ a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language “at least one of’ a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0203] Claim language or other language reciting “at least one processor configured to,” “at least one processor being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “at least one processor configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y. and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting “at least one processor configured to: X, Y, and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0204] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performsQualcomm Ref. 2503354WO73the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g.. an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0205] Where reference is made to an entity (e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively ) to perform the functions. The one or more components of the entity may include one or more memories, one or more processors, one or more communication interfaces, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference is made to the entityperforming functions, the entity' may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0206] Illustrative aspects of the disclosure include:

[0207] Aspect 1. An apparatus for rendering one or more images, comprising: one or more memories; and one or more processors coupled to the one or more memories and configured to: identify a plurality of objects within a three-dimensional (3D) representation of a scene; determine that a first object of the plurality of objects corresponds to a first object template in a group of object templates; and render the 3D representation based on the first object template.Qualcomm Ref. 2503354WO74

[0208] Aspect 2. The apparatus of Aspect 1, wherein the first object template has less graphical complexity than the first object.

[0209] Aspect 3. The apparatus of any of Aspects 1 to 2, wherein the one or more processors are configured to: determine alignment information associated with the first object template; and store the alignment information associated with the first object template separately from the 3D representation.

[0210] Aspect 4. The apparatus of Aspect 3, wherein the alignment information includes orientation, position, and scale of the first object template.

[0211] Aspect 5. The apparatus of any of Aspects 3 to 4, wherein the one or more processors are configured to: determine whether the first object template will be rendered into an image frame; and render the first object template into corresponding elements associated with the 3D representation using the alignment information based on whether the first object template will be rendered.

[0212] Aspect 6. The apparatus of any of Aspects 1 to 5, wherein the one or more processors are configured to: determine alignment information associated with the first object template; modify the first object template based on the alignment information into a substitute object; and replace the first object with the substitute object.

[0213] Aspect 7. The apparatus of any of Aspects 1 to 6, wherein the one or more processors are configured to: determine a semantic score associated wi th an importance of a second object within the 3D representation; and skip identification of the second object within the group of object templates based on the semantic score.

[0214] Aspect 8. The apparatus of Aspect 7, wherein the identification of the second object is skipped when the semantic score is greater than a threshold.

[0215] Aspect 9. The apparatus of any of Aspects 1 to 8, wherein the one or more processors are configured to: identify at least one parameter associated with the first object template to modify based on a parameter of the first object; and modify the at least one parameter of the first obj ect template to correspond to the parameter of the first obj ect.Qualcomm Ref. 2503354WO75

[0216] Aspect 10. The apparatus of any of Aspects 1 to 9, wherein the 3D representation of the scene comprises a plurality of Gaussian primitives, and each Gaussian primitive includes a covariance matrix defining a volume within the 3D representation of the scene.

[0217] Aspect 11. The apparatus of any of Aspects 1 to 10, further comprising one or more displays configured to display the rendered 3D representation of the scene.

[0218] Aspect 12. The apparatus of any of Aspects 1 to 11. wherein the group of object templates includes a library of object templates.

[0219] Aspect 13. A method for rendering one or more images, comprising: identifying a plurality of objects within a three-dimensional (3D) representation of a scene; determining that a first obj ect of the plurality of obj ects corresponds to a first obj ect template in a group of object templates; and rendering the 3D representation based on the first object template.

[0220] Aspect 14. The method of Aspect 13, wherein the first object template has less graphical complexity than the first obj ect.

[0221] Aspect 15. The method of any of Aspects 13 to 14, further comprising: determining alignment information associated with the first object template; and storing the alignment information associated with the first object template separately from the 3D representation.

[0222] Aspect 16. The method of Aspect 15, wherein the alignment information includes orientation, position, and scale of the first object template.

[0223] Aspect 17. The method of any of Aspects 15 to 16, further comprising: determining whether the first object template will be rendered into an image frame; and rendering the first object template into corresponding elements associated with the 3D representation using the alignment information based on whether the first object template will be rendered.

[0224] Aspect 18. The method of any of Aspects 13 to 17, further comprising: determining alignment information associated with the first object template; modifyingQualcomm Ref. 2503354WO76the first object template based on the alignment information into a substitute object; and replacing the first object with the substitute object.

[0225] Aspect 19. The method of any of Aspects 13 to 18, further comprising: determining a semantic score associated with an importance of a second object within the 3D representation; and skipping identification of the second object within the group of object templates based on the semantic score.

[0226] Aspect 20. The method of Aspect 19, wherein the identification of the second object is skipped when the semantic score is greater than a threshold.

[0227] Aspect 21. The method of any of Aspects 13 to 20, further comprising: identifying at least one parameter associated with the first obj ect template to modify based on a parameter of the first object; and modifying the at least one parameter of the first object template to correspond to the parameter of the first object.

[0228] Aspect 22. The method of any of Aspects 13 to 21, wherein the 3D representation of the scene comprises a plurality of Gaussian primitives, and each Gaussian primitive includes a covariance matrix defining a volume within the 3D representation of the scene.

[0229] Aspect 23. The method of any of Aspects 13 to 22, wherein the group of obj ect templates includes a library of object templates.

[0230] Aspect 24. A non-transitoiy computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any of Aspects 13 to 23.

[0231] Aspect 25. An apparatus for rendering one or more images, comprising one or more means for performing operations according to any of Aspects 13 to 23.

Claims

Qualcomm Ref. 2503354WO77CLAIMSWhat is claimed is:

1. An apparatus for rendering one or more images, comprising:one or more memories; andone or more processors coupled to the one or more memories and configured to:identify a plurality of objects within a three-dimensional (3D) representation of a scene;determine that a first object of the plurality of objects corresponds to a first object template in a group of object templates; andrender the 3D representation based on the first object template.

2. The apparatus of claim 1, wherein the first object template has less graphical complexity than the first object.

3. The apparatus of claim 1, wherein the one or more processors are configured to:determine alignment information associated with the first object template; and store the alignment information associated with the first object template separately from the 3D representation.

4. The apparatus of claim 3, wherein the alignment information includes orientation, position, and scale of the first object template.

5. The apparatus of claim 3, wherein the one or more processors are configured to:determine whether the first object template will be rendered into an image frame: andrender the first object template into corresponding elements associated with the 3D representation using the alignment information based on whether the first object template will be rendered.

6. The apparatus of claim 1, wherein the one or more processors are configured to:determine alignment information associated with the first object template;Qualcomm Ref. 2503354WO78modify the first object template based on the alignment information into a substitute object; andreplace the first object with the substitute object.

7. The apparatus of claim 1, wherein the one or more processors are configured to:determine a semantic score associated with an importance of a second object within the 3D representation; andskip identification of the second object within the group of object templates based on the semantic score.

8. The apparatus of claim 7, wherein the identification of the second object is skipped when the semantic score of is greater than a threshold.

9. The apparatus of claim 1, wherein the one or more processors are configured to:identify at least one parameter associated with the first object template to modify based on a parameter of the first object; andmodify the at least one parameter of the first object template to correspond to the parameter of the first object.

10. The apparatus of claim 1, wherein the 3D representation of the scene comprises a plurality of Gaussian primitives, and each Gaussian primitive includes a covariance matrix defining a volume within the 3D representation of the scene.

11. The apparatus of claim 1, further comprising one or more displays configured to display the rendered 3D representation of the scene.

12. The apparatus of claim 1, wherein the group of object templates includes a library of object templates.

13. A method for rendering one or more images, comprising:identifying a plurality of objects within a three-dimensional (3D) representation of a scene;Qualcomm Ref. 2503354WO79determining that a first object of the plurality of objects corresponds to a first object template in a group of object templates; andrendering the 3D representation based on the first object template.

14. The method of claim 13, further comprising:determining alignment information associated with the first object template; and storing the alignment information associated with the first object template separately from the 3D representation.

15. The method of claim 14, further comprising:determining whether the first object template will be rendered into an image frame; andrendering the first object template into corresponding elements associated with the 3D representation using the alignment information based on whether the first object template will be rendered.

16. The method of claim 13, further comprising:determining alignment information associated with the first object template; modifying the first object template based on the alignment information into a substitute object; andreplacing the first object with the substitute object.

17. The method of claim 13. further comprising:determining a semantic score associated with an importance of a second object within the 3D representation; andskipping identification of the second object within the group of object templates based on the semantic score.

18. The method of claim 17, wherein the identification of the second obj ect is skipped when the semantic score of is greater than a threshold.

19. The method of claim 13, further comprising:Qualcomm Ref. 2503354WO80identifying at least one parameter associated with the first object template to modify based on a parameter of the first object; andmodifying the at least one parameter of the first object template to correspond to the parameter of the first object.

20. The method of claim 13. wherein the 3D representation of the scene comprises a plurality of Gaussian primitives, and each Gaussian primitive includes a covariance matrix defining a volume within the 3D representation of the scene.