Selective update of gaussian scene representation based on difference pixels and update targets

Selective update strategies for Gaussian scene representations address computational challenges by reducing resource consumption through partial and full updates based on input data and conditions, enhancing efficiency in applications like autonomous vehicle control.

WO2026240941A1PCT designated stage Publication Date: 2026-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/024817
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-27
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Gaussian scene representations require significant computing resources for updates due to frequent changes in the scene, particularly in applications like autonomous vehicle control, leading to computational challenges.

Method used

Implementing selective update strategies for Gaussian scene representations, including full, partial, and no updates, based on input pixel data, sensor data, and device conditions to reduce computational complexity.

Benefits of technology

Significantly reduces computing resources required for maintaining an up-to-date Gaussian scene representation by selectively updating only relevant portions, thereby optimizing resource usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026024817_19112026_PF_FP_ABST
    Figure US2026024817_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A device includes a memory configured to store a gaussian scene representation. The device also includes one or more processors configured to obtain input pixel data and, based at least in part on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, an update strategy for the gaussian scene representation. The reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.
Need to check novelty before this filing date? Find Prior Art

Description

QUALCOMM Ref. No. 2501497WO- 1 / 94 -SELECTIVE UPDATE OF GAUSSIAN SCENE REPRESENTATION BASED DIFFERENCE PIXELS AND UPDATE TARGETSI. Cross-Reference to Related Applications

[0001] The present application claims priority from the commonly owned U. S.Provisional Patent Application No. 63 / 804,910, filed May 13, 2025, U. S. Provisional Patent Application No. 63 / 831,815, filed June 27, 2025, U. S. Provisional Patent Application No. 63 / 825,138, filed June 17, 2025, and U. S. Provisional Patent Application No. 63 / 820,238, filed June 9, 2025, the contents of each of which are expressly incorporated herein by reference in their entirety.II. Field

[0002] The present disclosure is generally related to gaussian scene representation.III. Description of Related Art

[0003] Computer science uses various techniques to generate three dimensional (3D) models or four dimensional (4D) models (where time is a fourth dimension) of a scene based on images of the scene. One example of a scene model used in computer science is a gaussian scene representation, which uses primitive elements commonly referred to as "gaussians".

[0004] A process referred to as gaussian splatting can be used to generate views based on the gaussian scene representation. The views can include views of the scene from various perspectives, and in particular, the gaussian scene representation can be used to generate novel views of the scene (e.g., views of the scene that do not correspond directly to any particular image used to generate the gaussian scene representation). For example, a gaussian scene representation can be used to generate images or sequences of images (e.g., videos) that simulate how the scene would appear from a different vantage point, perspective, or lighting condition, gaussian scene representations can be used to generate imagery for computer vision, graphics, video games, extended reality visualization, 3D reconstruction, security and surveillance, medical imaging, computer assisted navigation, as well as other uses.QUALCOMM Ref. No. 2501497WO- 2 / 94 -

[0005] One challenge associated with using gaussian scene representations is that a large set of training data (e.g., images of the scene from a variety of viewpoints) are generally processed in order to generate an accurate gaussian scene representation. Gaussian scene representations also tend to be large (e.g., in terms of data storage used). As a result, use of a gaussian scene representation to generate a novel view consumes significant computing resources (e.g., memory utilization, processor utilization, power, etc.), and training the gaussian scene representation uses even more computer resources than view generation. Consumption of computing resources is particularly challenging in situations where a gaussian scene representation is updated frequently due to changes in the scene over time. To illustrate, in an autonomous vehicle control use case, a gaussian scene representation can be used for various computer vision tasks, such as obstacle avoidance, compliance with traffic laws, etc. In order to perform these tasks well, the gaussian scene representation may be updated frequently as the vehicle moves through the scene and / or as other vehicles, pedestrians, etc., move nearby.IV. Summary

[0006] According to one implementation of the present disclosure, a device includes a memory configured to store a gaussian scene representation. The device also includes one or more processors coupled to the memory and configured to obtain input pixel data. The one or more processors are configured to, based at least in part on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, an update strategy for the gaussian scene representation. The reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0007] According to one implementation of the present disclosure, a method includes obtaining input pixel data. The method also includes, based on differences between the input pixel data and reference pixel data, selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation, where the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0008] According to one implementation of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by one or moreQUALCOMM Ref. No. 2501497WO- 3 / 94 -processors, cause the one or more processors to obtain input pixel data. The instructions also cause the one or more processors to, based on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, an update strategy for a gaussian scene representation, where the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0009] According to one implementation of the present disclosure, an apparatus includes means for obtaining input pixel data. The apparatus also includes means for selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation based on differences between the input pixel data and reference pixel data, where the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.V. Brief Description of the Drawings

[0011] FIG. 1 is a block diagram of an illustrative system operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure.

[0012] FIG. 2 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0013] FIGS. 3A, 3B, 3C, and 3D, together, illustrate particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0014] FIG. 4 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501497WO- 4 / 94 -

[0015] FIG. 5 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0016] FIG. 6 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0017] FIG. 7 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0018] FIG. 8 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0019] FIG. 9 illustrates an example of an integrated circuit operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0020] FIG. 10 is a diagram of a mobile device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0021] FIG. 11 is a diagram of a mixed reality or augmented reality glasses device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0022] FIG. 12 is a diagram of a wearable electronic device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0023] FIG. 13 is a diagram of a voice-controlled speaker system operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501497WO- 5 / 94 -

[0024] FIG. 14 is a diagram of a camera operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0025] FIG. 15 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0026] FIG. 16 is a diagram of a first example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0027] FIG. 17 is a diagram of a second example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0028] FIG. 18 is a block diagram of a particular illustrative example of a device that is operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.VI. Detailed Description

[0029] Use of a gaussian scene representation in a situation in which the gaussian scene representation is periodically or occasionally updated to capture new scene information (e.g., new image(s)) presents significant challenges due to the computational complexity of updating gaussian scene representations. Aspects disclosed herein reduce the computational complexity associated with updating a gaussian scene representation by using different update strategies to update the gaussian scene representation in different situations. For example, one update strategy can include performing a full update of the gaussian scene representation, and another update strategy is to perform a partial update of the gaussian scene representation. As used herein, a "partial update” refers to updating a specified subset of the gaussians of the gaussian scene representation while holding other gaussians of the gaussian scene representation constant. The specified subset of the gaussians updated during a partial update are selected based, at least inQUALCOMM Ref. No. 2501497WO- 6 / 94 -part, on changes detected in the scene (e.g., motion, lighting changes, differences between an input image and a reference image, etc.).

[0030] Selectively updating only a portion of the gaussian scene representation when the partial update strategy is used uses significantly fewer computing resources than performing the full update. In contrast to conventional update systems, which always perform a full update when any update is performed, the disclosed aspects provide computationally efficient mechanisms for identifying the portions of the gaussian scene representation that should be updated based on available information. As a result, computing resources used to maintain an up-to-date gaussian scene representation can be significantly reduced. Thus, a technical advantage of the disclosed aspects is a reduction of computing resources used to update gaussian scene representations.

[0031] In a particular aspect, a system includes one or more processors configured to obtain input pixel data associated with a scene and to select, based on an update criterion, an update strategy for the gaussian scene representation based on the input pixel data. The update criterion can be based on the input pixel data itself. For example, the input pixel data can be analyzed to identify motion, lighting changes, areas with pixel changes, or other characteristics, where changes are relative to training data (e.g.. a training image) used to train the gaussian scene representation or synthetic data (e g., a synthetic image generated using the gaussian scene representation). In this example, the update criterion can include one or more thresholds, and results of the analysis of the input pixel data can be compared to the threshold(s) to select an update strategy’, from among a plurality' of available update strategies, to be used to update the gaussian scene representation based on the input pixel data.

[0032] As another example, the update condition can be based on sensor data. For example, the sensor data can be analyzed to identify motion, lighting changes, or other characteristics associated with updating the gaussian scene representation. In this example, the sensor data can include lighting data (e.g., to indicate lighting changes) or motion data (e.g., from an accelerometer or based on image analysis), as illustrative examples. In this example, the update criterion can include one or more thresholds, and results of the analysis of the sensor data can be compared to the threshold(s) to determine whether to update the gaussian scene representation based on the input pixelQUALCOMM Ref. No. 2501497WO- 7 / 94 -data. To illustrate, the sensor data can include a value of a lighting measurement from a lighting sensor, which can be compared to a reference lighting value to determine a lighting change value. The lighting change value can be compared to one or more thresholds to select an update strategy for the gaussian scene representation. A technical advantage of using sensor data to select the update strategy is that processing the sensor data uses fewer computing resources than processing pixel data and / or uses fewer computing resources than performing a full update of the gaussian scene representation; accordingly, using the sensor data to detect changes in the scene in order to select an update strategy can save significant computing resources.

[0033] As another example, the update condition can be based on a device condition. In this example, the device condition can be indicative of usage of a computational resource. To illustrate, the device condition can include or indicate a battery level, a processor load, a bus load, a cache load, a memory’ usage, an available network bandwidth, and / or a memory access latency, as non-limiting examples. In this example, the update strategy for the gaussian scene representation can be selected based on levels of available computing resources. A technical advantage of using the device condition to select the update strategy' for the gaussian scene representation is that the gaussian scene representation can be maintained using appropriate updates (e.g.. full or partial) when sufficient computing resources are available, and more computationally expensive updates (e.g., full updates or partial updates using more gaussians) can be deferred or limited at other times, thereby limiting the impact of updating the gaussian scene representation on other processes.

[0034] The available update strategies that can be selected among can include at least a full update, a partial update, and no update. Further, in some cases, the available update strategies can include various types of partial updates, such as object-based updates, pixel-based updates, updates that are limited to specified parameters of a selected subset of gaussians, staged search updates, etc. In a full update, all of the gaussians of the gaussian scene representation are processed (generally, based on all of the available views). For example, during a full update, a large number of camera poses are used, and a rendered view is generated for each. Each rendered view is compared to a groundtruth view (e.g., an image of the scene captured by a camera) to calculate a loss function. A value or values of the loss function are used to determine gradients based onQUALCOMM Ref. No. 2501497WO- 8 / 94 -the differences between an output of the gaussian scene representation (e.g., a rendered view) and an expected output of the gaussian scene representation (e.g., a ground-truth view), and parameters of one or more of the gaussians of the gaussian scene representation are updated based on the gradients. In this example, during a full update, gradients are determined for every parameter of every gaussian of the gaussian scene representation, which is very computationally expensive.

[0035] In contrast to a full update, during a partial update only a selected subset of the gaussians of the gaussian scene representation are updated and remaining gaussians are not modified. For example, a first set of gaussians of the gaussian scene representation are selected for update and the remaining gaussians of the gaussian scene representation are held constant during the partial update. In some cases, only selected parameters of the first set of gaussians are updated. For example, each gaussian is specified by a set of parameters including location parameters indicating a location of a center of the gaussian in a scene space, an opacity parameter indicating an opacity (e.g., a maximum opacity) of the gaussian, one or more orientation parameters indicating an orientation of the gaussian in the scene space, a set of color parameters indicating spherical harmonics coefficients for defining a color of the gaussian as viewed from various directions, and so on. In this example, for some partial updates, a first subset of the parameters of each selected gaussian are updated and a second subset of the parameters of each selected gaussian are held constant. To illustrate, the location parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant. As another illustrative example, one or more color parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant.

[0036] The processor(s) can be configured to select an update strategy to be used, from among a plurality of available update strategies, based on the input pixel data, based on sensor data, based on a device condition, based on other factors (e.g., user input), or based on a combination thereof. For example, the processor(s) can be configured to perform no update when a metric associated with the input pixel data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. In this example, the metric can be based on an amount ofQUALCOMM Ref. No. 2501497WO- 9 / 94 -difference between the input pixel data and reference pixel data, a size or dimensions of a region that differs between the input pixel data and reference pixel data, an amount of motion detected in the input pixel data, an amount of lighting change detected in the input pixel data, or other factors that are indicative of whether the gaussian scene representation accurately captures information in the input pixel data. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated with the sensor data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated with the device condition fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. Metrics based on separate threshold comparisons for each factor are provided as an illustrative example; in other examples a metric can be based on a comparison of a combination of factors and corresponding thresholds. In addition to selecting between no update, a partial update, and a full update, in some cases, the processor(s) can select among several available partial update strategies.

[0037] The processor(s) can be configured to select which portions of the gaussian scene representation (e.g., which gaussians) are eligible to be modified during a partial update based on the input pixel data, based on sensor data, or both. For example, the processor(s) can identify a region of interest associated with differences between the input pixel data and the reference pixel data. In this example, the processor(s) can use mapping data to identify gaussians of the gaussian scene representation that are associated with the region of interest. In this example, the identified gaussians are selected to be updated (e.g., are eligible to be modified) as part of the partial update. In some cases, the selected gaussians can affect pixels associated with the region of interest as well as other pixels. In such cases, mapping data can be used to determine which pixels of the input pixel data are to be used as training data to update the selected gaussians.QUALCOMM Ref. No. 2501497WO- 10 / 94 -

[0038] In a particular aspect, the mapping data can be generated based on gradient data. For example, saliency mapping can be used to estimate which gaussians most affect particular output pixel data from the gaussian scene representation to generate pixel-to-gaussian mapping data. Likewise, saliency mapping can be used to estimate which pixels of input data are most important to training of particular gaussians to generate gaussian-to-pixel mapping data.

[0039] Saliency mapping generally involves at least one forward pass through the gaussian scene representation (e.g., rendering one or more views) to generate gradients used to determine the pixel-to-gaussian mapping data, and at least one backpropagation pass to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially when a gaussian scene representation is updated frequently. To reduce the computational demand associated with running the gaussian scene representation to generate mapping data, particular aspects disclosed herein use a staged approach to manage the mapping data. In the staged approach, after mapping data (e g., pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, as described above, the mapping data can be stored for subsequent reference. For example, after mapping data associated with a particular region of interest is generated, the mapping data can be stored in a searchable data store. In this example, when new input image data is to be used for a partial update of the gaussian scene representation, a new region of interest in the new input image data is identified, and the searchable data store is searched to determine whether mapping data corresponding to pixels of the new region of interest is available (e.g., has already been generated and stored). If the mapping data is available, the mapping data can be retrieved from the searchable data store and used without the need to perform saliency mapping operations to generate the mapping data.

[0040] The region of interest used in the staged approach can be associated with a particular set of pixels, with a particular spatial area within the scene, with a particular object (or a portion of an object) identified in the scene, or with another identifiable feature of the scene. As one example, objects within the scene can be identified and saliency mapping can be used to associate one or more of the objects with gaussians of the gaussian scene representation. In this example, when new input image data is obtained, object detection can be performed to identify objects in the new input imageQUALCOMM Ref. No. 2501497WO- 11 / 94 -data. If a particular object identified in the new input image data is associated with a change that triggers a partial update of the gaussian scene representation, the searchable data store is searched to determine whether mapping data is available for the particular object. If the mapping data for the particular object is available in the searchable data store, the mapping data is used for the partial update of the gaussian scene representation. If the mapping data for the particular object is not available in the searchable data store, the mapping data can be generated using saliency mapping.

[0041] Alternatively, in some cases, if particular mapping data is not available in a first searchable data store, a second searchable data store can be searched for the mapping data. For example, a set of most frequently used mapping data can be stored at a first searchable data store, a set of next most frequently used mapping data can be stored at a second searchable data store, and so forth, to an Nth data store (where N is an integer greater than 2). In such cases, full mapping data generation (e.g., using saliency mapping) is performed if none of the searchable data stores include the desired mapping data. Further, in some cases, after full mapping data generation is performed, the resulting mapping data can be added to one or more of the searchable data stores. Each of the searchable data stores can use an eviction policy in which a set of most recently- used or most frequently used mapping data is retained at the first searchable data store, and each subsequent searchable data store retains successively less recently used or less frequently accessed mapping data.

[0042] In some cases, the staged approach can use a single searchable data store, rather than several stages of searchable data stores. In such cases, the single searchable data store can evict the oldest or the least recently accessed mapping data when new mapping data is added. In such cases, full mapping data generation is performed if the single searchable data store does not include the desired mapping data. A technical advantage of the staged approach is that significant computing resources can be preserved by avoiding recreating mapping data when new input image data is to be used for a partial update of the gaussian scene representation.

[0043] In the present discussion, a '“gaussian” refers to a graphical element of a gaussian scene representation. Each gaussian is specified by a set of parameters indicating characteristics, described further below, of the gaussian in scene space. InQUALCOMM Ref. No. 2501497WO- 12 / 94 -some examples, gaussians have a generally ellipsoidal 3D shape in the scene space, with a maximum opacity at a center position of the gaussian. In some examples, the opacity diminishes following an equation similar to a Gaussian probability distribution.

[0044] In a particular aspect, the parameters specifying a set of gaussians include:• a position vector, **p**_i ,which indicates the center position of a gaussian with index• an opacity parameter, ti∈ [0,1], which indicates a maximum opacity of gaussian;• orientation parameters, qf, which can include a quaternion defining the spatial orientation (rotation) of the gaussian;• a scale vector, sf, which defines a size and shape of the ellipsoid of the gaussian;and• a set of spherical harmonics (SH) coefficients, **H**_i, which can include a matrix with columns composed of 3D color vectors (in portions of this disclosure **H**_i is referred to as an SH coefficient array).

[0045] In some implementations, not all of the parameters described above are used to specify a gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacity parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a view-independent color) rather than higher-order spherical harmonics that capture view-dependent color variations. Using a reduced parameter set can decrease memory usage and computational complexity associated with storing, updating, and rendering the gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.QUALCOMM Ref. No. 2501497WO- 13 / 94 -

[0046] In some implementations, the gaussian scene representation may use two-dimensional (2D) gaussians rather than three-dimensional (3D) gaussians. A 2D gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D gaussians, which distribute density throughout a model the complete volumetric ellipsoid, 2D gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.

[0047] A 2D gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parameters (e.g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D gaussian primitive may have fewer parameters than a corresponding 3D gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.

[0048] Using 2D gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D gaussian primitives can be more tightly aligned to object surfaces than 3D gaussians. Additionally, 2D gaussians may provide more accurate geometry' representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.

[0049] In some implementations, the gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each gaussian of the gaussian scene representation can be used to represent a triangular face of a mesh, enabling conversion between gaussian representations and mesh representations. This approach provides a one-to-one mapping between gaussians and triangular mesh facesQUALCOMM Ref. No. 2501497WO- 14 / 94 -that is differentiable, enabling the gaussian scene representation to be modified in a manner similar to modifying a mesh.

[0050] To convert a 2D gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D gaussian may be performed. The covariance matrix S can be decomposed as Σ = RΛRT, where R is a matrix of eigenvectors (e1, e2, e3) and Λ is a diagonal matrix of eigenvalues (λ1, λ2, λ3). The two largest eigenvalues (λ1, λ2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (λ3) corresponds to the normal direction. A triangular face can then be constructed using the mean of the gaussian as a center point and defining triangle vertices along the tangent directions. For example, triangle vertices may be defined as v1 = μ + α1e1, v2 = μ + α2e2, and v3 = μ − α1e1, where α1 and α2 are scale factors proportional to square roots of their respective eigenvalues λ1 and λ2 and μ is the center point corresponding to the mean of the 2D gaussian. This creates a triangle aligned with the gaussian's orientation and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the gaussian.

[0051] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the gaussian. When the mesh is modified, the gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the gaussians when the mesh is animated or otherwise modified. During rendering, the gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of gaussians of the gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between gaussians and triangular faces.

[0052] Throughout this disclosure, references to gaussians, gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D gaussians, 2D gaussians, mesh-integrated gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the type of gaussian representation used, as described herein.

[0053] The following elements may be determined based on the parameters above:QUALCOMM Ref. No. 2501497WO- 15 / 94 - • **Σ** is a 3×3 matrix similar to the covariance matrix in Gaussian probability distributions.• c is a 3D view-dependent color vector for each gaussian, computed using the set of spherical harmonics coefficients of the gaussian.

[0054] The three dimensions of the color vector c may correspond to three different color component values, such as a red value, a green value, and a blue value, that specify a color for the gaussian. The color vector c is view-dependent in the sense that the gaussian may appear to have different colors depending on a position from which the gaussian is viewed, e.g., due to reflections. The view-dependent color vector c for the gaussian may be derived from a camera position f. the center position p, of the gaussian, and the SH coefficient array Hi of the gaussian. The camera position f and the center position p, may each be a vector of spherical coordinates (0, cp). Thus, a viewing direction vector comprising a Q and a coordinate value that defines a viewing direction from the camera position f to the center position p, may be described as (f -p;). Therefore, the view-dependent color vector c for a gaussian i having central position p, from camera position f may be written as c / (f - p;, H;) or simply c(f - p, H).

[0055] In certain 3D Gaussian Splatting (3DGS) processes, H;includes up to 16 spherical harmonics coefficients per color vector component (e.g., 16 spherical harmonics components for red, 16 spherical harmonics components for green, and 16 spherical harmonics components for blue). A Tenderer may calculate a value of a color component in the view-dependent color vector c based on a linear combination LC of a set of terms Ti... Ymax scaled by the spherical harmonics coefficients in H, for the color component. That is, the Tenderer may calculate the linear combination Cij for gaussian i and color component j as:LC Ij, J j = H; L, J;, 1± Y ±i + H; (■,];,2. T2+ ”• H; L, J;,m1 / LalxA,'^m IIladxA>’ j- / =1, '2, '3, ' (1) 'Equation (1) may be rewritten as follows:**c**_i = **H**_i **y**, (2)QUALCOMM Ref. No. 2501497WO- 16 / 94 -where H_{i,j,1}, H_{i,j,2}, etc., are spherical harmonics coefficients for gaussian i in **H**_i for the color component j. Each of the terms in vector y may be defined by the following spherical harmonics equation:In equation (3) above, the values (θ, φ) specify the viewing direction vector derived from **f** and **p**_i. The values l and m are the degree and order of the spherical harmonics equation, where − l ≤ m ≤ l. Thus, in an example where there are 16 spherical harmonics coefficients, the maximum value of l is 3, and when l is 0, m is 0; when l is 1, m is 1, 0, -1; when l is 2, m is 2, 1, 0, -1, -2, and so on. Thus, Y_1 may correspond to l being 0 and m being 0, Y_2 may correspond to l being 1 and m being 1, Y_3 may correspond to l being 1 and m being 0, Y_4 may correspond to l being 1 and m being -1, Y_5 may correspond to l being 2 and m being 2, and so on. With respect to conventional RGB values, the Tenderer may calculate the value of a color component in the view-dependent color vector c by¬ applying a sigmoid function to LC and multiplying by a dynamic range scaling factor, such as 255.

[0056] To generate a rendered image based on a set of gaussians, a renderer may determine color values of pixels in the rendered image. A rendered image is a type of digital image that is composed of a grid of individual pixels, each with its own color vector. Each pixel is associated with a camera ray extending from the pixel through a virtual aperture of a virtual camera. In conventional 3DGS rendering, for each pixel of the rendered image, the renderer may identify gaussians that intersect the camera ray associated with the pixel and sort the identified gaussians based on a depth from the virtual camera. For each of the identified gaussians, the renderer calculates an opacity value for a location within the gaussian based on the position vector of the gaussian, the maximum opacity of the gaussian, the orientation vector of the gaussian, and the scale vector of the gaussian. The renderer calculates the opacity value (α) for 3D location **x** along the camera ray within a gaussian i (i.e., α_i(**x**)) as follows:QUALCOMM Ref. No. 2501497WO- 17 / 94 -i = 1,2,···,N, (4)

[0057] In equation (4) and throughout this disclosure, the exp function is ex. where e is Euler’s number, (,.)Tindicates matrix transposition, f...]'1indicates a matrix inversion, and N indicates the total quantity of identified gaussians that intersect the camera ray. In general terms, equation (4) scales the maximum opacity value (ti) of the gaussian based on a Gaussian distribution function according to a distance of the location x from a center pi of the gaussian. The location x may be defined as a vector of coordinates, e.g., spherical coordinates, Cartesian coordinates, etc.

[0058] When processing a location **x** within a gaussian, the renderer also calculates a view-dependent color vector **c** for the gaussian. The view-dependent color vector c may include color component values that define a color for the gaussian, such as red, green, and blue values. Considering a virtual camera with a focal point (i.e., virtual aperture) at position f, the Tenderer may calculate an alpha-scaled color vector r;for gaussian i. The alpha-scaled color vector specifies a contribution of a location x within the gaussian i to a color of a pixel. The alpha-scaled color vector may be calculated using the following formula:**r**_i(**x**, **f**) = α_i(**x**) **c**(**f** − **p**_i, **H**_i), i = 1,2,···,N. (5)

[0059] Thus, equation (5) determines an alpha-scaled color vector r for a location x viewed from position f by scaling the view-dependent color vector c of a gaussian i that contains the location x based on the previously determined opacity value for the location **x** (i.e., α_i(**x**)) The view-dependent color vector c may be determined using a spherical harmonics equation based on the SH coefficient matrix Hi of the gaussian and the differences between coordinates the center position of the gaussian (f - pi) as described above. The view-dependent color vector of a gaussian may be dependent on the viewing position f because of the directions of light sources, which can cause shadows, reflections, and other optical effects.QUALCOMM Ref. No. 2501497WO- 18 / 94 -

[0060] Depending on the size and orientation of a gaussian and a discretization of the 3D space into discrete locations, there may be multiple locations along a camera ray that intersect the same gaussian. Hence, the renderer may calculate multiple opacity values and color vectors for the same gaussian. The renderer may use these opacity values and color values in a rendering equation for determining a color vector of a pixel of a rendered image. However, for ease of explanation, this disclosure and the equations thereof assume that there is only one location per gaussian.

[0061] An advantage of using the formula of equation (4) is that mapping gaussians to the 2D rendered image can be approximated by a 2D gaussian with a 2x2 covariance matrix that can be computed as**Σ**2D= **J** **W** **Σ**3D**W**T**J**T, (6)where W is a matrix defined by the camera’s image-generation transformation, and J is the Jacobian matrix defined by an affine approximation of the projective camera transformation.

[0062] For ease of explanation, this disclosure omits some details about how rendering is discretized for image pixels, and considers that, for all gaussians, there are opacity values α_i[m, n] and color vectors **c**_i[m, n] defined for each image pixel with integer image coordinates [m, n]. The image coordinates of a pixel indicate a 2D location of the pixel within a rendered image. In other words, each pixel of a rendered image is identified by a set of coordinates m and n. Due to the transparency of gaussians, before a scene is rendered, the gaussians to be rendered are sorted according to their distance from the camera position f. Formally, a sorting sequence s[ / c] of indexes of gaussians is determined according to increasing distance, such thatk, l = 1,2,···,N. (7)In equation (7), the double straight brackets (i.e., ||... ||) indicate Euclidean distance.QUALCOMM Ref. No. 2501497WO- 19 / 94 -

[0063] Thus, the renderer processes locations (e.g., gaussians) along the camera ray associated with a pixel in an order from closest to the virtual camera to further from the virtual camera. Hence, if the opacity values of locations within the gaussians intersected by the camera ray associated with the pixel reach a level corresponding to complete opacity, later gaussians that intersect the camera ray do not contribute to the color vector of the pixel. Hence, the renderer can skip processing of locations along the camera ray associated with the pixel that intersect the later gaussians.

[0064] The Tenderer may compute a final color vector for a pixel based on the opacity values and the view-dependent color values for the locations along the camera ray associated with the pixel. For instance, after sorting the gaussians based on depth, the Tenderer may compute a rendered color vector r for the pixel according to the following equation, which corresponds to the computer graphics technique of alpha-blending:

[0065] In equation (8), N indicates a total quantity of graphics elements (e.g., gaussians) along the camera ray associated with the pixel at coordinates [m, n]. The term **c**_{s[k]}[m, n] indicates a color vector for a distance-sorted graphics element having index value k in a sorting order of the graphics elements. In an example where the graphics element is a gaussian, **c**_{s[k]}[m, n] may be a view-dependent color vector for the gaussian. The term α_{s[k]}[m, n] indicates an opacity value for the distance-sorted graphics element having an index value k in the sorting order. Similarly,[m, n] indicates an opacity value for the distance-sorted graphics element having an index value I in the sorting order. Note that in equation (8), the termis calculated individually for each of index values k=l to N so that the scaled color vector for the graphics element with index value k (cs[k] [m, n]n]) is further scaled based on the opacity values of graphics elements closer to the camera position than the graphics element with index value k. Thus, a renderer using equation (8) sorts the graphics elements based on depth from the camera position.QUALCOMM Ref. No. 2501497WO- 20 / 94 -

[0066] In complex scenes, like those used in computer games, most of the graphics elements are assumed to be opaque (i. e.. at= 1 in the object’s surface), so equation (8) can be implemented in a massively parallel manner by storing, for each pixel, the minimum depth of the elements already rendered, and only updating the color vector of a pixel if the newly visited element has smaller depth (Z-buffer updates). This is what enables current graphics processing units (GPUs) to render complex scenes at high resolutions and frame rates. On the other hand, when there is transparency, equation (8) defines a strictly sequential order for processing elements, and the only way to parallelize rendering is to create sets of non-overlapping elements and implement sequential alpha-blending on each set.

[0067] This made introduction of transparency in games challenging, but transparency is needed to represent many features, like flames and smoke, clouds, ‘‘force fields,” and so on. One proposed empiric solution to avoid sorting, named Order-Independent Transparency (OIT), is to approximate equation (8) with a weighted sum. There are several OIT variants, such as to replace equation (8) with the following equation for determining a rendered color vector **f̂** for a pixel at coordinates [m, n] of a rendered image:where z_i is the distance to a virtual camera, **c**_0[m, n] is the background color for the pixel at position [m, n] in the rendered image, and w(z, α) is a function that decreases with distance, so that objects nearer to the camera are assigned larger weights in the alphanormalized weighted sum of equation (9). The renderer may calculate the term ā[m,n] using the following equation:Nā[m,n] = ∏(1 − α_i[m, n]), (10) i=1 to NNote that equation (9) is a weighted sum of a first value and a second value, where the first value is the background color vector **c**_0 for the camera ray associated with pixel [m, n] scaled by the accumulated opacity (ā[m, n]) of the camera ray, and the second term is aQUALCOMM Ref. No. 2501497WO- 21 / 94 -composited color vector scaled by the accumulated transparency (1 — a[m, n]) of the camera ray. In the O1T rendering of equation (9), there are two summation operations ∑ᵢ₌₁ᴺ ... for determining a color vector for a pixel, and since addition is commutative, the terms in the summation operations can be computed in any order. However, the final color vector for a pixel can only be computed after the two sums are known, which can be done with a single pass over all pixels, performing the divisions in equation (9). In other words, the Tenderer can perform the summation operations for all pixels, and then, after the summation operations for all pixels are complete, perform the division operations for all pixels. Since OIT uses the term w(z, a) instead of the actual opacity values of closer graphics elements, OIT may eliminate the need for sorting the graphics elements based on depth during rendering. However, use of the term w(z, a) instead of the actual opacity values of closer graphics elements means that OIT only provides an approximation of alpha blending and does not model truly opaque surfaces.

[0068] Because OIT allows rendering transparency without sorting, OIT may be considered for eliminating the computationally expensive sorting phase of Gaussian splatting rendering. However, while OIT methods have been extensively used in computer games for several years, the OIT methods were developed to be used with conventional computer graphics, and only meant to represent semi-transparent elements. Furthermore, the OIT methods are not expected to generate good approximations since the OIT methods are used with elements of shape and appearance w here differences are difficult to see, like flames and smoke. Finally, human intervention may be needed to identify when the OIT methods are not working correctly, and to fix problems.

[0069] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms “a,” 'an." and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG. 1 depicts a device 102 including one or more processors (“processor(s)” 130 of FIG. 1), which indicates that in some implementations the device 102 includes a single processor 130 and in other implementations the deviceQUALCOMM Ref. No. 2501497WO- 22 / 94 - 102 includes multiple processors 130. For ease of reference herein, such features are generally introduced as "‘one or more” features and may subsequently be referred to in the singular or optional plural (as indicated by “(s)”) unless aspects related to multiple of the features are being described.

[0070] As used herein, the terms "comprise.” “comprises.” and “comprising” may be used interchangeably with “include,” “includes,” or “including.” Additionally, the term “wherein” may be used interchangeably with “where.” As used herein, “exemplary” indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

[0071] The phrase “corresponds to” as used herein is a relational phrase indicating correspondence, equivalence, or matching. For example, if A corresponds to B. then A is B, there is a mapping between A and B, or A matches B. The phrase “is associated with” as used herein is a broad relational phrase indicating a looser or more general relationship such as, for example, a categorical relationship (e.g., A is part of or belongs to B). a causal relationship (e.g., A causes B), a logical relationship (e.g., If A then B), correlation (e.g., when A is present B is present), a structural relationship (e.g., the B that is coupled to A), and other possible relationships. Correspondence always includes association; whereas association can, but does not always, indicate correspondence. For example, if A is associated with B, there can be a mapping between A and B that could be described as correspondence between A and B.

[0072] As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled.” and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., aQUALCOMM Ref. No. 2501497WO- 23 / 94 -wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, “directly coupled” may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0073] In the present disclosure, terms such as “obtaining,” “determining,” “calculating,” “estimating,” “shifting,” “adjusting,” etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “obtaining,” “generating,” “calculating.” “estimating,” “using,” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “obtaining,” “generating,” “calculating,” “estimating,” or “determining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, receiving, or accessing the parameter (or signal) that is already generated, such as by another component or device.

[0074] As used herein, “image data” and “pixel data” should be understood according to any of their usual and customary meanings in the fields of image processing, computer vision, photography, medical imaging, and / or artificial intelligence. For example, image data includes a digital representation of visual information, such as color, intensities, and spatial arrangement of portions of an image, and may also include metadata and / or auxiliary information, such as a time, a date, and / or a location associated with the image data; field of view information; camera pose information; camera intrinsic parameters; etc. The color, intensities, and spatial arrangement information of an image is represented in pixel data, where each pixel has a corresponding set of values representing color, intensity, and position within a field of view of the image.QUALCOMM Ref. No. 2501497WO- 24 / 94 -

[0075] FIG. 1 is a block diagram of an illustrative system 100 that is operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure. In FIG. 1, the system 100 includes a device 102 that includes a memory 120 configured to store one or more gaussian scene representations, such as illustrative gaussian scene representation 154, and one or more processors 130 configured to selectively update one or more of the gaussian scene representation(s) based on input data.

[0076] FIG. 1 also illustrates a schematic representation of a scene 132 that can be represented by the gaussian scene representation 154. The scene 132 can be a 3D scene or a 4D scene, where a fourth dimension of the scene 132 is time, such that the 4D scene models the 3D scene over time. FIG. 1 also illustrates a view 136 associated with the scene 132. For example, the view 136 represents what a camera 134 with a particular pose and field of view would capture of the scene 132.

[0077] The device 102 includes or is coupled to one or more sources 104 of input data. For example, the source(s) 104 illustrated in FIG. 1 include one or more image sources 106, one or more sensors 110, a modem 114, or a combination thereof. In other examples, the source(s) 104 can also, or alternatively, include other devices or systems, such as a memory device, a media player, a game system, etc. Although the image source(s) 106, the sensor(s) 110, and the modem 114 are illustrated outside the device 102 in FIG. 1; in other examples, the image source(s) 106, the sensor(s) 110, the modem 114, or a combination thereof, are integrated within the device 102.

[0078] The image source(s) 106 are configured to provide image data 108 to the device 102. For example, the image source(s) 106 can include or correspond to one or more still cameras, one or more video cameras, or both. As another example, the image source(s) 106 can include a memory that stores the image data 108 or a computer or similar device that generates the image data 108 by executing instructions, such as a game engine that generates images to represent a virtual reality, augmented reality, or mixed reality scene. The image source(s) 106 can be controlled manually (e.g., based on user input) or automatically (e.g., based on sensor data 112. based on position information, based on a schedule, etc ).QUALCOMM Ref. No. 2501497WO- 25 / 94 -

[0079] The sensor(s) 110 are configured to provide sensor data 112 to the device 102. For example, the sensor(s) 110 can include a light sensor that is configured to generate light data representing lighting intensity or lighting changes within a scene. As another example, the sensor(s) 110 can include a motion sensor that is configured to generate motion data representing movement of the device 102, movement of another device, or movement within a portion of the scene 132. In still other examples, the sensor(s) 110 can include one or more ranging sensor(s), such as sonar, lidar, or radar sensor(s). In some cases, the image data 108 can be processed to generate motion data, light data, or both.

[0080] The modem 114 is configured to provide data 116 received from a remote device to the device 102. The data 116 received via the modem 114 can include sensor data 112, image data 108, user input, mapping data 122, reference images 124, gaussians 156 of a gaussian scene representation 154, other data, or a combination thereof.

[0081] In a particular aspect, the gaussian scene representation 154 can be stored in the memory 120. The memory 120 can also include one or more buffer(s) 118 configured to store data received from the source(s) 104 for analysis and / or use in updating the gaussian scene representation 154. For example, the buffer(s) 118 can include an image buffer configured to store the image data 108, a sensor data buffer configured to store the sensor data 112, or both. The memory 120 may also be configured to store reference images 124. The reference images 124 can include one or more training images 126 (e.g., an image of the scene 132 used to train the gaussian scene representation 154), synthetic images 128 (e.g., an image of the scene 132 generated as output by the gaussian scene representation 154), or both. The reference images 124 can be used as a reference for a current state of a gaussian scene representation 154 for purposes of determining whether and / or how to update the gaussian scene representation 154 based on newly received input pixel data.

[0082] In FIG. 1, the processor(s) 130 include a selective model update pipeline 140, a Tenderer 158, a controller 160, and a view selector 162. FIG. 1 also shows the gaussian scene representation 154 in the processor(s) 130; however, which may be the case during certain operations. However, in other situations, such as during a partial update of the gaussian scene representation 154, only a subset of the gaussian sceneQUALCOMM Ref. No. 2501497WO- 26 / 94 -representation 154 (e.g., selected gaussians 156) may be loaded to the processor(s) 130. Similarly, the view selector 162 and the Tenderer 158 may only be loaded to the processor(s) 130 for particular operations, such as rendering the view 136 or rendering views during a full update of the gaussian scene representation 154.

[0083] As explained further below, the selective model update pipeline 140 is configured to manage updating of the gaussian scene representation 154 based on input pixel data. The Tenderer 158 is configured to generate views (e.g., the view 136) of the scene 132 based on the gaussian scene representation 154 and based on view selection data from the view selector 162. The view selector 162 is configured to provide the gaussian scene representation 154, the Tenderer 158, or both, with the view selection data, where the view selection data indicates a location, a pose, a field of view, a focal length, other information specifying a portion of the scene 132 to be rendered, or a combination thereof. In the example of FIG. 1, the Tenderer 158 is coupled to a display 138 that is configured to display the view 136; however, in other cases, the view 136 can be stored or sent to another device for use. To illustrate, the view 136 can be sent via the modem 114 to a remote device for further processing or display.

[0084] The controller 160 is configured to coordinate tasks among various components of the system 100. For example, the controller 160 can cause one or more of the source(s) 104 to provide data to the processor(s) 130 and / or the memory 120 at particular times or intervals. As another example, various components of the system 100 can provide status information associated with their respective tasks to the controller 160, and the controller 160 can use the status information to initiate or control other components of system 100. For example, when the device 102 receives new image data 108 from the image source(s), the controller 160 can be notified and can initiate the selective model update pipeline 140 or a portion thereof in response to the new image data 108. As another example, when the update strategy selector 144 selects a particular update strategy, the controller 160 can cause components associated with the particular update strategy to be initiated. To illustrate, a full update 148 includes rendering of various views of the gaussian scene representation 154, whereas a partial update 150 may not entail rendering of views. Accordingly, when the update strategy selector 144 selects the full update 148, the controller 160 can be notified and can cause the processor(s) 130 to load or initiate the view selector 162 and the renderer 158,QUALCOMM Ref. No. 2501497WO- 27 / 94 -

[0085] In the example illustrated in FIG. 1, the selective model update pipeline 140 includes one or more data analyzers (such as illustrated data analyzer 142). an update strategy selector 144, and update engines 146. In this example, the data analyzer 142 is configured to analyze input data, such as at least a portion of the image data 108, at least a portion of the sensor data 112, or other data, to generate one or more metrics, classification data, or other data indicative of estimated, expected, or detected changes in the scene 132. For example, changes in the scene 132 can be detected based on differences between input pixel data (e.g., at least a portion of the image data 108) and reference pixel data of the reference images 124. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and reference pixel data (e.g., based on a pixel-by-pixel comparison) and generate one or more difference metrics. A difference metric can indicate, for example, a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.

[0086] The update strategy selector 144 is configured to select an update strategy to be used (if any) based on one or more update criteria. For example, the update strategy selector 144 can select from among a full update 148, one or more different types of partial updates (including a partial update 150 in FIG. 1), or no update 152 based on an update criterion (or based on more than one update criteria). If the partial update 150 is to be performed, the update strategy selector 144 may also select a portion of the input pixel data to be used to update a corresponding set of gaussians 156 of a gaussian scene representation 154.

[0087] In a particular aspect, the update strategy selector 144 selects an update strategy based on output of the data analyzer 142 (e.g., based on one or more of the difference metrics). Additionally, or alternatively, the update strategy selector 144 can select the update strategy based on a device condition associated with the device 102 or a component thereof. For example, the sensor data 112 can include device condition data indicating a location of the device 102, a battery level of the device 102, whether the device 102 is charging, a temperature of the device 102, etc. Additionally, or alternatively, the processor(s) 130, the memory 120, or another component of the deviceQUALCOMM Ref. No. 2501497WO- 28 / 94 - 102 can provide device condition data indicating a level of usage of a computing resource of the device 102. To illustrate, the device condition data can include a processor utilization metric (e.g., a processor load), a memory utilization metric (e.g., memory usage, cache load, or memory access latency), a communication metric (e.g., a bus load, an available network bandwidth, or network latency), or a similar metric. Examples of a processor utilization metric include, without limitation, user processor time (e.g., time spent executing user-space processes), system processor time (e.g., time spent on kernel-level operations), idle time (e.g., percentage of time a processor 130 is not executing any tasks), input / output wait time, load average (e.g., average number of processes waiting to execute in a given time period), a count of context switches per second, a count of processor throttling events (e.g., due to overheating or power-saving measures), or a combination thereof.

[0088] For example, the update strategy selector 144 can be configured to compare one or more difference metrics from the data analyzer 142 to corresponding thresholds to select the update strategy. To illustrate, the update strategy selector 144 can compare a value of a difference metric based on the differences between the input pixel data and the reference pixel data to one or more thresholds to determine whether to perform the partial update 150, the full update 148, or no update 152. As anon-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of the difference metric is less than a first threshold, determine to perform the partial update 150 if the value of the difference metric is greater than the first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the difference metric is greater than the second threshold. As a second non-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of a size metric is less than a first threshold, determine to perform the partial update 150 if the value of the size metric is greater than a first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the size metric is greater than the second threshold. In this illustration, the size metric can be based on dimension of a difference region, a count of pixels of the difference region, a count of gaussians 156 of the gaussian scene representation 154 associated with the difference region (also referred to herein as "‘difference gaussians”), or a combinationQUALCOMM Ref. No. 2501497WO- 29 / 94 -thereof. For example, the size metric can be based on a count of the difference gaussians and a count of gaussians 156 of the gaussian scene representation 154.

[0089] When the update strategy selector 144 determines to perform the full update 148, the update engines 146 perform operations that are similar to initial creation of a gaussian scene representation except that the full update 148 is initialized from the existing gaussian scene representation 154. For example, during the full update 148, all of the gaussians 156 of the gaussian scene representation 154 can be processed from various views (as described further with reference to FIGS. 3A and 3B) to update their respective parameters. In this example, processing the gaussians 156 includes rendering selected views, comparing the rendered views to ground-truth image data (e.g., images of the scene) to determine loss values, and using the loss values to update the gaussians 156 (e.g., using backpropagation and / or other model training techniques). The full update 148 can also include operations such as adaptive density control which can generate new gaussians 156 or eliminate or merge gaussians 156. Thus, the full update 148 is very computationally intensive.

[0090] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all of the gaussians 156) of the gaussian scene representation 154 are updated, as described further with reference to FIGS. 3A and 3C. For example, the update strategy selector 144 may indicate that a first subset of gaussians 156 are selected for update and a second subset of gaussians 156 of the gaussian scene representation 154 are not selected for update and are to be held constant. The update strategy selector 144 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and optionally a second subset of the input pixel data that is not to be used for the partial update 150.

[0091] In a particular aspect, the update strategy selector 144 selects the first subset of gaussians 156 to be updated and the first subset of the input pixel data used in the partial update 150 based on differences between the input pixel data and reference pixel data of the reference images 124, based on the sensor data 112, based on user input, or a combination thereof. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data can be used to estimate the difference region. To illustrate, changes inQUALCOMM Ref. No. 2501497WO- 30 / 94 -lighting, as detected by a light sensor, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by a motion sensor and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data. As another example, a user can provide user input designating the difference region.

[0092] The update strategy selector 144 can use the mapping data 122 (e.g., pixel-to-gaussian mapping data) to identify a subset of the gaussians 156 associated with pixels of the difference region. In this example, the pixels of the difference region can be used for the partial update 150, and the subset of the gaussians 156 associated with pixels of the difference region can be updated during the partial update 150. Other gaussians 156 of the gaussian scene representation 154 are not modified during the partial update 150.

[0093] It is likely that there will not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, a particular gaussian 156 may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g.. the view 136). Thus, in some cases, the update strategy selector 144 is also configured to use the mapping data 122 (e.g., gaussian-to-pixel mapping data) to identify pixels that are influenced by the subset of the gaussians 156 associated with pixels of the difference region. In such cases, the pixels of the input pixel data that correspond to the pixels that are influenced by the subset of the gaussians 156 are included in the training data used for the partial update 150. Updating a gaussian in this context refers to changing one or more parameters of the gaussian.

[0094] In a particular aspect, the update strategy selector 144 is configured to generate and / or update the mapping data 122. For example, the update strategy selector 144 can use saliency mapping to estimate which gaussians 156 most affect particular output pixel data from the gaussian scene representation 154 to generate pixel-to-gaussian mapping data and / or can use saliency mapping to estimate which pixels of input pixel data are most important to training of particular gaussians 156 of the gaussian scene representation 154 to generate gaussian-to-pixel mapping data.QUALCOMM Ref. No. 2501497WO- 31 / 94 -

[0095] Saliency mapping generally involves at least one forward pass through rendering operations (e.g., projection and rasterization) to generate gradient values used to determine the pixel-to-gaussian mapping data and at least one backward pass through the rendering operations to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially if the gaussian scene representation 154 is large and / or updated frequently. To reduce the computational demand associated with running the gaussian scene representation 154 to generate mapping data 122, the selective model update pipeline 140 can use a staged approach to manage the mapping data 122. In the staged approach, after mapping data 122 (e.g.. pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, the selective model update pipeline 140 stores the mapping data 122 in a manner that facilitates reuse of the mapping data 122. For example, after mapping data associated with a particular region of interest (e.g., a range of pixels, an object, or some other identifiable portion of the input pixel data) is generated, the mapping data 122 can be stored along with information identifying the region of interest in a searchable data store. As one example, the region of interest can be associated with an object or a portion of an object in the scene 132. In this example, the mapping data 122 can include information that maps the object (or the portion of the object) to gaussians that represent the object. The mapping data 122 can also, or alternatively, include pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both, associated with the object. In this example, object detection (which uses fewer computing resources than saliency mapping) can be used to process input pixel data, and if an object detected in the region of interest is represented in the mapping data 122 the mapping data entry associated with the object can be retrieved from the mapping data 122. In contrast, if the object is not represented in the mapping data 122, a mapping data generation process (e.g., saliency mapping) can be performed.

[0096] In some cases, the update strategy selector 144 can use a staged approach with multiple data stores that each include mapping data 122, where a first stage includes a set of the most frequently used mapping data 122, a second stage includes a set of next most frequently used mapping data 122, and so forth. In such cases, full mapping data generation is performed if none of the searchable data stores includes the desired mapping data.QUALCOMM Ref. No. 2501497WO- 32 / 94 -

[0097] In a particular aspect, the update strategy selector 144 is configured to select an update strategy based, at least in part, on a device condition associated with the device 102. In some examples, the update strategy selection decision is based on the device condition and is independent of the magnitude or type of differences detected or expected in the input pixel data. For example, irrespective of the update strategy the update strategy selector 144 selects based on the detected or expected differences in the input pixel data, the update strategy selector 144 can select a different update strategy based on the device condition. In this example, the update strategy selected based on the device condition prevails or sets an upper bound on the type of update actually performed. To illustrate, if the device condition indicates that usage of computational resources is relatively low (e.g., below a first threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the full update 148, the partial update 150, or no update 152. However, if the device condition indicates that usage of computational resources is relatively high (e.g., above the first threshold and below a second threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the partial update 150 or no update 152. Further, if the device condition indicates that usage of computational resources is very high (e.g., above the second threshold), the update strategy selector 144 selects no update 152 irrespective of the detected or expected differences in the input pixel data.

[0098] In a particular aspect, results of operations performed by the device 102 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation 154 in which one or more parameters of one or more of the gaussians 156 has been modified), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views 136 generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with the scene 132. For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as anQUALCOMM Ref. No. 2501497WO- 33 / 94 -electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of the device 102 or another device, such as to turn on or turn off a display based on an aspect of the scene 132. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory or haptic notification or a combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene 132. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene 132.

[0099] FIG. 2 is a block diagram of an illustrative system 200 that is operable to update one or more gaussian scene representations 154, in accordance with some examples of the present disclosure. Various components, features, and / or elements of the system 200 can be stored in the memory 120 of FIG. 1 as data or instructions, can be performed by the processor(s) 130 of FIG. 1, or both. Additionally, various components of the system 200 correspond to components of the system 100 of FIG. 1, and can be configured asQUALCOMM Ref. No. 2501497WO- 34 / 94 -described with reference to FIG. 1. For example, the system 200 includes the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130. and the memory 120, one or more of which can be configured as described with reference to FIG. 1.

[0100] In the example illustrated in FIG. 2, the system 200 includes one or more buses, such as illustrated bus 216. coupled to the image source(s) 106. the modem 114, the sensors 110, the processor(s) 130, the memory 120, one or more other components or devices, or a combination thereof, via respective bus interfaces 131, 133, 135, 137, 139, 141, and 143. For example, the bus 216 can enable communication of data or signals (e.g., control signals, status signals, etc.), between and among the image source(s) 106. the modem 114, the sensors 110, the processor(s) 130, and the memory 120. The sensors 110 can include one or more motion sensors 210, one or more light sensors 212, one or more device sensors 214, other sensors, or a combination thereof.

[0101] In FIG. 2, the processor(s) 130 include the selective model update pipeline 140, which in this example, includes the data analyzer 142, the update strategy selector 144, the update engines 146, an update data selector 224, and a mapper 222. The processor(s) 130 optionally also include the controller 160, the renderer 158, or both.

[0102] The image source(s) 106 are configured to generate (e.g., capture), receive, or otherwise obtain image data 147 representing a scene. As an example, the image source(s) 106 can include one or more cameras, a memory' storing images, etc. In a particular aspect, the image source(s) 106 can provide image data 147 to the bus 216 via the bus interface 131 for communication to the processor(s) 130 (e.g., via the bus interface 143), to the memory 120 (e.g., via the bus interface 141), or both. In FIG. 2, the image data 147 is illustrated as provided to the data analyzer 142; however, in some case, the image data 147 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142. In some cases, the controller 160 can provide control signals to the image source(s) 106 (e.g., via the bus 216 and the bus interfaces 143, 131) to control when images are captured or provided to the memory 120 and / or the processor(s) 130. When the image source(s) 106 provide the image data 147 to the memory 120, the image data 147 can be stored in an image buffer (e.g., one of the buffer(s) 1 18 of FIG. 1).QUALCOMM Ref. No. 2501497WO- 35 / 94 -

[0103] The sensor(s) 110 are optional, and if present, are configured to generate sensor data 145. For example, the motion sensor(s) 210 can generate motion data indicating movement of a camera of the image source(s) 106 relative to the scene or movement of an object in the scene. As another example, the light sensor(s) 212 can generate light data indicative of changes in lighting of a portion of the scene. As yet another example, the device sensor(s) 214 can generate device condition data. The device condition data can indicate a condition associated with a device that includes the selective model update pipeline 140. To illustrate, the device condition data can indicate a battery level, whether the device is connected to an external power source, a temperature of the device, a level of usage of a computing resource of the device (such as a processor utilization metric or a memory utilization metric), or other similar device condition information. In FIG. 2, the sensor data 145 is illustrated as provided to the data analyzer 142; however, in some cases, the sensor data 145 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142.

[0104] In a particular aspect, the data analyzer 142 is configured to determine differences between input pixel data of the image data 147 and reference pixel data (e.g., pixel data of one or more of the reference images 124 of FIG. 1) and to provide information related to such differences to the update strategy selector 144 and optionally to an update data selector 224. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and the reference pixel data based on a pixel-by-pixel comparison and / or using optical flow methods. As another example, the data analyzer 142 can be configured to estimate or predict differences between the input pixel data and the reference pixel data based on motion indicated in motion data, based on lighting changed indicated in light data, or both.

[0105] The data analyzer 142 may be configured to generate one or more difference metrics associated with the differences between the input pixel data and the reference pixel data. For example, the difference metrics can indicate a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.QUALCOMM Ref. No. 2501497WO- 36 / 94 -

[0106] In a particular aspect, the data analyzer 142 is configured to identify a region of interest (ROI) based on the input pixel data. For example, the data analyzer 142 can identify the ROI based on the differences between the input pixel data and the reference pixel data, based on motion data associated with the input pixel data, based on sensor data associated with the input pixel data, based on other considerations, or a combination thereof. The data analyzer 142 is configured to perform a comparison of ROI data associated with the ROI and update target data 226 associated with the gaussian scene representation 154 to determine whether the ROI matches an update target specified by the update target data 226.

[0107] The update target data 226 can include data specifying update exclusion regions, update inclusion regions, update exclusion objects, update inclusion objects, other types or locations of scene changes associated with particular ty pes of update strategies, or a combination thereof. The update target data 226 can be specified manually (e.g., by one or more users or other persons configuring the selective model update pipeline 140), automatically (e.g., by the one or more processors 130 executing instructions), or a combination thereof (e.g., some update target data 226 provided automatically and some manually and / or some update target data 226 suggested automatically and approved manually). As an example of manual designation of a portion of the update target data 226, a user can provide input to designate a boundary of a region (e.g., an update inclusion region or an update exclusion region). In this example, the user can draw the boundary or a portion of the boundary, or the user can select an object or feature of the scene and the processor(s) 130 can use boundary’ detection operations to propose boundaries of a region corresponding to the object or feature. To illustrate, when the scene includes a playing field and spectator seats, the user can draw one or more lines or points identifying edges of the playing field to designate the playing field as a region. If the user draws points, the processor(s) 130 can generate segments connecting the points to generate the boundaries. Alternatively, the user can select one or more points in the playing field, and the processor(s) 130 can use boundary detection operations to identify boundaries of the playing field, which may optionally be presented to the user for approval. As another example, the user can provide text input to describe a region, and the processor(s) 130 can use language analysis operations (e.g., a large language model) to process the text input to identify boundaries of the region. As yet another example,QUALCOMM Ref. No. 2501497WO- 37 / 94 -the processor(s) 130 can identify the boundaries based on event detection. To illustrate, the processor(s) 130 can include a classifier (e.g., one or more artificial intelligence models) that is configured to classify the scene and automatically determine features or regions of the scene for inclusion in or exclusion from the gaussian scene representation based on the classification of the scene. As one specific example, when the scene is classified as a roadway intersection, vehicles may be designated as inclusion or exclusion objects.

[0108] In addition to, or instead of, designating boundaries of regions as described above, the same techniques can be used to designate objects as update inclusion objects or update exclusion objects. For example, based on output of a classifier, the processor(s) 130 can automatically determine (or propose to a user) objects that should be used to trigger updates of the gaussian scene representation. As another example, a user can select an object in the scene or provide text input designating the object, and the processor(s) 130 can analyze the input to detect the object and designate the object as an update exclusion or inclusion object.

[0109] In some cases, particular objects or regions can be designated for inclusion or exclusion as a default setting (which optionally can be overridden in some cases). For example, by default, the selective model update pipeline 140 can designate birds or leaves as update exclusion objects in outdoor scenes to avoid frequent updates which may not be relevant to the intended use of the gaussian scene representation 154.However, if the gaussian scene representation 154 is being used as a model of a natural environment, the default exclusion of birds or leaves may be overridden.

[0110] An “update exclusion region” refers to a region in the scene or a region in an input image where changes do not trigger an update of the gaussian scene representation 154. For example, if the only detected ROI associated with an input image is located in an update exclusion region, the update strategy selector 144 selects no update 152 irrespective of other considerations associated with the ROI. An “update inclusion region” refers to a region in the scene or a region in an input image where changes can (but do not necessarily) trigger an update of the gaussian scene representation 154. For example, if the only detected ROI associated with an input image is located in an update inclusion region, the update strategy selector 144 selects from among the availableQUALCOMM Ref. No. 2501497WO- 38 / 94 -update strategies based on one or more considerations described herein. To illustrate, the data analyzer 142 can determine one or more metrics (e.g., a size metric, a difference metric, or both) associated with the ROI and select an update strategy from among the full update 148, the partial update 150, or no update 152 based on the metric(s). The update inclusion regions, the update exclusion regions, or both, can be specified in terms of boundaries associated with the scene (e.g.. three dimensional volumes), pixel boundaries (e.g., two-dimensional areas) of images from a particular view, or both. In some cases, the update inclusion regions, the update exclusion regions, or both, can be tied to features of the scene. To illustrate, when the scene includes an arena with a playing field and spectator seats, the update target data 226 can include data identifying the spectator seats as an update exclusion region, data identifying the playing field as an update inclusion region, or both.[OHl] An “update exclusion object'’ refers to an object or object type that does not trigger an update of the gaussian scene representation 154. For example, if the only detected ROI associated with an input image corresponds to an update exclusion object, the update strategy selector 144 selects no update 152 irrespective of other considerations associated with the ROI. An “update inclusion object’' refers to an object or object type that can (but does not necessarily) trigger an update of the gaussian scene representation 154. For example, if the only detected ROI associated with an input image corresponds to an update inclusion object, the update strategy selector 144 selects from among the available update strategies based on one or more considerations described herein.

[0112] The update target data 226 can include any combination of update exclusion regions, update inclusion regions, update exclusion objects, and update inclusion objects. When the update target data 226 includes more than one type of data (e.g., an update exclusion region and an update inclusion object), the data analyzer 142 can be configured to consider the various data types in any order of priority. To illustrate, the data analyzer 142 can first check the location of the ROI against update inclusion and / or exclusion locations and can subsequently check objects detected in the ROI against update inclusion and / or exclusion objects. In this illustration, one type of update data may override another based on settings associated with the update strategy selector 144. For example, if spectator seats are designated as an update exclusion region, a ball isQUALCOMM Ref. No. 2501497WO- 39 / 94 -designated as an update inclusion object, and an input image includes an image of the ball leaving the playing field and entering the spectator seats, the update strategy selector 144 can select no update 152 because the ball is in an update exclusion region. Alternatively, the update strategy selector 144 can select a partial update 150 or a full update 148 because the ball is an update inclusion object. Different settings for the update strategy selector 144 can be established to emphasize desired tradeoffs between conserving computing resources (e.g., by limiting updates) and accurately modeling the scene (e.g., by updating more frequently).

[0113] The update strategy selector 144 is configured to select an update strategy based on output of the data analyzer 142. Thus, the update strategy selector 144 can select the update strategy based, at least in part, on the differences between input pixel data and the reference pixel data and results of the comparison of difference data associated with the difference pixels to the update target data 226. For example, based at least in part on data provided by the data analyzer 142, the update strategy selector 144 can signal to the update engines 146 to perform no update 152, the partial update 150, or the full update 148.

[0114] When the update strategy selector 144 indicates that the partial update 150 is to be performed, the update data selector 224 can identify which pixels of the input pixel data are to be used for the partial update 150, which gaussians 156 of the gaussian scene representation 154 are to be updated, or both. For example, the update data selector 224 can use the mapping data 122 to identify a subset of the gaussians 156 that correspond to (e.g., influence rendering of) pixels of a difference region. The pixels of the difference region can correspond to or include pixels of a region of interest, in which case pixel-to-gaussian mapping data 274 can be used to identify the subset of the gaussians 156. Alternatively, the pixels of the difference region can correspond to or include pixels associated with a particular object, in which case object mapping data 270 can be used to identify' the subset of the gaussians 156.

[0115] The update data selector 224 can also, or alternatively, use gaussian-to-pixel mapping data 276 of the mapping data 122 to identify pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156. Because each gaussian 156 can influence rendering of more than one pixel, the identified subset ofQUALCOMM Ref. No. 2501497WO- 40 / 94 -gaussians can influence more pixels than the pixels used to identify the subset of gaussians. Thus, for example, the pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156 can include pixels within the region of interest and one or more nearby pixels that are also influenced by the identified subset of the gaussians 156.

[0116] In some cases, the selective model update pipeline 140 includes a mapper 222 that is configured to generate or control the mapping data 122. For example, the mapper 222 can perform operations to generate the mapping data 122 (or portions thereof), to update the mapping data 122, to determine whether new mapping data 122 is to be generated, etc. As a particular example, the mapper 222 can perform operations associated with the staged approach to managing the mapping data 122 described with reference to FIG. 1.

[0117] In some embodiments, the update strategy selector 144 is configured to select the update strategy based on the differences between input pixel data and the reference pixel data and additional information. For example, the data analyzer 142 can analyze the sensor data 145 and provide analysis results to the update strategy selector 144. In this example, the update strategy selector 144 can select the update strategy based, at least in part, on the sensor data 145.

[0118] As one example, the update strategy selector 144 may select to perform no update 152 when the differences between input pixel data and the reference pixel data are insignificant (e.g., less than a first threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are insignificant, and / or when the sensor data 145 indicate that computing resources are strained (e.g., battery¬ level is low, available processing capacity is low, etc.). Continuing this example, the update strategy selector 144 may select to perform the full update 148 when the differences between input pixel data and the reference pixel data are large (e.g., greater than a second threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are significant, and / or when the sensor data 145 indicate that significant computing resources are available. Further, the update strategy selector 144 may select to perform the partial update 150 when the differences between the input pixel data and the reference pixel data are constrained to one or moreQUALCOMM Ref. No. 2501497WO- 41 / 94 -identified regions of interest (based on the image data 147, the sensor data 145. or both) and / or when the sensor data 145 indicate that computing resources are available but may be constrained.

[0119] Although the example above describes selecting an update strategy from among no update 152, the partial update 150, and the full update 148 based on relative states (e.g., significant, insignificant, large, constrained, etc.), in practice, the selection can be based on comparisons of various metrics to respective thresholds. For example, an “insignificant” difference or value can correspond to a value of a metric that is less than a first threshold, a “large” difference or value can correspond to a value of the metric that is greater than a second threshold, and a “significant” difference can correspond to a value of the metric between the first threshold and the second threshold. In some cases, more than two or fewer than two thresholds can be used, depending on the specific implementation.

[0120] When the update strategy selector 144 determines to perform the full update 148, all of the gaussians 156 of the gaussian scene representation 154 are eligible to be updated based on the input pixel data, and all of the input pixel data is used to perform the full update 148. For example, the update strategy selector 144 can provide the input pixel data to the update engines 146, and the update engines 146 can render views of the gaussian scene representation 154, determine loss function values based on comparison of the rendered views to ground-truth views, use the loss function values to calculate gradient values, and use the gradient values to determine updated parameter values 149 for each of the gaussians 156.

[0121] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all) of the gaussian scene representation 154 are updated based on the input pixel data. For example, the update data selector 224 can select a first subset of gaussians 156 that are to be modified, in which case a second subset of gaussians 156 are not to be modified and are to be held constant. The update data selector 224 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and a second subset of the input pixel data that are not to be used for the partial update 150.QUALCOMM Ref. No. 2501497WO- 42 / 94 -

[0122] In a particular aspect, the update data selector 224 selects the first subset of the gaussians 156 to be updated and the first subset of the input pixel data used for the partial update 150 based on differences between the input pixel data and reference pixel data, based on the sensor data 145, or both. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data 145 can be used to estimate the difference region. To illustrate, changes in lighting, as detected by the light sensor(s) 212, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by the motion sensor(s) 210 and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data.

[0123] The update data selector 224 can select the first subset of the gaussians 156 to be updated based on the pixels of the input pixel data to be used for the partial update 150 (e.g., based on the difference region). For example, the update data selector 224 can designate a subset of the gaussians 156 that correspond to or affect the pixels to be used for the partial update 150 as eligible to be modified by the partial update 150. In this example, other gaussians 156 of the gaussian scene representation 154 are ineligible to be modified during the partial update 150.

[0124] In many cases, there may not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, particular gaussians may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g., the view 136). Thus, in some cases, the update data selector 224 can use the difference region to identify the particular gaussians associated with pixels of the difference region. In such cases, the particular gaussians can be updated based on a selected set of pixels that include pixels of the difference region and additional pixels (typically near the difference region) that are influenced by the particular gaussians. In a particular aspect, the update data selector 224 (or the mapper 222) can use saliency mapping, look-up tables, or other mapping data to identify associations between pixels and gaussians of the gaussian scene representation 154.QUALCOMM Ref. No. 2501497WO- 43 / 94 -

[0125] In a particular aspect, results of operations performed by the system 200 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation with updated gaussians 156), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with a scene (e.g., the scene 132 of FIG. 1). For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of a device (e.g., the device 102 or another device), such as to turn on or turn off a display based on an aspect of the scene. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory or haptic notification or combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing orQUALCOMM Ref. No. 2501497WO- 44 / 94 -suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene.

[0126] FIGS. 3A, 3B, and 3C, together, depict a system 300 that illustrates particular aspects of the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The system 300 illustrates additional details related to updating a gaussian scene representation 154, and in particular, illustrates differences between operations associated with performing a full update (e.g., the full update 148 of FIGS. 1 and 2) and a partial update (e.g.. the partial update 150 of FIGS. 1 and 2). Due to the complexity of the system 300, the system 300 is split over three pages and linked by off-page references.

[0127] The system 300 includes one or more sensors 302. such as a first sensor 302A, a second sensor 302B, and a Pth sensor 302P (where P is an integer greater than two). At least one of the sensor(s) 302 is an image sensor (e.g., a camera) configured to generate image data. One or more others of the sensor(s) 302 can include or correspond to motion sensors, light sensors, device condition sensors, etc., such as described with reference to FIGS. 1 and 2. The sensor(s) 302 are configured to generate sensor data 303. The system 300 optionally includes a sensor controller 304 that is configured to control operation of the sensor(s) 302 and / or to select a subset of the sensor data 303 to generate sensor data 305, which is used for further processing. If the sensor controller 304 is omitted, the sensor(s) 302 can be externally controlled (e.g., responsive to user input), can operate according to a schedule, or can generate the sensor data 303 responsive to detecting a condition (e.g., in response to detecting movement). If the sensor controller 304 is omitted, the sensor data 305 is the same as the sensor data 303.

[0128] In FIG. 3A, the system 300 includes a pose estimator 306 that is configured to determine pose data (e.g., a position and / or orientation of one of the sensor(s) 302 within a scene), which can be provided along with the sensor data 305 as input 307 to an image subspace selector 312. In some cases, the sensor data 305 can include pose information (e.g., as data or metadata). In such cases, the pose estimator 306 is optional. If the pose estimator 306 is omitted, the sensor controller 304 (or the sensor(s) 302 if theQUALCOMM Ref. No. 2501497WO- 45 / 94 -sensor controller 304 is also omitted), provide the input 307 including the sensor data 305 and corresponding pose data to the image subspace selector 312.

[0129] During initial generation of the gaussian scene representation 154, the pose estimator 306, the sensor controller 304, or the sensor(s) 302 provide input 311 including sensor data 305 and corresponding pose data to a model initializer 310. which uses similar operations as described with reference to FIG. 3B for the full update 148 to generate initial parameters for the gaussians 156 of the gaussian scene representation 154. After initial generation of the gaussian scene representation 154, the model initializer 310 can be bypassed, and the input 307 can be provided to an image subspace selector 312.

[0130] In the example illustrated in FIG. 3 A, the gaussian scene representation 154 includes a plurality of gaussians 156, such as: a gaussian 156A, a gaussian 156B. and a gaussian 156Q (where Q is an integer greater than 2). Each gaussian 156 is characterized by a plurality of parameters describing various properties of the gaussian 156. For example, in FIG. 3 A, the gaussian 156A is characterized by an X-axis center position coordinate (‘‘center pos. (X)”) 180, Y-axis center position coordinate (“center pos. (Y)”) 181. and Z-axis center position coordinate (“center pos. (Z)”) 182. Together, the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 indicate a center position of the gaussian 156A and correspond to the position vector, p,, discussed above with reference to Equations 1-10. In FIG. 3 A, the gaussian 156A is also characterized by an X-axis scale parameter 183, a Y-axis scale parameter 184, and a Z-axis scale parameter 185.Together the X-axis scale parameter 183, the Y-axis scale parameter 184, and the Z-axis scale parameter 185 indicate a size and shape of the ellipsoid of the gaussian 156A and correspond to the scale vector, si, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO) 186, rotation (ql) 187, rotation (q2) 188, rotation (q3) 189) specified as quaternions in the example illustrated in FIG. 3A. Together the rotation parameters 186-189 indicate a spatial orientation of the ellipsoid of the gaussian 156A and correspond to the orientation parameters, qi, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by an opacity parameter 190, which indicates a maximum opacity of the gaussian 156A. The opacity parameter 190QUALCOMM Ref. No. 2501497WO- 46 / 94 -corresponds to the opacity parameter,discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by a set of color parameters, e.g., color parameters 191, 192, and 193 in FIG. 3 A. The color parameter(s) 191-193 can be specified as a set of spherical harmonics (SH) coefficients, H as discussed above with reference to Equations 1-10. Although FIG. 3A illustrates the gaussian 156A as characterized by a set of fourteen parameters, in other examples, the gaussians 156 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters 191-193 can be used, depending on the order of the SH coefficients used to represent the colors of the gaussians 156. To illustrate, in some implementations, a reduced set of approximately 14 parameters may be used, where the color parameters 191-193 are simplified to a single base color value (e.g., an RGB triplet) rather than a full set of spherical harmonics coefficients. In such implementations, the spherical harmonics coefficients may be omitted entirely, reducing memory and computational requirements for storing and updating the gaussians 156.

[0131] The image subspace selector 312 is configured to, based on the input 307, identify one or more patches of pixels (e.g., image patches) for a region of interest (ROI), or more than one ROI, associated with the sensor data 305 to generate ROI data 313. In aparticular aspect, the image subspace selector 312 can correspond to, include, or be included within the data analyzer 142 of FIGS. 1 and 2. For example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to an object that is in motion in the scene (e.g.. based on motion data from a motion sensor, based on optical flow data from two or more images, etc.). In this example, the ROI can correspond to the image patch (or patches) associated with the object that is in motion. As another example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to pixels of input pixel data of the sensor data 303 or 305 that differ (by at least a threshold amount) from pixels of reference pixel data. In this example, the ROI can be defined by the location of the image patch (or patches) associated with the pixels that differ between the input pixel data and the reference pixel data. The ROI data 313 identifies one or more sets of pixels of one or more images.QUALCOMM Ref. No. 2501497WO- 47 / 94 -

[0132] The update strategy selector 144 of the system 300 is configured to select an update strategy based on the strategy selection data 320. The strategy selection data 320 can include the sensor data 303 or 305, the input 307, the ROI data 313, results of the comparison of the ROI data 313 to update target data, other data, or a combination thereof. For example, the update strategy selector 144 can select the update strategy' using any of the techniques described with reference to FIGS. 1 and 2 and / or any of the techniques described with reference to FIGS. 4-8. The update strategy selector 144 is configured to provide a selection indication 321 to the update engines 146, where the selection indication 321 causes the update engines 146 to initiate operations associated with the selected update strategy. For example, if the selection indication 321 indicates that a full update strategy is selected, the update engines 146 initiate the full update 148, and if the selection indication 321 indicates that a partial update strategy is selected, the update engines 146 initiate the partial update 150. If no update is selected, the update strategy selector 144 can omit generation of the selection indication 321 or can generate a “no update’" selection indication.

[0133] When the update engines 146 select the full update 148, the update engines 146 cause a pose selector 324 of FIG. 3B (as indicated by off-page reference A) to begin operations associated with the full update 148. The pose selector 324 is configured to select a pose 325 from training data 360 that includes a set of training images (e.g., images 361 A-361K) and pose data (e.g., poses 362A-362K) associated with each image 361A-361K (where K is an integer greater than two). An image associated with the selected pose 325 is used as a ground-truth image 349 for loss calculation after a view associated with the pose 325 is rendered.

[0134] The pose 325 describes the position and orientation of a virtual camera 330 relative to the gaussians 156 of the gaussian scene representation 154 to be rendered as a view. Due to the pose 325 and intrinsic parameters (e.g., focal length, principal point, etc.) of the virtual camera 330, some of the gaussians (e.g., gaussians 331) are visible to the virtual camera 330, and some of the gaussians (e.g., gaussians 333) are not visible to the virtual camera 330. For example, for a pose 325A. a virtual camera 330A can see gaussians 331 A; however, gaussians 333A are not visible to the virtual camera 330A in the pose 325 A. As another example, for a pose 325K, a virtual camera 330K can seeQUALCOMM Ref. No. 2501497WO- 48 / 94 -gaussians 331K; however, gaussians 333K are not visible to the virtual camera 330K in the pose 325K.

[0135] To ensure that the full update 148 updates each gaussian 156 based on all relevant data, the full update 148 generally entails rendering a view for every available pose (e.g., poses 362A to 362K) and comparing each rendered view to the ground-truth image (e.g., each of images 361 A to 361K) associated with the pose 362 used to render the view. Thus, performing the full update 148 is similar to operations performed by the model initializer 310 to generate an initial version of the gaussian scene representation 154 based on a set of images except that the full update 148 is initialized from a preexisting gaussian scene representation 154.

[0136] A projector 326 identifies the gaussians 331 that are visible to the virtual camera 330 for a pose 325. The projector 326 performs gaussian splatting, which entails projecting the visible gaussians 331, which are three dimensional, onto a 2D image plane 332 based on the pose 325 of the virtual camera 330 and the intrinsic parameters of the virtual camera 330. Each gaussian 331 is represented as an ellipse 335 in the 2D image plane 332. The ellipses 335 can overlap one another and can have various sizes and orientations depending on parameters of the gaussians 331 that are projected and the pose 325 of the virtual camera 330. For example, the gaussians 331 A that are visible to the virtual camera 330A in the pose 325A are projected on to a 2D image plane 332A as ellipses 335A, and the gaussians 331K that are visible to the virtual camera 330K in the pose 325K are projected on to a 2D image plane 332K as ellipses 335K

[0137] Data 334 describing the ellipses 335 on the 2D image plane 332 is provided to a differentiable tile rasterizer 340. For example, data 334A describes the ellipses 335A associated with the pose 325A, and data 334B describes the ellipses 335B associated with the pose 325K. The differentiable tile rasterizer 340 is configured to perform operations to blend the ellipses 335 to generate pixel data representing a 2D image that would be captured by the virtual camera 330 for the particular selected pose 325. To generate the pixel data, the differentiable tile rasterizer 340 divides the 2D image plane 332 into tiles (represented in FIG. 3B by bold lines and different shading in the 2D image planes 332A and 332K of the differentiable tile rasterizer 340). For purposes of this description, each tile is treated as corresponding to a single pixel; however, theQUALCOMM Ref. No. 2501497WO- 49 / 94 -exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel. Additionally, to simplify the drawings, in FIG. 3B, the 2D image plane 332 includes some regions that are not associated with any gaussian; however, it will generally be the case that each tile is associated with at least one gaussian 156 to provide color information for the tile.

[0138] The differentiable tile rasterizer 340 obtains depth information 342 for the visible gaussians 331 of the pose 325, where depth refers to the distance in the scene of a gaussian from the virtual camera 330. For example, the depth information 342 can be calculated for each of the visible gaussians 331 based on the location of the virtual camera 330 in the scene (as indicated by data describing the pose 325) and the location of the center of the gaussian in the scene (e.g., as indicated by the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 of each gaussian). For example, depth information 342A indicates distances in the scene between each of the gaussians 331 A and the virtual camera 330A, and depth information 342K indicates distances in the scene between each of the gaussians 33 IK and the virtual camera 330K.

[0139] The differentiable tile rasterizer 340 can perform tile-depth association 344 to, for each tile of the 2D image plane 332, sort the ellipses 335 that at least partially overlap the tile based on the depth information 342. Since the gaussians 156 are generally not opaque, a first gaussian that is more distant from the virtual camera 330 in the scene can be partially visible through a second gaussian that is closer to the virtual camera 330. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 340 accounts for the colors of each gaussian of a set of gaussians that influence the particular pixel, the opacities of the gaussians of the set, and the order of the gaussians of the set as viewed from the virtual camera 330.

[0140] To parallelize generation of the pixel data, the differentiable tile rasterizer 340 can perform thread association 346 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources, such computing resources can include one or more cores of one or more central processing unit (CPU), one or more cores of one or more graphicsQUALCOMM Ref. No. 2501497WO- 50 / 94 -processing unit (GPU), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colors of the gaussians (as determined based on the pose 325 of the virtual camera 330 and color parameters specified by spherical harmonic color coefficients, opacity of the gaussians, and the depth information 342).

[0141] Blending the contributions of the gaussians results in pixel data 341 representing an image (e.g., a view) captured by the virtual camera 330 for the particular pose 325. The pixel data 341 is provided to a loss calculator 348, which is configured to compare the pixel data 341 to the ground-truth image 349 corresponding to the pose 325. The loss calculator 348 determines a loss value for each pixel (e.g., a photometric L2 loss) to quantify how the pixel data 341 differs from the ground-truth image 349. Gradient values 343. 345 based on the loss values are backpropagated through the differentiable tile rasterizer 340 and the projector 326 to estimate updated parameter values 347 for the visible gaussians 331 of the pose 325.

[0142] The full update 148 continues by iteratively repeating the processes of pose selection, projection, rasterization, loss calculation, and backpropagation of gradients until all of the poses 362 have been processed. At one or more points during the full update 148, an adaptive density controller 350 performs operations that change the number of gaussians 156 of the gaussian scene representation 154, such as pruning one or more gaussians 156 (e.g., the gaussian 156B) or splitting one or more gaussians 156. Thus, in some senses, a different gaussian scene representation 154 exists after the full update 148 than existed before the full update 148 since a different number and arrangement of gaussians 156 can be present, and any gaussian 156 that is carried over can have different parameters. For example, in FIG. 3B, the gaussian scene representation 154 is illustrated as including a first gaussian 156A, a second gaussian 156B, and one or more additional gaussians up to a Qth gaussian 156Q (where Q is an integer greater than 2). After the full update 148, the gaussian scene representation 154 can include more than Q or fewer than Q gaussians 156. Further, the full update 148 is a feedback-based process since, during each iteration, the visible gaussians 331 for a pose 325 are updated based on loss values indicating differences between an image (e.g., theQUALCOMM Ref. No. 2501497WO- 51 / 94 -pixel data 341) of a view rendered for the pose 325 and a reference image (e.g., the ground-truth image 349 associated with the pose 325).

[0143] When the update engines 146 of FIG. 3 A select the partial update 150, the update engines 146 cause a gaussian selector 354 of FIG. 3C (as indicated by off-page reference B) to select a set of gaussians (e.g., gaussians 156A and 156B in the example of FIG. 3C) that are to be updated based on the ROI data 313 and mapping data 122. As an example, for illustrative purposes, the sensor data 303 of FIG. 3 A can include an image 351 of FIG. 3C. In this illustrative example, the image subspace selector 312 can determine that the ROI includes a ball 365 depicted in the image 351. in which case the ROI data 313 can include data identifying the ball 365. The data identifying the ball 365 can include object identification data (e.g., a text label or a numerical designator), a set of pixels that represent the ball 365 in the image 351, identifiers of the set of pixels that represents the ball 365 in the image 351. or other data that enables the gaussian selector 354 to select the set of gaussians based on the mapping data 122 and the ROI data 313. In a particular aspect, the image subspace selector 312, the gaussian selector 354, or both, are integrated within the update data selector 224 of FIG. 2.

[0144] The gaussian selector 354 uses the ROI data 313 and the mapping data 122 to generate data 353 identifying the selected set of gaussians, and provides the data 353 to a partial updater 356. The data 353 identifying the selected set of gaussians can include the gaussians themselves, memory addresses of the selected set of gaussians, indices of the selected set of gaussians. or other information that enables the partial updater 356 to obtain the selected set of gaussians.

[0145] The partial updater 356 is configured to determine updates 357 to the selected set of gaussians (leaving remaining gaussians 156 of the gaussian scene representation 154 unchanged) to generate updated gaussians 156’. In some implementations, the partial updater 356 can use a feedforward approach to determine the updates 357. For example, the partial updater 356 can determine the updates 357 based on the sensor data 305 of FIG. 3 A. As one example, the sensor data 303 can include motion data that can be used to determine how far an object (e.g., the ball 365) in the scene moved, and the partial updater 356 can determine the updates 357 based on the determined movement of the object. As another example, optical flow can be used to compare the image 351 toQUALCOMM Ref. No. 2501497WO- 52 / 94 -one or more reference images to identify movement of one or more objects in the scene, and the partial updater 356 can determine the updates 357 based on the identified movement. As yet another example, the sensor data 303 can include light data which can be used to determine changes in lighting or color in a portion of the scene, and the partial updater 356 can determine the updates 357 based on the changes in lighting or color.

[0146] The partial updater 356 can apply the updates 357 to the selected set of gaussians and store updated versions of the selected set of gaussians (e.g., updated gaussians 156A’ and 156B’ in FIG. 3C) in an update buffer 358. After all gaussians of the selected set of gaussians have been updated, the updated gaussians 156’ can be written to a memory where the gaussian scene representation 154 is stored.

[0147] Optionally, in some embodiments, a rendering-based error calculator 359 can be used to check the updated gaussians 156’ before the gaussian scene representation 154 is modified. The rendering-based error calculator 359 is configured to perform projection, rendering, and error calculation, as described with reference to the projector 326, the differentiable tile rasterizer 340, and the loss calculator 348, respectively, of FIG. 3B to generate a rendered view of the gaussian scene representation 154. During the partial update 150, the rendering-based error calculator 359 can determine (e.g. obtain from the sensor data 305 or from the pose estimator 306 of FIG. 3 A) a pose associated with the image 351. The rendering-based error calculator 359 can render a view of the gaussian scene representation 154 (with the updated gaussians 156’) from the pose associated with the image 351, and compare the rendered view to the image 351 to determine whether the updated gaussians 156’ are acceptable or whether additional updates 357 should be determined. For example, the rendering-based error calculator 359 can determine loss value(s) (e.g., using similar operations as described with reference to the loss calculator 348 of FIG. 3B) based on differences between the image 351 and the rendered view. In this example, if the loss value(s) satisfy a threshold, the updated gaussians 156’ are used to update the gaussian scene representation 154. However, if the loss value(s) do not satisfy a threshold, the partial updater 356 generates a second set of updated gaussians 156’ and repeats the process above to determine whether loss value(s) for the second set of updated gaussians 156’ satisfy the threshold. This process can be repeated iteratively until a terminationQUALCOMM Ref. No. 2501497WO- 53 / 94 -condition is reached. The termination condition can include identification of a set of updated gaussians 156' that satisfies the threshold, or performance of a specified number of iterations. If no set of updated gaussians 156’ that satisfies the threshold is identified after the specified number of iterations, the partial updater 356 can change to another approach to generate updates 357, or the update strategy selector 144 of FIG.3A can select a different update strategy (e.g., no update, a different type of partial update, or the full update 148). In some examples, rather than process one set of updated gaussians 156’ at a time, as described above, the partial updater 356 can generate multiple candidate sets of updated gaussians 156’ in parallel, and check which, if any, best match the image 351.

[0148] As one specific example, in FIG. 3C, the image 351 represents a portion of a soccer pitch and includes a goal and a ball 365. FIG. 3C also illustrates an example 370A of the gaussian scene representation 154 before the partial update, and an example 370B of the gaussian scene representation 154 after the partial update.

[0149] In this example, the ROI data 313 can include object data identifying an object (e.g., the ball 365) that has changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the object mapping data 270 of FIG. 2, which identifies a set of the gaussians 156 that represent the object, and the gaussian selector 354 can select the set of gaussians that represent the object as the set of gaussians that are to be updated. In the specific example shown in FIG. 3C, the gaussians 156A and 156B represent a ball 367 in the scene; thus, the data 353 includes the gaussians 156 A and 156B.

[0150] As another example, the ROI data 313 can identify a first set of pixels that have changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the pixel-to-gaussian mapping data 274 of FIG. 2, which identifies a set of the gaussians 156 that correspond to the first set of pixels, and the gaussian selector 354 can select the set of gaussians that correspond to the first set of pixels as the set of gaussians that are to be updated. The selection data 353 can include the selected set of gaussians or information indicating how the partial updater 356 can access the selected set of gaussians, such as memory addresses of the selected set of gaussians, indices of the selected set of gaussians, etc.QUALCOMM Ref. No. 2501497WO- 54 / 94 -

[0151] The partial updater 356 is configured to modify parameters of the selected set of gaussians (e.g.. the gaussians 156A and 156B in this example) based on the sensor data 305, the input 307, and optionally other data. For example, when the ROI data 313 indicates that an object in the scene has moved, the partial updater 356 can change location parameters and / or orientation parameters of the gaussians 156A and 156B that represent the object in the gaussian scene representation 154 in order to update the gaussian scene representation 154 to reflect movement of the object. In other cases, the partial updater 356 can change color parameters (e.g., spherical harmonics coefficients) and / or opacity' parameters of the gaussians 156A and 156B to represent lighting changes in the scene.

[0152] A technical benefit of performing the partial update 150 rather than the full update 148 is that significant computer resources can be conserved. For example, when the update to be performed can be calculated or estimated based on the sensor data 305, the rendering operations associated with the projector 326 and the differentiable tile rasterizer 340, the loss calculations performed by the loss calculator 348, and backpropagation of gradients 343, 345 based on the loss values can be omitted during the partial update 150. Further, even if the partial update 150 uses the rendering-based error calculator 359 to check one or more candidate updated gaussians 156’, only one or a few views are rendered, rather than views associated with all of the images 361 of training data 360 of FIG. 3B, saving significant rendering time and resources. Further, backpropagation of gradients based on the loss values can be omitted during the partial update 150, which also results in conservation of significant computing resources.

[0153] FIG. 3D illustrates particular aspects of the system 300 of FIGs. 3A-3C, in accordance with some examples of the present disclosure. In particular, FIG. 3D illustrates aspects related to comparison of ROI data and update target data 226. FIG.3D illustrates an example of an input image (e.g., the image 351) that includes a portion of a playing field including the ball 365 and a person 381. FIG. 3D also illustrates the image subspace selector 312, the update target data 226, a comparator 393, the strategy' selection data 320, and the update strategy selector 144, which is configured to select among various update strategies, such as the full update 148, the partial update 150, and no update 152. In FIG. 3D, the image subspace selector 312 optionally includes an object detector 397 that is configured to detect particular objects in the image 351. InQUALCOMM Ref. No. 2501497WO- 55 / 94 -other implementations, the object detector 397 is omitted or is not part of the image subspace selector 312. To illustrate, the object detector 397 can be a component of the selective model update pipeline 140 of FIG. 1 or can be integrated within the data analyzer 142 of FIG. 1. In FIG. 3D, the comparator 393 is shown outside the image subspace selector 312 (e.g., as a separate component of the selective model update pipeline 140 of FIG. 1); however, in other implementations, the comparator 393 is part of the image subspace selector 312.

[0154] In FIG. 3D, the image 351 is annotated to show examples of an update inclusion region 382 and an update exclusion region 383. To illustrate, in FIG. 3D, the update inclusion region 382 corresponds to an area around a goal of a playing field, and the update exclusion region 383 corresponds to an area outside the update inclusion region 382 (such as an out of bounds area of the playing field). In this example, changes in the scene within the update exclusion region 383 do not trigger updates to the gaussian scene representation 154 (unless such updates are indicated by other update targets). For example, the image subspace selector 312 is configured to include information indicating location(s) of ROI(s) of the image 351 as part of the ROI data 313. As an illustrative example, the ROI data 313 can include information designating the location of the ball 365 as an ROI. The comparator 393 can compare the location of the ball 365 to update target data 226 identifying update inclusion regions 391, update exclusion regions 390, or both.

[0155] If the ROI data 313 indicates that the ball 365 is in one of the update inclusion regions 391, the comparator 393 adds information to the strategy selection data 320 to enable the update strategy selector 144 to choose among the full update 148, the partial update 150, or no update 152 of the gaussian scene representation 154 based on various factors. If the ROI data 313 indicates that the ball 365 is in one of the update exclusion regions 390, the comparator 393 does not add data associated with the ball 365 to the strategy selection data 320, resulting in the update strategy selector 144 choosing no update 152 of the gaussian scene representation 154 due to changes associated with the ball 365 in the update exclusion region 390.

[0156] In some cases, the update target data 226 does not include data specifying the update inclusion regions 391 and the update exclusion regions 390 or the updateQUALCOMM Ref. No. 2501497WO- 56 / 94 -inclusion regions 391 and the update exclusion regions 390 are not decisive as to an update strategy to be used. For example, if no update exclusion regions 390 are specified, or if all detected ROIs are in update inclusion regions 391, the update strategy selector 144 selects among the available update strategies based on other factors as described above.

[0157] In some cases, the update target data 226 specifies the update exclusion objects 395, the update inclusion objects 396, or both. In such cases, the update target data 226 can optionally also specify the update inclusion regions 391, the update exclusion regions 390, or both. Using the image 351 as an example, the person 381, the ball 365, or both, can correspond to examples of update inclusion objects 396 or update exclusion objects 395. In this example, the object detector 397 is configured to include object identifiers of objects detected in the image 351 as part of the ROI data. As an illustrative example, the ROI data 313 can include an identifier of the ball 365 to indicate that the ball 365 is detected in the image 351 and is associated with an ROI. The comparator 393 can compare the identifier of the ball 365 to the update target data 226 identifying the update inclusion objects 396, the update exclusion objects 395, or both.

[0158] If the ball 365 matches one of the update inclusion objects 396, the comparator 393 adds information to the strategy selection data 320 to enable the update strategy selector 144 to choose among the full update 148, the partial update 150, or no update 152 of the gaussian scene representation 154 based on various factors. If the ball 365 matches one of the update exclusion objects 395, the comparator 393 does not add data associated with the ball 365 to the strategy selection data 320, resulting in the update strategy selector 144 choosing no update 152 of the gaussian scene representation 154 due to changes associated with the ball 365. It is noted that if other changes in the scene are sufficient to cause the update strategy selector 144 to select the full update 148. the full update 148 can optionally include changes associated with the ball 365 even though the ball 365 is an update exclusion object 395. Similarly, if the person 381 is an update inclusion object 396, the update strategy selector 144 can choose among the full update 148, the partial update 150, or no update 152 of the gaussian scene representation 154 based on various factors; whereas, if the person 381 is an update exclusion object 395, the update strategy selector 144 chooses no update 152 of the gaussian scene representation 154 due to changes associated with the person 381.QUALCOMM Ref. No. 2501497WO- 57 / 94 -

[0159] If the update target data 226 includes update data related to both objects and regions (e.g.. includes data specifying update exclusion regions 390 and / or update inclusion regions 391 and also includes data specifying update exclusion objects 395 and / or update inclusion objects 396), then the update target data 226 can also include data indicating an order or priority associated with the various update data. To illustrate, update exclusion regions 390 may be prioritized over update data related to objects. In this illustrative example, if an ROI associated with an object that matches one of the update inclusion objects 396 is identified in an update exclusion region 390, the comparator 393 does not add information related to the object to the strategy selection data 320, resulting in no update 152 being triggered based on the ROI associated with the object. As another illustrative example, update inclusion objects 396 may be prioritized over update data related to regions. In this illustrative example, if an ROI associated with an object that matches one of the update inclusion objects 396 is identified in an update exclusion region 390, the comparator 393 adds information related to the object to the strategy selection data 320. and the update strategy selector 144 selects from among the full update 148, the partial update 150, and no update 152. Depending on goals or priorities associated with the gaussian scene representation 154, any of the update target data types can be prioritized higher than or lower than any other update target data type.

[0160] A technical benefit of selecting an update strategy for a gaussian scene representation based on update targets is that certain types of changes in a scene can be ignored, which can conserve computing resources when such changes are not important to the particular purposes for which the gaussian scene representation is maintained.

[0161] FIG. 4 is a flow chart illustrating particular aspects of a method 400 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 400 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 400 can be initiated, performed, or controlled by the controller 160.QUALCOMM Ref. No. 2501497WO- 58 / 94 -

[0162] The method 400 includes, at block 402, obtaining input pixel data. For example, the input pixel data can correspond to. include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1. As another example, the input pixel data can correspond to, include, or be included within image data 147 from the image source(s) 106 of FIG. 2. As another example, the input pixel data can correspond to. include, or be included within the sensor data 305 or the ROI data 313 of FIG. 3 A.

[0163] The method 400 includes, at block 404, based at least in part on differences between the input pixel data and reference pixel data, selecting, from among a plurality’ of update strategies, an update strategy' for the gaussian scene representation. The reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation. For example, the update strategy’ selector 144 can receive information indicating a level of differences between the input pixel data and reference pixel data from the data analyzer 142 and can select to perform no update 152. the partial update 150, or the full update 148 based on the level of differences.

[0164] FIG. 5 is a flow chart illustrating particular aspects of a method 500 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance yvith some examples of the present disclosure. For example, the method 500 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 500 can be initiated, performed, or controlled by the controller 160.

[0165] The method 500 includes, at block 506, determining difference(s) between input pixel data 502 and reference pixel data 504. The input pixel data 502 can correspond to, include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1; image data 147 from the image source(s) 106 of FIG. 2; or the sensor data 305 or the ROI data 313 of FIG. 3 A. The reference pixel data 504 can correspond to, include, or be included within the reference image(s) 124 (e.g., the training image(s) 126 and / or the synthetic image(s) 128) of FIG. 1.

[0166] In a particular aspect, determining the difference(s) between the input pixel data 502 and the reference pixel data 504 includes determining value(s) of one or moreQUALCOMM Ref. No. 2501497WO- 59 / 94 -difference metrics. For example, the input pixel data 502 can be compared (e.g., pixel-by-pixel or region-by-region) to the reference pixel data 504 to identify pixel values that differ significantly (e.g., by more than a threshold amount) between corresponding pixels or regions of the input pixel data 502 and the reference pixel data 504. Locations associated with significant differences between the input pixel data 502 and the reference pixel data 504 can be identified as difference regions (or regions of interest), and a value of a difference metric can be determined for each difference region. In various examples, the difference metric can indicate a size of or dimensions of the difference region, a count of pixels within the difference region, a magnitude of difference between pixels of the difference region (e.g., based on distance between points in a color space of the pixels), or a combination thereof.

[0167] The method 500 also includes, at block 508, selecting an update strategy based on the difference(s) between the input pixel data 502 and the reference pixel data 504. For example, the update strategy can be selected by the update strategy selector 144 of any of FIGS. 1 to 3 A. The update strategy can be selected, for example, from among no update 552, a partial update 550, or a full update 548. As an example, the update strategy can be selected based on a number (e.g., a count) of identified difference regions, a size of one or more difference regions, a magnitude of difference associated with one or more difference regions, a location of a difference region within an image (e.g., in the input pixel data 502 or in the reference pixel data 504), a location within a scene (e.g., a three-dimensional location) associated wi th a difference region, or a combination thereof.

[0168] In some cases, the update strategy can be selected based on other factors in addition to the difference(s) between the input pixel data 502 and the reference pixel data 504. For example, the update strategy can be selected based, in part, on device condition information. To illustrate, for a first set of differences between the input pixel data 502 and the reference pixel data 504, the partial update 550 may be selected (based on the first set of differences) if a device updating the gaussian scene representation has a first device condition (e.g., is plugged into an external power source) and no update may be selected if the device has a second device condition (e.g., is operating on battery power).QUALCOMM Ref. No. 2501497WO- 60 / 94 -

[0169] FIG. 6 is a flow chart illustrating particular aspects of a method 600 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 600 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 600 can be initiated, performed, or controlled by the controller 160.

[0170] The method 600 includes, at block 606, determining a value (x) of a difference metric based on differences between input pixel data 602 and reference pixel data 604. The input pixel data 602 can correspond to, include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1; image data 147 from the image source(s) 106 of FIG. 2; or the sensor data 305 or the ROI data 313 of FIG. 3 A. The reference pixel data 604 can correspond to, include, or be included within the reference image(s) 124 (e.g., the training image(s) 126 and / or the synthetic image(s) 128) of FIG. 1.

[0171] Determining the value of the difference metric (x) based on the differences between the input pixel data 602 and the reference pixel data 604 can include, for example, comparing pixels or regions of the input pixel data 602 to corresponding pixels or regions of the reference pixel data 604 to identify one or more difference regions. The value of the difference metric can indicate a size of or dimensions of the difference region(s), a count of pixels within the difference region(s). a magnitude of difference between pixels of the difference region(s) (e.g., based on distance between points in a color space of the pixels), or a combination thereof.

[0172] If more than one difference region is identified, a value of the difference metric can be determined for each difference region or a single representative value of the difference metric can be determined for all of the difference metrics together. To illustrate, the representative value of the difference metric can correspond to a largest value of the difference metrics associated with the difference regions, a smallest value of the difference metrics associated with the difference regions, an average value of the difference metrics associated with the difference regions, a value of the difference metrics associated with a particular one of the difference regions (e.g., a differenceQUALCOMM Ref. No. 2501497WO- 61 / 94 -region associated with particular characteristics or associated with particular sensor data), etc.

[0173] The method 600 also includes, at block 608, selecting an update strategy' based on the value(s) of the difference metric(s). For example, the value(s) of the difference metric(s) can be compared to one or more thresholds. In the example illustrated in FIG.6, the value(s) of the difference metric(s) are compared to a first threshold and a second threshold. In this example, if a value of the difference metric is less than the first threshold (e.g., x < 1st Threshold), the selected update strategy' is no update 652. If the value of the difference metric is greater than the second threshold (e.g., x > 2nd Threshold), the selected update strategy is a full update 648. If the value of the difference metric is between the first threshold and the second threshold (e.g., 1st Threshold < x < 2nd Threshold), the selected update strategy is a partial update 650.

[0174] In some cases, the update strategy can be selected based on other factors in addition to the value(s) of the difference metric(s). For example, the update strategy can be selected based on the value(s) of the difference metric(s) and based on device condition information. To illustrate, a lower value of the second threshold can be set when significant computing resources are available to update the gaussian scene representation, and a higher value of the second threshold can be set when the available computing resources are more constrained.

[0175] FIG. 7 is a flow chart illustrating particular aspects of a method 700 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 700 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 700 can be initiated, performed, or controlled by the controller 160.

[0176] The method 700 includes, at block 706, identifying a difference region based on differences between input pixel data 702 and reference pixel data 704. The input pixel data 702 can correspond to, include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1; image data 147 from the image source(s) 106 of FIG. 2; or the sensor data 305 or the ROI data 313 ofQUALCOMM Ref. No. 2501497WO- 62 / 94 - FIG. 3 A. The reference pixel data 704 can correspond to, include, or be included within the reference image(s) 124 (e.g., the training image(s) 126 and / or the synthetic image(s) 128) of FIG. 1.

[0177] In a particular aspect, identifying the difference region includes identifying difference pixels by comparing the input pixel data 702 to the reference pixel data 704, where a difference pixel is associated with pixel values that differ significantly (e.g., by more than a threshold amount) between the input pixel data 702 and the reference pixel data 704. Locations associated with one or more difference pixels can be identified as difference regions. As one example, a difference region includes difference pixels that are within a specified distance from one another. To illustrate, clustering operations (e.g., density-based clustering) can be performed to group identified difference pixels into difference regions, based on distances between the difference pixels, or a distance between a difference pixel and a cluster of other difference pixels. In some cases, one or more clusters of difference pixels can be enclosed by a bounding box, and the bounding box can be identified as the difference region.

[0178] The method 700 also includes, at block 708, determining a size metric (d) based on dimensions of a difference region. To illustrate, the size metric can include an area of the difference region, a characteristic dimension of the difference region (e.g., a largest or an average diameter of multiple difference regions). As another example, the size metric can be determined based on a number (e.g., a count) of pixels associated with the difference region. To illustrate, the size metric can correspond to a count of the difference pixels or a count of all pixels (including difference pixels and non-difference pixels) within a bounding box that corresponds to the difference region.

[0179] If more than one difference region is identified at block 706, a size metric can be determined for each difference region, or a single representative size metric can be determined for all of the difference regions together. To illustrate, a representative size metric can correspond to a largest value of the size metrics associated with the difference regions, a smallest value of the size metrics associated with the difference regions, an average value of the size metrics associated with the difference regions, a value of the size metrics associated with a particular one of the difference regions (e.g..QUALCOMM Ref. No. 2501497WO- 63 / 94 -a difference region associated with particular characteristics or associated with particular sensor data), etc.

[0180] The method 700 also includes, at block 710, selecting an update strategy' based on the size metric. For example, the value(s) of the size metric can be compared to one or more thresholds. In the example illustrated in FIG. 7. the value(s) of the size metric are compared to a first threshold and a second threshold. In this example, if a value of the size metric is less than the first threshold (e.g., d < 1st Threshold), the update strategy selected is no update 752. If the value of the size metric is greater than the second threshold (e.g., d > 2nd Threshold), the update strategy selected is a full update 748. If the value of the size metric is between the first threshold and the second threshold (e.g., 1st Threshold < d < 2nd Threshold), the update strategy selected is a partial update 750. Specific values of the first and second thresholds can be set based on configuration data (e.g., user selections or settings), a specific purpose or use-case for the gaussian scene representation, available computing resources, other factors, or a combination thereof. As one specific example, the first threshold can be set within a range from about 1% to 10%, and the second threshold can be set within a range from about 10% to 30%, where the percentages are based on comparison of the size of the difference region(s) as compared to the size of each complete image. To illustrate, each image can include 1000x1000 pixels, and the difference metric can be determined for 10 images (i.e., 10 million total pixels). In this illustrative example, if the first threshold is set to 5%, then no update 752 is selected if the size metric indicates that the difference region is associated with fewer than 500,000 pixels. Further, in this illustrative example, if the second threshold is set to 20%, then the full update 748 is selected if the size metric indicates that the difference region is associated with more than 2 million pixels. In this illustrative example, the partial update 750 is selected if the size metric indicates that the difference region is associated with at least 500,000 pixels and no more than 2 million pixels.

[0181] In some cases, the update strategy can be selected based on other factors in addition to the value(s) of the size metric(s). For example, the update strategy can be selected based on the value(s) of the size metric(s) and based on device condition information. To illustrate, a lower value of the second threshold can be set when significant computing resources are available to update the gaussian sceneQUALCOMM Ref. No. 2501497WO- 64 / 94 -representation, and a higher value of the second threshold can be set when the available computing resources are more constrained.

[0182] FIG. 8 is a flow chart illustrating particular aspects of a method 800 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 800 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 800 can be initiated, performed, or controlled by the controller 160.

[0183] The method 800 includes, at block 806, identifying difference gaussians of a gaussian scene representation. The difference gaussians correspond to gaussians of the gaussian scene representation that are associated with differences between input pixel data 802 and reference pixel data 804. For example, the difference gaussians can include gaussians of the gaussian scene representation that influence rendering of pixels in a difference region, as described with reference to any of FIGS. 1-3C. The difference gaussians can be identified using pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both. In some cases, saliency mapping can be used to identify the difference gaussians.

[0184] The method 800 also includes, at block 808, determining a size metric (s) based on a count of the difference gaussians and a count of the gaussians of the gaussian scene representation. For example, the size metric can include or correspond to a ratio of difference gaussians to total gaussians of the gaussian scene representation. In this example, the size metric is indicative of a proportion of the gaussians of the gaussian scene representation that would be updated during a partial update.

[0185] The method 800 also includes, at block 810, selecting an update strategy based on the size metric. For example, the value(s) of the size metric can be compared to one or more thresholds. In the example illustrated in FIG. 8, the value(s) of the size metric are compared to a first threshold and a second threshold. In this example, if a value of the size metric is less than the first threshold (e g., s < 1st Threshold), the selected update strategy' is no update 852. If the value of the size metric is greater than the second threshold (e.g., s > 2nd Threshold), the selected update strategy is a full updateQUALCOMM Ref. No. 2501497WO- 65 / 94 - 848. If the value of the size metric is between the first threshold and the second threshold (e.g., 1st Threshold < s < 2nd Threshold), the selected update strategy is a partial update 850.

[0186] In some cases, the update strategy can be selected based on other factors in addition to the value(s) of the size metric(s). For example, the update strategy can be selected based on the value(s) of the size metric(s) and based on device condition information. To illustrate, a lower value of the second threshold can be set when significant computing resources are available to update the gaussian scene representation, and a higher value of the second threshold can be set when the available computing resources are more constrained.

[0187] Technical advantages of the methods 400, 500, 600, 700, and 800 include enabling updating of gaussian scene representations in a manner that is efficient in terms of computing resources utilized. For example, performing the full update uses significant computing resources and generally uses all of the input pixel data to perform calculations that account for all of the gaussians of the gaussian scene representation. In contrast, the methods 400, 500, 600, 700, and 800 enable selection of an update strategy (e.g., a partial update) that uses fewer calculations under particular circumstances, such as when the differences detected affect only a subset of the gaussians of the gaussian scene representation. Thus, fewer calculations can be used to keep the gaussian scene representation updated based on the input pixel data, resulting in savings of computing resources (e.g., processor time, power, and memory). Further, when the difference associated with the input pixel data are small, the methods 400, 500, 600, 700, and 800 can decide to perform no update, in which case even the computing resources associated with performing the partial update are conserved.

[0188] FIG. 9 depicts an implementation 900 of the device 102 of FIG. 1 as an integrated circuit 902 that includes the one or more processors 130. The integrated circuit 902 also includes one or more input connectors 906 (e.g., pins, contacts, leads, solder balls, etc.) to enable input data 904 to be received for processing. For example, the input data 904 can correspond to, include, or be included within, the image data 108, the sensor data 112, the data 116, the reference images 124, the mapping data 122, the gaussian scene representation 154, or other data described with reference to FIG. 1; theQUALCOMM Ref. No. 2501497WO- 66 / 94 -reference pixel data or other data described with reference to FIG. 2; the sensor data 303, the sensor data 305, the input 307, the ROI data 313, the strategy selection data 320, the mapping data 122, the gaussians 156, the ground-truth image 349 of FIGS. 3A-3C; or a combination thereof.

[0189] The integrated circuit 902 also includes one or more output connectors 908 (e.g., pins, contacts, leads, solder balls, etc.) to enable output data 910 to be sent to another device or component. For example, the output data 910 can correspond to, include, or be included within, the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the mapping data 122, the view 136, another output of the selective model update pipeline 140, or other data described with reference to FIG. 1; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154 or other data described with reference to FIG. 2; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the selection indication 321. the pixel data 341, the gradient values 343, the updated parameter values 347, or other data described with reference to FIGS. 3A-3C; or a combination thereof.

[0190] The processor(s) 130 of the integrated circuit 902 are configured to initiate, perform, or control various operations described with reference to FIGS. 1-8. For example, in FIG. 9, the processor(s) 130 of the integrated circuit 902 include the selective model update pipeline 140 (or components thereof) and the gaussian scene representation 154 (or components thereof, e.g., a subset of the gaussians 156 of FIG.1). Optionally, the processor(s) 130 can also include the renderer(s) 158, the view selector 162, or both.

[0191] The integrated circuit 902 enables implementation of selective partial gaussian scene representation updating as a component in a system that includes or is coupled to one or more image sources, such as a mobile phone or tablet as described with reference to FIG. 10; a mixed reality or augmented reality' glasses device as described with reference to FIG. 11; a wearable electronic device as described with reference to FIG.12; a voice-controlled speaker system described with reference to FIG. 13; a camera described with reference to FIG. 14; a virtual reality, mixed reality, or augmented realityQUALCOMM Ref. No. 2501497WO- 67 / 94 -headset described with reference to FIG. 15; or a vehicle described with reference to FIG. 16 or FIG. 17.

[0192] FIG. 10 depicts an implementation 1000 in which the device 102 includes a mobile device 1002, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 1002 includes one or more microphones 1010. one or more speakers 1006, one or more cameras 1008, and a display screen 1004. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the renderer 158 and the view selector 162, are integrated in the mobile device 1002 and are illustrated using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1002.

[0193] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1008 of the mobile device 1002 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1008 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as whether the mobile device 1002 is connected to an external power source, a battery state of the mobile device 1002, availability of computing resources onboard the mobile device 1002, etc.

[0194] The mobile device 1002 can use the gaussian scene representation to generate one or more views, which can be displayed to a user via the display screen. For example, the view(s) can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the mobile device 1002 and subsequently sent to another device for rendering to generate one or more views. InQUALCOMM Ref. No. 2501497WO- 68 / 94 -some cases, the mobile device 1002 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a remote computing device, such as a server). In some cases, the mobile device 1002 can be configured to perform partial updates or full updates of the gaussian scene representation based on input pixel data received from another device, such as augmented reality or mixed reality glasses, a headset device, a body cam, or another wearable electronic device.

[0195] FIG. 11 depicts an implementation 1100 in which the device 102 includes a portable electronic device that corresponds to augmented reality’ or mixed reality glasses 1102. The glasses 1102 include a holographic projection unit 1112 configured to project visual data onto a surface of a lens 1114 or to reflect the visual data off of a surface of the lens 1114 and onto the wearer’s retina. The glasses 1102 also include one or more cameras 1108, and optionally one or more microphones 1110, one or more speakers 1106. or both. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the renderer 158 and the view selector 162, are integrated in the glasses 1102.

[0196] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1108 of the glasses 1102 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1108 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the glasses 1102.

[0197] The glasses 1 102 can use the gaussian scene representation to generate one or more views, which can be projected by the holographic projection unit 1112 onto theQUALCOMM Ref. No. 2501497WO- 69 / 94 -lens 1114. For example, the view can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the glasses 1102 and subsequently sent to another device for rendering to generate one or more views. In some cases, the glasses 1102 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the glasses 1102 or a remote computing device, such as a server).

[0198] FIG. 12 depicts an implementation 1200 in which the device 102 includes a wearable electronic device 1202, illustrated as a wearable camera device (sometimes referred to as a “body cam”). A front 1222 and a back 1220 of the wearable electronic device 1202 are shown in FIG. 12.

[0199] In the example illustrated, the wearable electronic device 1202 includes one or more microphones 1210, one or more speakers 1206, and one or more cameras 1208. The wearable electronic device 1202 may also include a body 1224 with an integrated attachment mechanism, such as clip 1226. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the renderer 158 and the view selector 162, are integrated in the wearable electronic device 1202.

[0200] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1208 of the wearable electronic device 1202 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1208 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representationQUALCOMM Ref. No. 2501497WO- 70 / 94 -is further based on additional data, such as availability of computing resources and / or power onboard the wearable electronic device 1202.

[0201] The wearable electronic device 1202 can use the gaussian scene representation to facilitate user interaction with the scene. For example, the wearable electronic device 1202 can generate verbal output (e.g., instructions or information about the scene) based on the gaussian scene representation and can provide the verbal output to the user via the speaker(s) 1206. In some cases, the gaussian scene representation can be updated at the wearable electronic device 1202 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wearable electronic device 1202 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wearable electronic device 1202 or a remote computing device, such as a server).

[0202] FIG. 13 is an implementation 1300 in which the device 102 includes a wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 can have wireless network connectivity and is configured to execute an assistant operation. In the example illustrated, the wireless speaker and voice activated device 1302 includes one or more microphones 1310, one or more speakers 1306, one or more cameras 1308, and a display 1304. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the renderer 158 and the view selector 162, are integrated in the wireless speaker and voice activated device 1302.

[0203] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1308 of the wireless speaker and voice activated device 1302 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1308 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing noQUALCOMM Ref. No. 2501497WO- 71 / 94 -update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the wireless speaker and voice activated device 1302.

[0204] In a particular aspect, the wireless speaker and voice activated device 1302 can provide assistant operations based at least in part on the gaussian scene representation. For example, the wireless speaker and voice activated device 1302 can respond to verbal queries from the user based on the gaussian scene representation, such as by providing directions to an object or person in the scene. In some cases, the gaussian scene representation can be updated at the wireless speaker and voice activated device 1302 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wireless speaker and voice activated device 1302 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wireless speaker and voice activated device 1302 or a remote computing device, such as a server).

[0205] FIG. 14 depicts an implementation 1400 in which the device 102 includes a portable electronic device that corresponds to a camera device 1402. In the example illustrated, the camera device 1402 includes one or more microphones 1410, one or more speakers 1406, and one or more cameras 1408. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the camera device 1402.

[0206] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1408 of the camera device 1402 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1408 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussianQUALCOMM Ref. No. 2501497WO- 72 / 94 -scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the camera device 1402.

[0207] The camera device 1402 can use the gaussian scene representation to facilitate user interaction with the scene, documentation of the scene (e.g., by object or location tagging), etc. In some cases, the gaussian scene representation can be updated at the camera device 1402 and subsequently sent to another device for rendering to generate one or more views. In some cases, the camera device 1402 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the camera device 1402 or a remote computing device, such as a server).

[0208] FIG. 15 depicts an implementation 1500 in which the device 102 includes a portable electronic device that corresponds to a virtual reality, mixed reality, or augmented reality headset 1502. In the example illustrated, the headset 1502 includes one or more microphones 1510, one or more speakers 1506, and one or more cameras 1508. The headset 1502 also includes a display 1504 positioned to be visible to a user when the headset 1502 is w orn by the user. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the headset 1502.

[0209] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1508 of the headset 1502 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1508 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model updateQUALCOMM Ref. No. 2501497WO- 73 / 94 -pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the headset 1502.

[0210] In some cases, the headset 1502 can use the gaussian scene representation (e g., the updated gaussian scene representation) to generate a virtual reality or augmented reality view for a user. In some cases, the gaussian scene representation can be updated at the headset 1502 and subsequently sent to another device for rendering to generate one or more views. In some cases, the headset 1502 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the headset 1502 or a remote computing device, such as a server).

[0211] FIG. 16 depicts an implementation 1600 in which the device 102 corresponds to, or is integrated within, a vehicle 1602, illustrated as a manned or unmanned aerial device (e g., a package delivery' drone). In the example illustrated, the vehicle 1602 includes one or more microphones 1610, one or more speakers 1606, and one or more cameras 1608. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1602.

[0212] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1608 of the vehicle 1602 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1608 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy' (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representationQUALCOMM Ref. No. 2501497WO- 74 / 94 -is further based on additional data, such as availability of computing resources and / or power onboard the vehicle 1602.

[0213] In some cases, the vehicle 1602 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1602 and subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1602 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1602 or a remote computing device, such as a server).

[0214] FIG. 17 depicts another implementation 1700 in which the device 102 corresponds to. or is integrated within, a vehicle 1702, illustrated as a car. In the example illustrated, the vehicle 1702 includes one or more displays 1704, one or more microphones 1708, one or more speakers 1706, and one or more cameras 1710. For example, the cameras 1710 can include one or more interior cameras 1710A (which can be positioned to capture interior or exterior images), one or more front cameras 1710B, one or more rear cameras 1710C, one or more side cameras 1710D and 1710E. or a combination thereof. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1702.

[0215] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1710 of the vehicle 1702 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1710 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, aQUALCOMM Ref. No. 2501497WO- 75 / 94 -determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the vehicle 1702.

[0216] In some cases, the vehicle 1702 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1702 and subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1702 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1702 or a remote computing device, such as a server).

[0217] Referring to FIG. 18. a block diagram of a particular illustrative implementation of a device is depicted and generally designated 1800. In various implementations, the device 1800 may have more or fewer components than illustrated in FIG. 18. In an illustrative implementation, the device 1800 may correspond to the device 102. In an illustrative implementation, the device 1800 may perform one or more operations described with reference to FIGS. 1-17.

[0218] In a particular implementation, the device 1800 includes a processor 1806 (e.g., a CPU). The device 1800 may include one or more additional processors 1810 (e.g., one or more DSPs). In a particular aspect, the processor(s) 130 of FIG. 1 corresponds to the processor 1806, the processors 1810, or a combination thereof. The processors 1810 may include a speech and music coder-decoder (CODEC) 1808 that includes the selective model update pipeline 140, the gaussian scene representation 154, a voice coder ("‘vocoder’) encoder 1836, a vocoder decoder 1838, or a combination thereof.

[0219] The device 1800 may include a memory 1886 and a CODEC 1834. The memory 1886 may include instructions 1856, that are executable by the one or more additional processors 1810 (or the processor 1806) to implement the functionality described with reference to the selective model update pipeline 140, the gaussian scene representation 154, or both. The device 1800 may include a modem 1870 coupled, via a transceiver 1850, to an antenna 1852. In a particular aspect, the modem 1870 corresponds to the modem 114 of FIG. 1.QUALCOMM Ref. No. 2501497WO- 76 / 94 -

[0220] The device 1800 may include a display 1828 coupled to a display controller 1826. One or more speakers 1892 and one or more microphones 1894 may be coupled to the CODEC 1834. The CODEC 1834 may include a digital-to-analog converter (DAC) 1802, an analog-to-digital converter (ADC) 1804, or both. In a particular implementation, the CODEC 1834 may receive analog signals from the microphone(s) 1894. convert the analog signals to digital signals using the analog-to-digital converter 1804, and provide the digital signals to the speech and music codec 1808. In a particular implementation, the speech and music codec 1808 may provide digital signals to the CODEC 1834. The CODEC 1834 may convert the digital signals to analog signals using the digital-to-analog converter 1802 and may provide the analog signals to the one or more speakers 1892.

[0221] In a particular implementation, the device 1800 may be included in a system-in-package or system-on-chip device 1822. In a particular implementation, the memory 1886. the processor 1806, the processors 1810, the display controller 1826. the CODEC 1834, and the modem 1870 are included in the system-in-package or system-on-chip device 1822. In a particular implementation, an input device 1830 (e.g., one or more cameras) and a power supply 1844 are coupled to the system-in-package or the system-on-chip device 1822. Moreover, in a particular implementation, as illustrated in FIG. 18, the display 1828, the input device 1830, the one or more speakers 1892, the microphone(s) 1894, the antenna 1852, and the power supply 1844 are external to the system-in-package or the system-on-chip device 1822. In a particular implementation, each of the display 1828. the input device 1830, the one or more speakers 1892, the microphone(s) 1894, the antenna 1852, and the power supply 1844 may be coupled to a component of the system-in-package or the system-on-chip device 1822, such as an interface, or a controller.

[0222] The device 1800 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portableQUALCOMM Ref. No. 2501497WO- 77 / 94 -electronic device, a car, a computing device, a communication device, an intemet-of-things (loT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0223] In conjunction with the described implementations, an apparatus includes means for obtaining input pixel data. For example, the means for obtaining input pixel data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106, the modem 114, the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the sensor(s) 302, the sensor controller 304, the image subspace selector 312. the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108, the camera(s) 1208, the camera(s) 1308, the camera(s) 1408, the camera(s) 1508, the camera(s) 1608, the camera(s) 1710, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data, or any combination thereof.

[0224] The apparatus also includes means for selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation based on differences between the input pixel data and reference pixel data, where the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation. For example, the means for selecting an update strategy can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the update strategy selector 144, the integrated circuit 902. the processor 1806, the processor(s) 1810, one or more other circuits or components configured to select an update strategy for a gaussian scene representation, or any combination thereof.

[0225] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 1886) includes instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g., the one or more processors 1810 or the processor 1806), cause the one or more processors to obtain input pixel data and, based on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, an update strategy for a gaussian scene representation, where the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.QUALCOMM Ref. No. 2501497WO- 78 / 94 -

[0226] Particular aspects of the disclosure are described below in a first set of interrelated Examples:

[0227] According to Example 1, a device includes a memory configured to store a gaussian scene representation; and one or more processors coupled to the memory and configured to obtain input pixel data; and based at least in part on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, an update strategy for the gaussian scene representation, wherein the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0228] Example 2 includes the device of Example 1, wherein the plurality of update strategies include a full update strategy and one or more partial update strategies.

[0229] Example 3 includes the device of Example 2, wherein, to perform a partial update of the gaussian scene representation, the one or more processors are configured to modify a first subset of gaussians of the gaussian scene representation holding a second subset of the gaussians of the gaussian scene representation constant.

[0230] Example 4 includes the device of any of Examples 1 to 3, wherein the one or more processors are configured to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and compare the value of the difference metric to one or more first thresholds to select the update strategy.

[0231] Example 5 includes the device of any of Examples 1 to 4, wherein the one or more processors are configured to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and select a partial update as the update strategy' if the value is greater than a first threshold and less than a second threshold.

[0232] Example 6 includes the device of any of Examples 1 to 5, wherein the one or more processors are configured to identify a difference region based on the differences between the input pixel data and the reference pixel data; determine a size metric basedQUALCOMM Ref. No. 2501497WO- 79 / 94 -on dimensions of the difference region; and compare the size metric to one or more second thresholds to select the update strategy.

[0233] Example 7 includes the device of any of Examples 1 to 6, wherein the one or more processors are configured to identify difference gaussians of the gaussian scene representation, wherein the difference gaussians correspond to gaussians of the gaussian scene representation that are associated with the differences between the input pixel data and the reference pixel data; determine a size metric based on a count of the difference gaussians and a count of gaussians of the gaussian scene representation; and compare the size metric to one or more second thresholds to select the update strategy.

[0234] Example 8 includes the device of any of Examples 1 to 7 and further includes one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

[0235] Example 9 includes the device of any of Examples 1 to 8, wherein the memory includes an image buffer configured to store the input pixel data, the reference pixel data, or both.

[0236] Example 10 includes the device of any of Examples 1 to 9 and further includes one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data, and wherein the one or more processors are configured to select the update strategy further based on the sensor data.

[0237] Example 11 includes the device of any of Examples 1 to 10 and further includes a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the input pixel data, the reference pixel data, or both, via the modem.

[0238] According to Example 12, a method includes obtaining input pixel data; and based on differences between the input pixel data and reference pixel data, selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation, wherein the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.QUALCOMM Ref. No. 2501497WO- 80 / 94 -

[0239] Example 13 includes the method of Example 12 and further includes, when a partial update strategy is selected, performing a partial update of the gaussian scene representation including modifying a first subset of gaussians of the gaussian scene representation holding a second subset of the gaussians of the gaussian scene representation constant.

[0240] Example 14 includes the method of Example 12 or Example 13 and further includes determining a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and comparing the value of the difference metric to one or more first thresholds to select the update strategy.

[0241] Example 15 includes the method of any of Examples 12 to 14 and further includes determining a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and selecting, as the update strategy, a partial update if the value is greater than a first threshold and less than a second threshold.

[0242] Example 16 includes the method of any of Examples 12 to 15 and further includes identifying a difference region based on the differences between the input pixel data and the reference pixel data; determining a size metric based on dimensions of the difference region; and comparing the size metric to one or more second thresholds to select the update strategy.

[0243] Example 17 includes the method of any of Examples 12 to 16 and further includes identifying difference gaussians of the gaussian scene representation, wherein the difference gaussians correspond to gaussians of the gaussian scene representation that are associated with the differences between the input pixel data and the reference pixel data; determining a size metric based on a count of the difference gaussians and a count of gaussians of the gaussian scene representation; and comparing the size metric to one or more second thresholds to select the update strategy.

[0244] According to Example 18, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain input pixel data; and based on differences between the input pixel data and reference pixel data, select, from among a plurality of update strategies, anQUALCOMM Ref. No. 2501497WO- 81 / 94 -update strategy for a gaussian scene representation, wherein the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0245] Example 19 includes the non-transitory computer-readable medium of Example 18, wherein, when the update strategy selected is a partial update, the instructions cause the one or more processors to modify a first subset of gaussians of the gaussian scene representation holding a second subset of the gaussians of the gaussian scene representation constant.

[0246] Example 20 includes the non-transitory computer-readable medium of Example 18 or Example 19, wherein the instructions cause the one or more processors to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and compare the value of the difference metric to one or more first thresholds to select the update strategy.

[0247] Example 21 includes the non-transitory computer-readable medium of any of Examples 18 to 20, wherein the instructions cause the one or more processors to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data; and select, as the update strategy, a partial update if the value is greater than a first threshold and less than a second threshold.

[0248] Example 22 includes the non-transitory computer-readable medium of any of Examples 18 to 21, wherein the instructions cause the one or more processors to identify a difference region based on the differences between the input pixel data and the reference pixel data; determine a size metric based on dimensions of the difference region; and compare the size metric to one or more second thresholds to select the update strategy.

[0249] Example 23 includes the non-transitory' computer-readable medium of any of Examples 18 to 22, wherein the instructions cause the one or more processors to identify’ difference gaussians of the gaussian scene representation, wherein the difference gaussians correspond to gaussians of the gaussian scene representation that are associated with the differences between the input pixel data and the reference pixel data; determine a size metric based on a count of the difference gaussians and a count ofQUALCOMM Ref. No. 2501497WO- 82 / 94 -gaussians of the gaussian scene representation; and compare the size metric to one or more second thresholds to select the update strategy.

[0250] According to Example 24, an apparatus includes means for obtaining input pixel data; and means for selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation based on differences between the input pixel data and reference pixel data, wherein the reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation.

[0251] Further aspects of the disclosure are described below in a second set of interrelated Examples:

[0252] According to Example 1, a device includes a memory configured to store a gaussian scene representation representing a scene and one or more processors coupled to the memory and configured to obtain input pixel data; identify a region of interest (ROI) based on the input pixel data; perform a comparison of ROI data associated with the ROI and update target data associated with the gaussian scene representation; and select an update strategy for the gaussian scene representation based on a result of the comparison.

[0253] Example 2 includes the device of Example 1, wherein the update target data identifies one or more update exclusion regions associated with the scene.

[0254] Example 3 includes the device of Example 2, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update exclusion regions.

[0255] Example 4 includes the device of any of Examples 1 to 3, wherein the update target data identifies one or more update inclusion regions associated with the scene.

[0256] Example 5 includes the device of Example 4, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update inclusion regions.

[0257] Example 6 includes the device of any of Examples 1 to 5, wherein the update target data identifies one or more update target objects.QUALCOMM Ref. No. 2501497WO- 83 / 94 -

[0258] Example 7 includes the device of Example 6, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more update target objects.

[0259] Example 8 includes the device of any of Examples 1 to 7, wherein the update target data identifies one or more exclusion target objects.

[0260] Example 9 includes the device of Example 8, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more exclusion target objects.

[0261] Example 10 includes the device of any of Examples 1 to 9, wherein, the one or more processors are configured to, based on a first result of the comparison, select a no update strategy for the gaussian scene representation.

[0262] Example 11 includes the device of Example 10, wherein, the one or more processors are configured to, based on a second result of the comparison, select between a partial update strategy and a full update strategy for the gaussian scene representation.

[0263] Example 12 includes the device of Example 11, wherein, when the partial update strategy is selected, the one or more processors are configured to update one or more gaussians associated with the ROI and hold constant remaining gaussians of the gaussian scene representation.

[0264] Example 13 includes the device of any of Examples 1 to 12, wherein the one or more processors are configured to obtain the input pixel data based on user input.

[0265] Example 14 includes the device of any of Examples 1 to 13, wherein the one or more processors are configured to identify the ROI based on user input.

[0266] Example 15 includes the device of any of Examples 1 to 14, wherein, to identify the ROI, the one or more processors are configured to identify differences between the input pixel data and reference pixel data, wherein the reference pixel data represents at least a portion of a view associated with the gaussian scene representation.

[0267] Example 16 includes the device of Example 15, wherein the one or more processors are configured to determine a value of a difference metric based on theQUALCOMM Ref. No. 2501497WO- 84 / 94 -differences between the input pixel data and the reference pixel data, wherein the update strategy is selected further based on the value of the difference metric.

[0268] Example 17 includes the device of any of Examples 1 to 16, wherein the one or more processors are configured to determine a size metric based on dimensions of the ROI, wherein the update strategy is selected further based on the size metric.

[0269] Example 18 includes the device of any of Examples 1 to 17 and further includes one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

[0270] Example 19 includes the device of any of Examples 1 to 18, wherein the memory includes an image buffer configured to store the input pixel data.

[0271] Example 20 includes the device of any of Examples 1 to 19 and further includes one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data.

[0272] Example 21 includes the device of Example 20, wherein the one or more processors are configured to select the update strategy' for the gaussian scene representation further based on the sensor data.

[0273] Example 22 includes the device of any of Examples 1 to 21 and further includes a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the input pixel data via the modem.

[0274] Example 23 includes the device of any of Examples 1 to 22, wherein the one or more processors are integrated into at least one of a mobile phone, a tablet computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, or an augmented reality headset.

[0275] Example 24 includes the device of any of Examples 1 to 22, wherein the one or more processors are integrated into a vehicle, the vehicle further including a camera configured to capture one or more images to generate the input pixel data.

[0276] Example 25 includes the device of any of Examples 1 to 24, wherein the one or more processors are included in an integrated circuit.QUALCOMM Ref. No. 2501497WO- 85 / 94 -

[0277] According to Example 26, a method includes obtaining, at one or more processors, input pixel data; identifying, by the one or more processors, a region of interest (ROI) based on the input pixel data; performing, by the one or more processors, a comparison of ROI data associated with the ROI and update target data associated with a gaussian scene representation that represents a scene; and selecting, by the one or more processors, an update strategy for the gaussian scene representation based on a result of the comparison.

[0278] Example 27 includes the method of Example 26, wherein the update target data identifies one or more update exclusion regions associated with the scene.

[0279] Example 28 includes the method of Example 27, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update exclusion regions.

[0280] Example 29 includes the method of any of Examples 26 to 28, wherein the update target data identifies one or more update inclusion regions associated with the scene.

[0281] Example 30 includes the method of Example 29, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update inclusion regions.

[0282] Example 31 includes the method of any of Examples 26 to 30, wherein the update target data identifies one or more update target objects.

[0283] Example 32 includes the method of Example 31, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more update target objects.

[0284] Example 33 includes the method of any of Examples 26 to 32, wherein the update target data identifies one or more exclusion target objects.

[0285] Example 34 includes the method of Example 33, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more exclusion target objects.QUALCOMM Ref. No. 2501497WO- 86 / 94 -

[0286] Example 35 includes the method of any of Examples 26 to 34 and further includes, based on a first result of the comparison, selecting ano update strategy for the gaussian scene representation.

[0287] Example 36 includes the method of Example 35 and further includes, based on a second result of the comparison, selecting between a partial update strategy and a full update strategy for the gaussian scene representation.

[0288] Example 37 includes the method of Example 36 and further includes, when the partial update strategy is selected, updating one or more gaussians associated with the ROI and holding constant remaining gaussians of the gaussian scene representation.

[0289] Example 38 includes the method of any of Examples 26 to 37 and further includes obtaining the input pixel data based on user input.

[0290] Example 39 includes the method of any of Examples 26 to 38 and further includes identifying the ROI based on user input.

[0291] Example 40 includes the method of any of Examples 26 to 39 and further includes, to identify the ROI, identifying differences between the input pixel data and reference pixel data, wherein the reference pixel data represents at least a portion of a view associated with the gaussian scene representation.

[0292] Example 41 includes the method of Example 40 and further includes determining a value of a difference metric based on the differences between the input pixel data and the reference pixel data, wherein the update strategy is selected further based on the value of the difference metric.

[0293] Example 42 includes the method of any of Examples 26 to 41 and further includes determining a size metric based on dimensions of the ROI, wherein the update strategy is selected further based on the size metric.

[0294] Example 43 includes the method of any of Examples 26 to 42, wherein the update strategy is selected further based on sensor data associated with the input pixel data.QUALCOMM Ref. No. 2501497WO- 87 / 94 -

[0295] According to Example 44, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain input pixel data; identify a region of interest (ROI) based on the input pixel data; perform a comparison of ROI data associated with the ROI and update target data associated w ith a gaussian scene representation that represents a scene; and select an update strategy for the gaussian scene representation based on a result of the comparison.

[0296] Example 45 includes the non-transitory computer-readable medium of Example44, wherein the update target data identifies one or more update exclusion regions associated with the scene.

[0297] Example 46 includes the non-transitory' computer-readable medium of Example45, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update exclusion regions.

[0298] Example 47 includes the non-transitory' computer-readable medium of any of Examples 44 to 46, wherein the update target data identifies one or more update inclusion regions associated with the scene.

[0299] Example 48 includes the non-transitory' computer-readable medium of Example 47, wherein the result of the comparison indicates whether a location of the ROI is within any of the one or more update inclusion regions.

[0300] Example 49 includes the non-transitory' computer-readable medium of any of Examples 44 to 48, wherein the update target data identifies one or more update target objects.

[0301] Example 50 includes the non-transitory' computer-readable medium of Example 49, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more update target objects.

[0302] Example 51 includes the non-transitory' computer-readable medium of any of Examples 44 to 50, wherein the update target data identifies one or more exclusion target objects.QUALCOMM Ref. No. 2501497WO- 88 / 94 -

[0303] Example 52 includes the non-transitory computer-readable medium of Example 51, wherein the result of the comparison indicates whether an object identified in the ROI matches any of the one or more exclusion target objects.

[0304] Example 53 includes the non-transitory' computer-readable medium of any of Examples 44 to 52, wherein, the instructions are configured to cause the one or more processors to, based on a first result of the comparison, select a no update strategy for the gaussian scene representation.

[0305] Example 54 includes the non-transitory computer-readable medium of Example53, wherein, the instructions are configured to cause the one or more processors to, based on a second result of the comparison, select between a partial update strategy and a full update strategy for the gaussian scene representation.

[0306] Example 55 includes the non-transitory computer-readable medium of Example54, wherein the instructions, when executed by the one or more processors, cause the one or more processors to, when the partial update strategy is selected, update one or more gaussians associated with the ROI and hold constant remaining gaussians of the gaussian scene representation.

[0307] Example 56 includes the non-transitory computer-readable medium of any of Examples 44 to 55, wherein the instructions, when executed by the one or more processors, cause the one or more processors to obtain the input pixel data based on user input.

[0308] Example 57 includes the non-transitory computer-readable medium of any of Examples 44 to 56, wherein the instructions, when executed by the one or more processors, cause the one or more processors to identify the ROI based on user input.

[0309] Example 58 includes the non-transitory' computer-readable medium of any of Examples 44 to 57, wherein the instructions, when executed by the one or more processors, cause the one or more processors to, to identify the ROI, identify differences between the input pixel data and reference pixel data, wherein the reference pixel data represents at least a portion of a view' associated w ith the gaussian scene representation.QUALCOMM Ref. No. 2501497WO- 89 / 94 -

[0310] Example 59 includes the non-transitory computer-readable medium of Example 58, wherein the instructions, when executed by the one or more processors, cause the one or more processors to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data, wherein the update strategy is selected further based on the value of the difference metric.

[0311] Example 60 includes the non-transitory computer-readable medium of any of Examples 44 to 59, wherein the instructions, when executed by the one or more processors, cause the one or more processors to determine a size metric based on dimensions of the ROI, wherein the update strategy is selected further based on the size metric.

[0312] According to Example 61, an apparatus includes means for obtaining input pixel data; means for identifying a region of interest (ROI) based on the input pixel data; and means for performing a comparison of ROI data associated with the ROI and update target data associated with a gaussian scene representation that represents a scene; and means for selecting an update strategy' for the gaussian scene representation based on a result of the comparison.

[0313] Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality'. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

[0314] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memoryQUALCOMM Ref. No. 2501497WO- 90 / 94 - (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to. the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

[0315] The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims

QUALCOMM Ref. No. 2501497WO- 91 / 94 - WHAT IS CLAIMED IS:

1. A device comprising:a memory configured to store a gaussian scene representation representing a scene; and one or more processors coupled to the memory and configured to:obtain input pixel data;identify difference pixels based on the input pixel data and reference pixel data; perform a comparison of difference data associated with the difference pixels and update target data associated with the gaussian scene representation; andselect an update strategy for the gaussian scene representation based on a result of the comparison.

2. The device of claim 1, wherein the update target data identifies one or more update exclusion regions associated with the scene.

3. The device of claim 2, wherein the result of the comparison indicates whether a location of the difference pixels is within any of the one or more update exclusion regions.

4. The device of claim 1, wherein the update target data identifies one or more update inclusion regions associated with the scene.

5. The device of claim 4, wherein the result of the comparison indicates whether a location of the difference pixels is within any of the one or more update inclusion regions.

6. The device of claim 1, wherein the update target data identifies one or more update target objects.QUALCOMM Ref. No. 2501497WO- 92 / 94 - 7. The device of claim 6, wherein the result of the comparison indicates whether an object identified in the difference pixels matches any of the one or more update target objects.

8. The device of claim 1, wherein the update target data identifies one or more exclusion target objects.

9. The device of claim 8, wherein the result of the comparison indicates whether an object identified in the difference pixels matches any of the one or more exclusion target objects.

10. The device of claim 1, wherein, the one or more processors are configured to, based on a first result of the comparison, select a no update strategy for the gaussian scene representation.

11. The device of claim 10, wherein, the one or more processors are configured to, based on a second result of the comparison, select between a partial update strategy and a full update strategy for the gaussian scene representation.

12. The device of claim 11, wherein, when the partial update strategy is selected, the one or more processors are configured to update one or more gaussians associated with the difference pixels and hold constant remaining gaussians of the gaussian scene representation.

13. The device of claim 1, wherein the one or more processors are configured to obtain the input pixel data based on user input.

14. The device of claim 1, wherein the one or more processors are configured to identify the difference pixels based on user input.

15. The device of claim 1, wherein, to identify the difference pixels, the one or more processors are configured to identify differences between the input pixel data andQUALCOMM Ref. No. 2501497WO- 93 / 94 -reference pixel data, wherein the reference pixel data represents at least a portion of a view associated with the gaussian scene representation.

16. The device of claim 15, wherein the one or more processors are configured to determine a value of a difference metric based on the differences between the input pixel data and the reference pixel data, wherein the update strategy is selected further based on the value of the difference metric.

17. The device of claim 1, wherein the one or more processors are configured to determine a size metric based on dimensions of the difference pixels, wherein the update strategy is selected further based on the size metric.

18. The device of claim 1, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

19. The device of claim 1, further comprising one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data.

20. The device of claim 19, wherein the one or more processors are configured to select the update strategy' for the gaussian scene representation further based on the sensor data.