Partially updating a gaussian scene representation using spatial transformations

Partial updates of gaussian scene representations address the computational challenges by selectively updating subsets based on input data and conditions, reducing resource consumption and maintaining scene accuracy.

WO2026240945A1PCT designated stage Publication Date: 2026-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/024841
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-27
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Gaussian scene representations require significant computational resources for training and updating due to their large data size and frequent updates, especially in dynamic environments like autonomous vehicle navigation, leading to resource constraints.

Method used

Implement partial updates of gaussian scene representations by selectively updating a subset of gaussians based on input pixel data, sensor data, or device conditions, using strategies like saliency mapping and staged data storage to reduce computational complexity.

Benefits of technology

Significantly reduces computing resources needed for maintaining an up-to-date gaussian scene representation by minimizing full updates, preserving resources for other processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026024841_19112026_PF_FP_ABST
    Figure US2026024841_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A device includes a memory configured to store a gaussian scene representation. The device also includes one or more processors coupled to the memory and configured to obtain input pixel data and select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. The one or more processors are configured to spatially transform the selected gaussians of the gaussian scene representation to partially update the gaussian scene representation.
Need to check novelty before this filing date? Find Prior Art

Description

QUALCOMM Ref. No. 2501501WO- 1 / 130 -PARTIALLY UPDATING A GAUSSIAN SCENE REPRESENTATION USING SPATIAL TRANSFORMATIONSI. Cross-Reference to Related Applications

[0001] The present application claims priority from the commonly owned U. S.Provisional Patent Application No. 63 / 804,910, filed May 13, 2025, U. S. Provisional Patent Application No. 63 / 831,815, filed June 27, 2025, U. S. Provisional Patent Application No. 63 / 822,752, filed June 12, 2025, U. S. Provisional Patent Application No. 63 / 822,760, filed June 12, 2025. U. S. Provisional Patent Application No.63 / 826.028, filed June 18, 2025. U. S. Provisional Patent Application No. 63 / 825,138, filed June 17, 2025, and U. S. Provisional Patent Application No. 63 / 820,238, filed June 9, 2025, the contents of each of which are expressly incorporated herein by reference in their entirety.II. Field

[0002] The present disclosure is generally related to gaussian scene representation.III. Description of Related Art

[0003] Computer science uses various techniques to generate three dimensional (3D) models or four dimensional (4D) models (where time is a fourth dimension) of a scene based on images of the scene. One example of a scene model used in computer science is a gaussian scene representation, which uses primitive elements commonly referred to as “gaussians”.

[0004] A process referred to as gaussian splatting can be used to generate views based on the gaussian scene representation. The views can include views of the scene from various perspectives, and in particular, the gaussian scene representation can be used to generate novel views of the scene (e g., views of the scene that do not correspond directly to any particular image used to generate the gaussian scene representation). For example, a gaussian scene representation can be used to generate images or sequences of images (e.g., videos) that simulate how the scene would appear from a different vantage point, perspective, or lighting condition, gaussian scene representations can be used to generate imagery for computer vision, graphics, video games, extended realityQUALCOMM Ref. No. 2501501WO- 2 / 130 -visualization. 3D reconstruction, security and surveillance, medical imaging, computer assisted navigation, as well as other uses.

[0005] One challenge associated with using gaussian scene representations is that a large set of training data (e.g., images of the scene from a variety of viewpoints) are generally processed in order to generate an accurate gaussian scene representation, gaussian scene representations also tend to be large (e.g., in terms of data storage used). As a result, use of a gaussian scene representation to generate a novel view consumes significant computing resources (e.g., memory utilization, processor utilization, power, etc.), and training the gaussian scene representation uses even more computer resources than view generation. Consumption of computing resources is particularly challenging in situations where a gaussian scene representation is updated frequently due to changes in the scene over time. To illustrate, in an autonomous vehicle control use case, a gaussian scene representation can be used for various computer vision tasks, such as obstacle avoidance, compliance with traffic laws, etc. In order to perform these tasks well, the gaussian scene representation may be updated frequently as the vehicle moves through the scene and / or as other vehicles, pedestrians, etc., move nearby.IV. Summary

[0006] According to some embodiments of the present disclosure, a device includes a memory configured to store a gaussian scene representation. The device includes one or more processors coupled to the memory and configured to obtain input pixel data. The one or more processors are configured to select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. The one or more processors are configured to spatially transform the selected gaussians of the gaussian scene representation.

[0007] According to some embodiments, a method includes obtaining, by one or more processors, input pixel data. The method includes selecting, by the one or more processors, based on the input pixel data, gaussians of a gaussian scene representation to be updated. The method includes spatially transforming, by the one or more processors, the selected gaussians of the gaussian scene representation.QUALCOMM Ref. No. 2501501WO- 3 / 130 -

[0008] According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations including obtaining input pixel data. The operations include selecting, based on the input pixel data, gaussians of agaussian scene representation to be updated, the gaussian scene representation representing a scene. The operations include spatially transforming the selected gaussians of the gaussian scene representation.

[0009] According to some embodiments, a device includes a memory configured to store a gaussian scene representation. The device includes one or more processors configured to obtain input pixel data. The one or more processors are configured to select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. The one or more processors are configured to determine a set of parameter adjustment candidates for parameters of the selected gaussians. The one or more processors are configured to update the selected gaussians based on the set of parameter adjustment candidates. In some embodiments, the set of parameter adjustment candidates may be determined using dithering.

[0010] According to some embodiments, a method includes obtaining, by one or more processors, input pixel data. The method includes selecting, by the one or more processors, based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation associated with a scene. The method includes determining, by the one or more processors, a set of parameter adjustment candidates for parameters of the selected gaussians. The method includes updating, by the one or more processors, the selected gaussians based on the set of parameter adjustment candidates. In some embodiments, the set of parameter adjustment candidates may be determined using dithering.

[0011] According to some embodiments anon-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations including obtaining input pixel data. The operations include selecting based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation associated with a scene. The operations include determining a set of parameter adjustment candidates forQUALCOMM Ref. No. 2501501WO- 4 / 130 -parameters of the selected gaussians. The method includes updating the selected gaussians based on the set of parameter adjustment candidates. In some embodiments, the set of parameter adjustment candidates may be determined using dithering.

[0012] According to some embodiments, a device includes a memory configured to store a gaussian scene representation representing a scene. The device includes one or more processors configured to obtain input pixel data corresponding to an input image depicting a portion of the scene. The one or more processors are configured to obtain motion data based on two or more prior images of the scene. The one or more processors are configured to select, based, at least in part, on the input image, gaussians of the gaussian scene representation to be updated. The one or more processors are configured to adjust parameters of the selected gaussians to spatially transform the selected gaussians of the gaussian scene representation based on the motion data.

[0013] According to some embodiments, a method includes obtaining, by one or more processors, input pixel data corresponding to an input image depicting a portion of a scene. The method includes obtaining, by the one or more processors, motion data based on two or more prior images of the scene. The method includes selecting, by the one or more processors, based, at least in part, on the input image, gaussians of a gaussian scene representation to be updated. The method includes adjusting, by the one or more processors, parameters of the selected gaussians to spatially transform the selected gaussians based on the motion data.

[0014] According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations including obtaining input pixel data corresponding to an image depicting a portion of a scene. The operations include obtaining motion data based on two or more prior images of the scene. The operations include selecting based, at least in part, on the input image, gaussians of a gaussian scene representation to be updated. The operations include adjusting parameters of the selected gaussians to spatially transform the selected gaussians based on the motion data.QUALCOMM Ref. No. 2501501WO- 5 / 130 -

[0015] Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.V. Brief Description of the Drawings

[0016] FIG. 1 is a block diagram of an illustrative system operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure.

[0017] FIG. 2 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0018] FIGS. 3A, 3B, and 3C, together, illustrate particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0019] FIGS. 3D, 3E, and 3F illustrate particular aspects of performing partial updates of a gaussian scene representation in accordance with some examples of the present disclosure.

[0020] FIG. 4 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0021] FIG. 5 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0022] FIG. 6 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0023] FIG. 7 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501501WO- 6 / 130 -

[0024] FIG. 8 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0025] FIG. 9 illustrates an example of an integrated circuit operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0026] FIG. 10 is a diagram of a mobile device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0027] FIG. 11 is a diagram of a mixed reality or augmented reality glasses device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0028] FIG. 12 is a diagram of a wearable electronic device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0029] FIG. 13 is a diagram of a voice-controlled speaker system operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0030] FIG. 14 is a diagram of a camera operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0031] FIG. 15 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0032] FIG. 16 is a diagram of a first example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501501WO- 7 / 130 -

[0033] FIG. 17 is a diagram of a second example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0034] FIG. 18 is a block diagram of a particular illustrative example of a device that is operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.VI. Detailed Description

[0035] Use of a gaussian scene representation in a situation in which the gaussian scene representation is periodically or occasionally updated to capture new scene information (e.g., new image(s)) presents significant challenges due to the computational complexity of updating gaussian scene representations. Aspects disclosed herein reduce the computational complexity associated with updating a gaussian scene representation by using different update strategies to update the gaussian scene representation in different situations. For example, one update strategy can include performing a full update of the gaussian scene representation, and another update strategy is to perform a partial update of the gaussian scene representation. As used herein, a "partial update” refers to updating a specified subset of the gaussians of the gaussian scene representation while holding other gaussians of the gaussian scene representation constant. The specified subset of the gaussians updated during a partial update are selected based, at least in part, on changes detected in the scene (e.g., motion, lighting changes, differences between an input image and a reference image, etc.).

[0036] Selectively updating only a portion of the gaussian scene representation when the partial update strategy is used uses significantly fewer computing resources than performing the full update. In contrast to conventional update systems, which always perform a full update when any update is performed, the disclosed aspects provide computationally efficient mechanisms for identifying the portions of the gaussian scene representation that should be updated based on available information. As a result, computing resources used to maintain an up-to-date gaussian scene representation can be significantly reduced. Thus, a technical advantage of the disclosed aspects is a reduction of computing resources used to update gaussian scene representations.QUALCOMM Ref. No. 2501501WO- 8 / 130 -

[0037] In a particular aspect, a system includes one or more processors configured to obtain input pixel data associated with a scene and to select, based on an update criterion, an update strategy for the gaussian scene representation based on the input pixel data. The update criterion can be based on the input pixel data itself. For example, the input pixel data can be analyzed to identify motion, lighting changes, areas with pixel changes, or other characteristics, where changes are relative to training data (e.g.. a training image) used to train the gaussian scene representation or synthetic data (e.g., a synthetic image generated using the gaussian scene representation). In this example, the update criterion can include one or more thresholds, and results of the analysis of the input pixel data can be compared to the threshold(s) to select an update strategy, from among a plurality of available update strategies, to be used to update the gaussian scene representation based on the input pixel data.

[0038] As another example, the update condition can be based on sensor data. For example, the sensor data can be analyzed to identify motion, lighting changes, or other characteristics associated with updating the gaussian scene representation. In this example, the sensor data can include lighting data (e.g., to indicate lighting changes) or motion data (e.g., from an accelerometer or based on image analysis), as illustrative examples. In this example, the update criterion can include one or more thresholds, and results of the analysis of the sensor data can be compared to the threshold(s) to determine whether to update the gaussian scene representation based on the input pixel data. To illustrate, the sensor data can include a value of a lighting measurement from a lighting sensor, which can be compared to a reference lighting value to determine a lighting change value. The lighting change value can be compared to one or more thresholds to select an update strategy for the gaussian scene representation. A technical advantage of using sensor data to select the update strategy is that processing the sensor data uses fewer computing resources than processing pixel data and / or uses fewer computing resources than performing a full update of the gaussian scene representation; accordingly, using the sensor data to detect changes in the scene in order to select an update strategy can save significant computing resources.

[0039] As another example, the update condition can be based on a device condition. In this example, the device condition can be indicative of usage of a computational resource. To illustrate, the device condition can include or indicate a battery level, aQUALCOMM Ref. No. 2501501WO- 9 / 130 -processor load, a bus load, a cache load, a memory- usage, an available network bandwidth, and / or a memory access latency, as non-limiting examples. In this example, the update strategy for the gaussian scene representation can be selected based on levels of available computing resources. A technical advantage of using the device condition to select the update strategy- for the gaussian scene representation is that the gaussian scene representation can be maintained using appropriate updates (e.g.. full or partial) when sufficient computing resources are available, and more computationally expensive updates (e.g., full updates or partial updates using more gaussians) can be deferred or limited at other times, thereby limiting the impact of updating the gaussian scene representation on other processes.

[0040] The available update strategies that can be selected among can include at least a full update, a partial update, and no update. Further, in some cases, the available update strategies can include various types of partial updates, such as object-based updates, pixel-based updates, updates that are limited to specified parameters of a selected subset of gaussians, staged search updates, etc. In a full update, all of the gaussians of the gaussian scene representation are processed (generally, based on all of the available views). For example, during a full update, a large number of camera poses are used, and a rendered view is generated for each. Each rendered view is compared to a groundtruth view (e.g., an image of the scene captured by a camera) to calculate a loss function. A value or values of the loss function are used to determine gradients based on the differences between an output of the gaussian scene representation (e.g., a rendered view) and an expected output of the gaussian scene representation (e.g., a ground-truth view), and parameters of one or more of the gaussians of the gaussian scene representation are updated based on the gradients. In this example, during a full update, gradients are determined for every- parameter of every- gaussian of the gaussian scene representation, which is very computationally expensive.

[0041] In contrast to a full update, during a partial update only a selected subset of gaussians of the gaussian scene representation are updated and remaining gaussians are not modified. For example, a first set of gaussians of the gaussian scene representation are selected for update and the remaining gaussians of the gaussian scene representation are held constant during the partial update. In some cases, only selected parameters of the first set of gaussians are updated. For example, each gaussian is specified by a set ofQUALCOMM Ref. No. 2501501WO- 10 / 130 -parameters including location parameters indicating a location of a center of the gaussian in a scene space, an opacity parameter indicating an opacity (e.g., a maximum opacity) of the gaussian, one or more orientation parameters indicating an orientation of the gaussian in the scene space, a set of color parameters indicating spherical harmonics coefficients for defining a color of the gaussian as viewed from various directions, and so on. In this example, for some partial updates, a first subset of the parameters of each selected gaussian are updated and a second subset of the parameters of each selected gaussian are held constant. To illustrate, the location parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant. As another illustrative example, one or more color parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant.

[0042] The processor(s) can be configured to select an update strategy to be used, from among a plurality of available update strategies, based on the input pixel data, based on sensor data, based on a device condition, based on other factors (e.g., user input), or based on a combination thereof. For example, the processor(s) can be configured to perform no update when a metric associated with the input pixel data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. In this example, the metric can be based on an amount of difference between the input pixel data and reference pixel data, a size or dimensions of a region that differs between the input pixel data and reference pixel data, an amount of motion detected in the input pixel data, an amount of lighting change detected in the input pixel data, or other factors that are indicative of whether the gaussian scene representation accurately captures information in the input pixel data. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated with the sensor data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated with the device condition fails to satisfy a first threshold, to perform a partial update when the metric satisfies theQUALCOMM Ref. No. 2501501WO- 11 / 130 -first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. Metrics based on separate threshold comparisons for each factor are provided as an illustrative example; in other examples a metric can be based on a comparison of a combination of factors and corresponding thresholds. In addition to selecting between no update, a partial update, and a full update, in some cases, the processor(s) can select among several available partial update strategies.

[0043] The processor(s) can be configured to select which portions of the gaussian scene representation (e.g., which gaussians) are eligible to be modified during a partial update based on the input pixel data, based on sensor data, or both. For example, the processor(s) can identify a region of interest associated with differences between the input pixel data and the reference pixel data. In this example, the processor(s) can use mapping data to identify gaussians of the gaussian scene representation that are associated with the region of interest. In this example, the identified gaussians are selected to be updated (e g., are eligible to be modified) as part of the partial update. In some cases, the selected gaussians can affect pixels associated with the region of interest as well as other pixels. In such cases, mapping data can be used to determine which pixels of the input pixel data are to be used as training data to update the selected gaussians.

[0044] In a particular aspect, the mapping data can be generated based on gradient data. For example, saliency mapping can be used to estimate which gaussians most affect particular output pixel data from the gaussian scene representation to generate pixel-to-gaussian mapping data. Likewise, saliency mapping can be used to estimate which pixels of input data are most important to training of particular gaussians to generate gaussian-to-pixel mapping data.

[0045] Saliency mapping generally involves at least one forward pass through the gaussian scene representation (e.g., rendering one or more views) to generate gradients used to determine the pixel-to-gaussian mapping data, and at least one backpropagation pass to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially when a gaussian scene representation is updated frequently. To reduce the computational demand associated with running theQUALCOMM Ref. No. 2501501WO- 12 / 130 -gaussian scene representation to generate mapping data, particular aspects disclosed herein use a staged approach to manage the mapping data. In the staged approach, after mapping data (e g., pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, as described above, the mapping data can be stored for subsequent reference. For example, after mapping data associated with a particular region of interest is generated, the mapping data can be stored in a searchable data store. In this example, when new input image data is to be used for a partial update of the gaussian scene representation, a new region of interest in the new input image data is identified, and the searchable data store is searched to determine whether mapping data corresponding to pixels of the new region of interest is available (e.g., has already been generated and stored). If the mapping data is available, the mapping data can be retrieved from the searchable data store and used without the need to perform saliency mapping operations to generate the mapping data.

[0046] The region of interest used in the staged approach can be associated with a particular set of pixels, with a particular spatial area within the scene, with a particular object (or a portion of an object) identified in the scene, or with another identifiable feature of the scene. As one example, objects within the scene can be identified and saliency mapping can be used to associate one or more of the objects with gaussians of the gaussian scene representation. In this example, when new input image data is obtained, object detection can be performed to identify objects in the new input image data. If a particular object identified in the new input image data is associated with a change that triggers a partial update of the gaussian scene representation, the searchable data store is searched to determine whether mapping data is available for the particular object. If the mapping data for the particular object is available in the searchable data store, the mapping data is used for the partial update of the gaussian scene representation. If the mapping data for the particular object is not available in the searchable data store, the mapping data can be generated using saliency mapping.

[0047] Alternatively, in some cases, if particular mapping data is not available in a first searchable data store, a second searchable data store can be searched for the mapping data. For example, a set of most frequently used mapping data can be stored at a first searchable data store, a set of next most frequently used mapping data can be stored at a second searchable data store, and so forth, to an Nth data store (where N is an integerQUALCOMM Ref. No. 2501501WO- 13 / 130 -greater than 2). In such cases, full mapping data generation (e.g., using saliency mapping) is performed if none of the searchable data stores include the desired mapping data. Further, in some cases, after full mapping data generation is performed, the resulting mapping data can be added to one or more of the searchable data stores. Each of the searchable data stores can use an eviction policy in which a set of most recently- used or most frequently used mapping data is retained at the first searchable data store, and each subsequent searchable data store retains successively less recently used or less frequently accessed mapping data.

[0048] In some cases, the staged approach can use a single searchable data store, rather than several stages of searchable data stores. In such cases, the single searchable data store can evict the oldest or the least recently accessed mapping data when new mapping data is added. In such cases, full mapping data generation is performed if the single searchable data store does not include the desired mapping data. A technical advantage of the staged approach is that significant computing resources can be preserved by avoiding recreating mapping data when new input image data is to be used for a partial update of the gaussian scene representation.

[0049] In the present discussion, a “gaussian” refers to a graphical element of a gaussian scene representation. Each gaussian is specified by a set of parameters indicating characteristics, described further below, of the gaussian in scene space. In some examples, gaussians have a generally ellipsoidal 3D shape in the scene space, with a maximum opacity at a center position of the gaussian. In some examples, the opacity diminishes following an equation similar to a gaussian probability distribution.

[0050] In a particular aspect, the parameters specify ing a set of gaussians include:• a position vector, p,, which indicates the center position of a gaussian with index i,'• an opacity parameter, tLe [0,1], which indicates a maximum opacity of the gaussian;• orientation parameters, qf, which can include a quaternion defining the spatial orientation (rotation) of the gaussian;a scale vector, sf, which defines a size and shape of the ellipsoid of the gaussian;QUALCOMM Ref. No. 2501501WO- 14 / 130 - and• a set of spherical harmonics (SH) coefficients, H,. which can include a matrix with columns composed of 3D color vectors (in portions of this disclosure H, is referred to as an SH coefficient array).

[0051] In some implementations, not all of the parameters described above are used to specify a gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacity parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a view-independent color) rather than higher-order spherical harmonics that capture view-dependent color variations. Using a reduced parameter set can decrease memory usage and computational complexity associated with storing, updating, and rendering the gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.

[0052] In some implementations, the gaussian scene representation may use two-dimensional (2D) gaussians rather than three-dimensional (3D) gaussians. A 2D gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D gaussians, which distribute density throughout a model the complete volumetric ellipsoid, 2D gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.

[0053] A 2D gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parametersQUALCOMM Ref. No. 2501501WO- 15 / 130 - (e.g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D gaussian primitive may have fewer parameters than a corresponding 3D gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.

[0054] Using 2D gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D gaussian primitives can be more tightly aligned to object surfaces than 3D gaussians. Additionally, 2D gaussians may provide more accurate geometry representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.

[0055] In some implementations, the gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each gaussian of the gaussian scene representation can be used to represent a triangular face of a mesh, enabling conversion between gaussian representations and mesh representations. This approach provides a one-to-one mapping between gaussians and triangular mesh faces that is differentiable, enabling the gaussian scene representation to be modified in a manner similar to modifying a mesh.

[0056] To convert a 2D gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D gaussian may be performed. The covariance matrix S can be decomposed as S = RART, where R is a matrix of eigenvectors (el, e2, e3) and A is a diagonal matrix of eigenvalues (XI, X2, 3). The two largest eigenvalues (XI, X2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (z.3 ) corresponds to the normal direction. A triangular face can then be constructed using the mean of the gaussian as a center point and defining triangle vertices along the tangent directions. For example, triangle vertices may be defined as vl = p + alel, v2 = p + a2e2, and v3 = p - alel, where al and a2 are scale factors proportional to square roots of their respective eigenvalues XI and X2 and p is the center point corresponding to the mean of the 2D gaussian. This creates a triangle aligned with the gaussian's orientationQUALCOMM Ref. No. 2501501WO- 16 / 130 -and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the gaussian.

[0057] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the gaussian. When the mesh is modified, the gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the gaussians when the mesh is animated or otherwise modified. During rendering, the gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of gaussians of the gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between gaussians and triangular faces.

[0058] Throughout this disclosure, references to gaussians, gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D gaussians, 2D gaussians, mesh-integrated gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the ty pe of gaussian representation used, as described herein.

[0059] The following elements may be determined based on the parameters above:• S is a 3x3 matrix similar to the covariance matrix in gaussian probability distributions.• c is a 3D view-dependent color vector for each gaussian, computed using the set of spherical harmonics coefficients of the gaussian.

[0060] The three dimensions of the color vector c may correspond to three different color component values, such as a red value, a green value, and a blue value, that specify a color for the gaussian. The color vector c is view-dependent in the sense that the gaussian may appear to have different colors depending on a position from which the gaussian is viewed, e.g., due to reflections. The view-dependent color vector c for the gaussian may be derived from a camera position f, the center position p, of the gaussian, and the SH coefficient array H, of the gaussian. The camera position f and the center position p, may each be a vector of spherical coordinates (0, cp). Thus, a viewing direction vector comprising a Q and a coordinate value that defines a viewingQUALCOMM Ref. No. 2501501WO- 17 / 130 -direction from the camera position f to the center position p(may be described as (f-p;). Therefore, the view-dependent color vector c for a gaussian z having central position p from camera position f may be written as c / (f - p / , Hz) or simply c(f - p, H).

[0061] In certain 3D gaussian Splatting (3DGS) processes, H, includes up to 16 spherical harmonics coefficients per color vector component (e.g., 16 spherical harmonics components for red, 16 spherical harmonics components for green, and 16 spherical harmonics components for blue). A Tenderer may calculate a value of a color component in the view-dependent color vector c based on a linear combination LC of a set of terms Ti... Ymax scaled by the spherical harmonics coefficients in H, for the color component. That is, the Tenderer may calculate the linear combination LCtj for gaussian i and color component j as:LCj I, J j = H; L, J,, i± ± + H; l)J;, 7 fy + H> L, J i t mlfLaClxA, naX’=1,2,3, (1)Equation (1) may be rewritten as follows:C; = Hi y, (2)where II(;1, II,2, etc., are spherical harmonics coefficients for gaussian z in H;for the color component j. Each of the terms in vector y may be defined by the following spherical harmonics equation:In equation (3) above, the values (0, ) specify the viewing direction vector derived from f and pi. The values I and m are the degree and order of the spherical harmonics equation, where - I < m < I. Thus, in an example where there are 16 spherical harmonics coefficients, the maximum value of / is 3, and when I is 0. m is 0; when / is 1, m is 1, 0, -1; when I is 2, m is 2, 1, 0, -1, -2, and so on. Thus, Yi may correspond to / being 0 and m being 0, ¥2 may correspond to I being 1 and m being 1, Y3 may correspond to I being 1QUALCOMM Ref. No. 2501501WO- 18 / 130 -and m being 0, 14 may correspond to I being 1 and m being -1, Is may correspond to I being 2 and m being 2. and so on. With respect to conventional RGB values, the Tenderer may calculate the value of a color component in the view-dependent color vector c by applying a sigmoid function to LC and multiplying by a dynamic range scaling factor, such as 255.

[0062] To generate a rendered image based on a set of gaussians, a Tenderer may determine color values of pixels in the rendered image. A rendered image is a type of digital image that is composed of a grid of individual pixels, each with its own color vector. Each pixel is associated with a camera ray extending from the pixel through a virtual aperture of a virtual camera. In conventional 3DGS rendering, for each pixel of the rendered image, the Tenderer may identify gaussians that intersect the camera ray associated with the pixel and sort the identified gaussians based on a depth from the virtual camera. For each of the identified gaussians, the Tenderer calculates an opacity value for a location within the gaussian based on the position vector of the gaussian, the maximum opacity of the gaussian, the orientation vector of the gaussian, and the scale vector of the gaussian. The Tenderer calculates the opacity' value (a) for 3D location x along the camera ray within a gaussian i (i.e., a;(x)) as follows:1,2, ■ • •, N, (4)

[0063] In equation (4) and throughout this disclosure, the exp function is ex, where e is Euler’s number, (,.)Tindicates matrix transposition, [...]-1indicates a matrix inversion, and N indicates the total quantity of identified gaussians that intersect the camera ray. In general terms, equation (4) scales the maximum opacity value ( ) of the gaussian based on a gaussian distribution function according to a distance of the location x from a center pi of the gaussian. The location x may be defined as a vector of coordinates, e.g., spherical coordinates, Cartesian coordinates, etc.

[0064] When processing a location x within a gaussian, the Tenderer also calculates a view-dependent color vector c for the gaussian. The view-dependent color vector c may include color component values that define a color for the gaussian, such as red, green,QUALCOMM Ref. No. 2501501WO- 19 / 130 -and blue values. Considering a virtual camera with a focal point (i.e., virtual aperture) at position f, the Tenderer may calculate an alpha-scaled color vector for gaussian i. The alpha-scaled color vector n specifies a contribution of a location x within the gaussian i to a color of a pixel. The alpha-scaled color vector may be calculated using the following formula:I7(x,f) = a;(x) c(f -pi, Hi),t = 1,2, -, N. (5)

[0065] Thus, equation (5) determines an alpha-scaled color vector r for a location x viewed from position f by scaling the view-dependent color vector c of a gaussian i that contains the location x based on the previously determined opacity value for the location x (i.e., oq(x)). The view-dependent color vector c may be determined using a spherical harmonics equation based on the SH coefficient matrix Hi of the gaussian and the differences between coordinates the center position of the gaussian (f - pi) as described above. The view-dependent color vector of a gaussian may be dependent on the viewing position f because of the directions of light sources, which can cause shadows, reflections, and other optical effects.

[0066] Depending on the size and orientation of a gaussian and a discretization of the 3D space into discrete locations, there may be multiple locations along a camera ray that intersect the same gaussian. Hence, the Tenderer may calculate multiple opacity values and color vectors for the same gaussian. The Tenderer may use these opacity7values and color values in a rendering equation for determining a color vector of a pixel of a rendered image. However, for ease of explanation, this disclosure and the equations thereof assume that there is only one location per gaussian.

[0067] An advantage of using the formula of equation (4) is that mapping gaussians to the 2D rendered image can be approximated by a 2D gaussian with a 2x2 covariance matrix that can be computed asS2D= J WS3DWTJT, (6)QUALCOMM Ref. No. 2501501WO- 20 / 130 -where W is a matrix defined by the camera’s image-generation transformation, and J is the Jacobian matrix defined by an affine approximation of the projective camera transformation.

[0068] For ease of explanation, this disclosure omits some details about how rendering is discretized for image pixels, and considers that, for all gaussians. there are opacity values ctj [m, n] and color vectors ct[m, n] defined for each image pixel with integer image coordinates [m, n]. The image coordinates of a pixel indicate a 2D location of the pixel within a rendered image. In other words, each pixel of a rendered image is identified by a set of coordinates m and n. Due to the transparency of gaussians, before a scene is rendered, the gaussians to be rendered are sorted according to their distance from the camera position f. Formally, a sorting sequence s[fc] of indexes of gaussians is determined according to increasing distance, such thatk > I => ||f - psW|| > ||f — Ps[ J IhZ= 1,2, T. (7)In equation (7), the double straight brackets (i.e., ||... ||) indicate Euclidean distance.

[0069] Thus, the Tenderer processes locations (e.g., gaussians) along the camera ray associated with a pixel in an order from closest to the virtual camera to further from the virtual camera. Hence, if the opacity values of locations within the gaussians intersected by the camera ray associated with the pixel reach a level corresponding to complete opacity, later gaussians that intersect the camera ray do not contribute to the color vector of the pixel. Hence, the Tenderer can skip processing of locations along the camera ray associated with the pixel that intersect the later gaussians.

[0070] The Tenderer may compute a final color vector for a pixel based on the opacity values and the view-dependent color values for the locations along the camera ray associated with the pixel. For instance, after sorting the gaussians based on depth, the Tenderer may compute a rendered color vector r for the pixel according to the following equation, which corresponds to the computer graphics technique of alpha-blending:QUALCOMM Ref. No. 2501501WO- 21 / 130 -

[0071] In equation (8), N indicates a total quantity of graphics elements (e.g., gaussians) along the camera ray associated with the pixel at coordinates [m, n]. The termcs[fc] [m>n] indicates a color vector for a distance-sorted graphics element having index value k in a sorting order of the graphics elements. In an example where the graphics element is a gaussian, cs[k] [m, n] may be a view-dependent color vector for the gaussian. The term as[kj [m, n] indicates an opacity value for the distance-sorted graphics element having an index value k in the sorting order. Similarly, as[q [m, n] indicates an opacity value for the distance-sorted graphics element having an index value I in the sorting order. Note that in equation (8), the termis calculated individually for each of index values k=l to N so that the scaled color vector for the graphics element with index value k (cs[k] [m, n] as[k] [m, n]) is further scaled based on the opacity values of graphics elements closer to the camera position than the graphics element with index value k. Thus, a Tenderer using equation (8) sorts the graphics elements based on depth from the camera position.

[0072] In complex scenes, like those used in computer games, most of the graphics elements are assumed to be opaque (i.e., cq = 1 in the object’s surface), so equation (8) can be implemented in a massively parallel manner by storing, for each pixel, the minimum depth of the elements already rendered, and only updating the color vector of a pixel if the newly visited element has smaller depth (Z-buffer updates). This is what enables current graphics processing units (GPUs) to render complex scenes at high resolutions and frame rates. On the other hand, when there is transparency, equation (8) defines a strictly sequential order for processing elements, and the only way to parallelize rendering is to create sets of non-overlapping elements and implement sequential alpha-blending on each set.

[0073] This made introduction of transparency in games challenging, but transparency is needed to represent many features, like flames and smoke, clouds, ‘‘force fields,'’ and so on. One proposed empiric solution to avoid sorting, named Order-IndependentQUALCOMM Ref. No. 2501501WO- 22 / 130 - Transparency (OIT), is to approximate equation (8) with a weighted sum. There are several OIT variants, such as to replace equation (8) with the following equation for determining a rendered color vector f for a pixel at coordinates [m, n] of a rendered image:where ztis the distance to a virtual camera, c0[m, n] is the background color for the pixel at position [m,n] in the rendered image, and w(z,a) is a function that decreases with distance, so that objects nearer to the camera are assigned larger weights in the alphanormalized weighted sum of equation (9). The Tenderer may calculate the term a[m, n] using the following equation:Note that equation (9) is a weighted sum of a first value and a second value, where the first value is the background color vector co for the camera ray associated with pixel \m,n\ scaled by the accumulated opacity (a[m, n]) of the camera ray, and the second term is a composited color vector scaled by the accumulated transparency (1 — a[m, n]) of the camera ray. In the OIT rendering of equation (9), there are two summation operations ■■■ f°rdetermining a color vector for a pixel, and since addition is commutative, the terms in the summation operations can be computed in any order. However, the final color vector for a pixel can only be computed after the two sums are known, which can be done with a single pass over all pixels, performing the divisions in equation (9). In other w ords, the Tenderer can perform the summation operations for all pixels, and then, after the summation operations for all pixels are complete, perform the division operations for all pixels. Since OIT uses the term w(z, a) instead of the actual opacity values of closer graphics elements, OIT may eliminate the need for sorting the graphics elements based on depth during rendering. However, use of the term iv(z, a) instead of the actual opacity values of closer graphics elements means that OIT only provides an approximation of alpha blending and does not model truly opaque surfaces.QUALCOMM Ref. No. 2501501WO- 23 / 130 -

[0074] Because OIT allows rendering transparency without sorting, OIT may be considered for eliminating the computationally expensive sorting phase of gaussian splatting rendering. However, while OIT methods have been extensively used in computer games for several years, the OIT methods were developed to be used with conventional computer graphics, and only meant to represent semi-transparent elements. Furthermore, the OIT methods are not expected to generate good approximations since the OIT methods are used with elements of shape and appearance where differences are difficult to see, like flames and smoke. Finally, human intervention may be needed to identify when the OIT methods are not working correctly, and to fix problems.

[0075] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology' is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms "a.” "'an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG. 1 depicts a device 102 including one or more processors (“processor(s)” 130 of FIG. 1), which indicates that in some implementations the device 102 includes a single processor 130 and in other implementations the device 102 includes multiple processors 130. For ease of reference herein, such features are generally introduced as “one or more” features and may subsequently be referred to in the singular or optional plural (as indicated by “(s)”) unless aspects related to multiple of the features are being described.

[0076] As used herein, the terms “comprise,” “comprises,” and “comprising” may be used interchangeably with “include,” “includes,” or “including.” Additionally, the term “wherein” may be used interchangeably with “where.” As used herein, “exemplary” indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). AsQUALCOMM Ref. No. 2501501WO- 24 / 130 -used herein, the term “set’" refers to one or more of a particular element, and the term "‘plurality” refers to multiple (e.g.. two or more) of a particular element.

[0077] The phrase “corresponds to” as used herein is a relational phrase indicating correspondence, equivalence, or matching. For example, if A corresponds to B. then A is B, there is a mapping between A and B. or A matches B. The phrase “is associated with” as used herein is a broad relational phrase indicating a looser or more general relationship such as, for example, a categorical relationship (e.g., A is part of or belongs to B), a causal relationship (e.g., A causes B), a logical relationship (e.g., If A then B), correlation (e.g., when A is present B is present), a structural relationship (e.g., the B that is coupled to A), and other possible relationships. Correspondence always includes association; whereas association can, but does not always, indicate correspondence. For example, if A is associated with B, there can be a mapping between A and B that could be described as correspondence between A and B.

[0078] As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, “directly coupled” may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0079] In the present disclosure, terms such as “obtaining,” “determining,” “calculating,” “estimating.” “shifting,” “adjusting,” etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similarQUALCOMM Ref. No. 2501501WO- 25 / 130 -operations. Additionally, as referred to herein, “obtaining,” “generating,” “calculating,” “estimating,” “using.” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “obtaining,” “generating,” “calculating,” “estimating,” or “determining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, receiving, or accessing the parameter (or signal) that is already generated, such as by another component or device.

[0080] As used herein, “image data” and “pixel data” should be understood according to any of their usual and customary meanings in the fields of image processing, computer vision, photography, medical imaging, and / or artificial intelligence. For example, image data includes a digital representation of visual information, such as color, intensities, and spatial arrangement of portions of an image, and may also include metadata and / or auxiliary information, such as a time, a date, and / or a location associated with the image data; field of view information; camera pose information; camera intrinsic parameters; etc. The color, intensities, and spatial arrangement information of an image is represented in pixel data, where each pixel has a corresponding set of values representing color, intensity, and position within a field of view of the image.

[0081] FIG. 1 is a block diagram of an illustrative system 100 that is operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure. In FIG. 1, the system 100 includes a device 102 that includes a memory 120 configured to store one or more gaussian scene representations, such as illustrative gaussian scene representation 154, and one or more processors 130 configured to selectively update one or more of the gaussian scene representation(s) based on input data.

[0082] FIG. 1 also illustrates a schematic representation of a scene 132 that can be represented by the gaussian scene representation 154. The scene 132 can be a 3D scene or a 4D scene, where a fourth dimension of the scene 132 is time, such that the 4D scene models the 3D scene over time. FIG. 1 also illustrates a view 136 associated with the scene 132. For example, the view 136 represents what a camera 134 with a particular pose and field of view would capture of the scene 132.QUALCOMM Ref. No. 2501501WO- 26 / 130 -

[0083] The device 102 includes or is coupled to one or more sources 104 of input data. For example, the source(s) 104 illustrated in FIG. 1 include one or more image sources 106, one or more sensors 110, a modem 114, or a combination thereof. In other examples, the source(s) 104 can also, or alternatively, include other devices or systems, such as a memory device, a media player, a game system, etc. Although the image source(s) 106, the sensor(s) 110. and the modem 114 are illustrated outside the device 102 in FIG. 1: in other examples, the image source(s) 106, the sensor(s) 110, the modem 114, or a combination thereof, are integrated within the device 102.

[0084] The image source(s) 106 are configured to provide image data 108 to the device 102. For example, the image source(s) 106 can include or correspond to one or more still cameras, one or more video cameras, or both. As another example, the image source(s) 106 can include a memory' that stores the image data 108 or a computer or similar device that generates the image data 108 by executing instructions, such as a game engine that generates images to represent a virtual reality, augmented reality, or mixed reality scene. The image source(s) 106 can be controlled manually (e g., based on user input) or automatically (e.g., based on sensor data 112, based on position information, based on a schedule, etc.).

[0085] The sensor(s) 110 are configured to provide sensor data 112 to the device 102. For example, the sensor(s) 110 can include a light sensor that is configured to generate light data representing lighting intensity or lighting changes within a scene. As another example, the sensor(s) 110 can include a motion sensor that is configured to generate motion data representing movement of the device 102, movement of another device, or movement within a portion of the scene 132. In still other examples, the sensor(s) 110 can include one or more ranging sensor(s), such as sonar, lidar, or radar sensor(s). In some cases, the image data 108 can be processed to generate motion data, light data, or both.

[0086] The modem 114 is configured to provide data 116 received from a remote device to the device 102. The data 116 received via the modem 114 can include sensor data 112, image data 108. user input, mapping data 122, reference images 124, gaussians 156 of a gaussian scene representation 154, other data, or a combination thereof.QUALCOMM Ref. No. 2501501WO- 27 / 130 -

[0087] In a particular aspect the gaussian scene representation 154 can be stored in the memory 120. The memory 120 can also include one or more buffer(s) 118 configured to store data received from the source(s) 104 for analysis and / or use in updating the gaussian scene representation 154. For example, the buffer(s) 118 can include an image buffer configured to store the image data 108, a sensor data buffer configured to store the sensor data 112, or both. The memory 120 may also be configured to store reference images 124. The reference images 124 can include one or more training images 126 (e.g., an image of the scene 132 used to train the gaussian scene representation 154), synthetic images 128 (e.g., an image of the scene 132 generated as output by the gaussian scene representation 154), or both. The reference images 124 can be used as a reference for a current state of a gaussian scene representation 154 for purposes of determining whether and / or how to update the gaussian scene representation 154 based on newly received input pixel data (e.g., at least a portion of the image data 108).

[0088] In FIG. 1, the processor(s) 130 include a selective model update pipeline 140, a Tenderer 158, a controller 160, and a view selector 1 2. FIG. 1 also shows the gaussian scene representation 154 in the processor(s) 130, which may be the case during certain operations. However, in other situations, such as during a partial update of the gaussian scene representation 154, only a subset of the gaussian scene representation 154 (e.g., selected gaussians 156) may be loaded to the processor(s) 130. Similarly, the view selector 162 and the Tenderer 158 may only be loaded to the processor(s) 130 for particular operations, such as rendering the view 136 or rendering views during a full update of the gaussian scene representation 154.

[0089] As explained further below, the selective model update pipeline 140 is configured to manage updating of the gaussian scene representation 154 based on input pixel data. The Tenderer 158 is configured to generate views (e.g.. the view 136) of the scene 132 based on the gaussian scene representation 154 and based on view selection data from the view selector 162. The view selector 162 is configured to provide the gaussian scene representation 154, the Tenderer 158, or both, with the view selection data, where the view selection data indicates a location, a pose, a field of view, a focal length, other information specifying a portion of the scene 132 to be rendered, or a combination thereof. In the example of FIG. 1, the Tenderer 158 is coupled to a display 138 that is configured to display the view 136; however, in other cases, the view 136QUALCOMM Ref. No. 2501501WO- 28 / 130 -can be stored or sent to another device for use. To illustrate, the view 136 can be sent via the modem 114 to a remote device for further processing or display.

[0090] The controller 160 is configured to coordinate tasks among various components of the system 100. For example, the controller 160 can cause one or more of the source(s) 104 to provide data to the processor(s) 130 and / or the memory 120 at particular times or intervals. As another example, various components of the system 100 can provide status information associated with their respective tasks to the controller 160, and the controller 160 can use the status information to initiate or control other components of system 100. For example, when the device 102 receives new image data 108 from the image source(s) 106, the controller 160 can be notified and can initiate the selective model update pipeline 140 or a portion thereof in response to the new image data 108. As another example, when the update strategy selector 144 selects a particular update strategy, the controller 160 can cause components associated with the particular update strategy to be initiated. To illustrate, a full update 148 includes rendering of various views of the gaussian scene representation 154, whereas a partial update 1 0 may not entail rendering of views. Accordingly, when the update strategy selector 144 selects the full update 148, the controller 160 can be notified and can cause the processor(s) 130 to load or initiate the view selector 162 and the Tenderer 158.

[0091] In the example illustrated in FIG. 1, the selective model update pipeline 140 includes one or more data analyzers (such as illustrated data analyzer 142), an update strategy’ selector 144, and update engines 146. In this example, the data analyzer 142 is configured to analyze input data, such as at least a portion of the image data 108, at least a portion of the sensor data 112, or other data, to generate one or more metrics, classification data, or other data indicative of estimated, expected, or detected changes in the scene 132. For example, changes in the scene 132 can be detected based on differences between input pixel data (e.g., at least a portion of the image data 108) and reference pixel data of the reference images 124. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and reference pixel data (e.g., based on a pixel-by-pixel comparison) and generate one or more difference metrics. A difference metric can indicate, for example, a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster ofQUALCOMM Ref. No. 2501501WO- 29 / 130 -difference pixels, a magnitude of differences associated with the difference pixels, or other values. In some cases, a user can provide user input to designate a particular portion of the input pixel data as a region of interest, and the difference metric(s) can be determined based on the user designated region of interest.

[0092] The update strategy selector 144 is configured to select an update strategy to be used (if any) based on one or more update criteria. For example, the update strategy selector 144 can select from among a full update 148, one or more different types of partial updates (including a partial update 150 in FIG. 1), or no update 152 based on an update criterion (or based on more than one update criteria). If the partial update 150 is to be performed, the update strategy selector 144 may also select a portion of the input pixel data to be used to update a corresponding set of gaussians 156 of a gaussian scene representation 154.

[0093] In a particular aspect, the update strategy selector 144 selects an update strategy based on output of the data analyzer 142 (e.g., based on one or more of the difference metrics). Additionally, or alternatively, the update strategy7selector 144 can select the update strategy based on a device condition associated with the device 102 or a component thereof. For example, the sensor data 112 can include device condition data indicating a location of the device 102, a battery level of the device 102, whether the device 102 is charging, a temperature of the device 102, etc. Additionally, or alternatively, the processor(s) 130, the memory 120, or another component of the device 102 can provide device condition data indicating a level of usage of a computing resource of the device 102. To illustrate, the device condition data can include a processor utilization metric (e.g., a processor load), a memory utilization metric (e.g., memory7usage, cache load, or memory access latency), a communication metric (e.g., a bus load, an available network bandyvidth, or network latency), or a similar metric. Examples of a processor utilization metric include, without limitation, user processor time (e.g., time spent executing user-space processes), system processor time (e g., time spent on kernel-level operations), idle time (e.g., percentage of time a processor 130 is not executing any tasks), input / output wait time, load average (e.g., average number of processes waiting to execute in a given time period), a count of context switches per second, a count of processor throttling events (e.g., due to overheating or power-saving measures), or a combination thereof.QUALCOMM Ref. No. 2501501WO- 30 / 130 -

[0094] For example, the update strategy selector 144 can be configured to compare one or more difference metrics from the data analyzer 142 to corresponding thresholds to select the update strategy. To illustrate, the update strategy selector 144 can compare a value of a difference metric based on the differences between the input pixel data and the reference pixel data to one or more thresholds to determine whether to perform the partial update 150, the full update 148, or no update 152. As anon-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of the difference metric is less than a first threshold, determine to perform the partial update 150 if the value of the difference metric is greater than the first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the difference metric is greater than the second threshold. As a second non-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of a size metric is less than a first threshold, determine to perform the partial update 150 if the value of the size metric is greater than a first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the size metric is greater than the second threshold. In this illustration, the size metric can be based on dimension of a difference region, a count of pixels of the difference region, a count of gaussians 156 of the gaussian scene representation 154 associated with the difference region (also referred to herein as "‘difference gaussians”), or a combination thereof. For example, the size metric can be based on a count of the difference gaussians and a count of gaussians 156 of the gaussian scene representation 154.

[0095] When the update strategy selector 144 determines to perform the full update 148, the update engines 146 perform operations that are similar to initial creation of a gaussian scene representation except that the full update 148 is initialized from the existing gaussian scene representation 154. For example, during the full update 148, all of the gaussians 156 of the gaussian scene representation 154 can be processed from various views (as described further with reference to FIGS. 3A and 3B) to update their respective parameters. In this example, processing the gaussians 156 includes rendering selected views, comparing the rendered views to ground-truth image data (e.g., images of the scene) to determine loss values, and using the loss values to update the gaussians 156 (e.g.. using backpropagation and / or other model training techniques). The full update 148 can also include operations such as adaptive density control which canQUALCOMM Ref. No. 2501501WO- 31 / 130 -generate new gaussians 156 or eliminate or merge gaussians 156. Thus, the full update 148 is very computationally intensive.

[0096] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all of the gaussians 156) of the gaussian scene representation 154 are updated, as described further with reference to FIGS. 3A and 3C. For example, the update strategy selector 144 may indicate that a first subset of gaussians 156 are selected for update and a second subset of gaussians 156 of the gaussian scene representation 154 are not selected for update and are to be held constant. The update strategy’ selector 144 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and optionally a second subset of the input pixel data that is not to be used for the partial update 150.

[0097] In a particular aspect, the update strategy selector 144 selects the first subset of gaussians 156 to be updated and the first subset of the input pixel data used in the partial update 150 based on differences between the input pixel data and reference pixel data of the reference images 124, based on the sensor data 112, based on user input, or a combination thereof. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data can be used to estimate the difference region. To illustrate, changes in lighting, as detected by a light sensor, associated w ith a particular area may be expected to correspond to differences betw een the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by a motion sensor and associated with a particular area may¬ be expected to correspond to differences betw een the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data. As another example, a user can provide user input designating the difference region.

[0098] The update strategy selector 144 can use the mapping data 122 (e.g., pixel-to-gaussian mapping data) to identify a subset of the gaussians 156 associated with pixels of the difference region. In this example, the pixels of the difference region can be used for the partial update 150, and the subset of the gaussians 156 associated with pixels of the difference region can be updated during the partial update 150. Other gaussians 156 of the gaussian scene representation 154 are not modified during the partial update 150.QUALCOMM Ref. No. 2501501WO- 32 / 130 -

[0099] It is likely that there will not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, a particular gaussian 156 may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view7(e.g., the view7136). Thus, in some cases, the update strategy selector 144 is also configured to use the mapping data 122 (e.g., gaussian-to-pixel mapping data) to identify pixels that are influenced by the subset of the gaussians 156 associated with pixels of the difference region. In such cases, the pixels of the input pixel data that correspond to the pixels that are influenced by the subset of the gaussians 156 are included in the training data used for the partial update 150. Updating a gaussian in this context refers to changing one or more parameters of the gaussian.

[0100] In a particular aspect, the update strategy selector 144 is configured to generate and / or update the mapping data 122. For example, the update strategy7selector 144 can use saliency mapping to estimate which gaussians 156 most affect particular output pixel data from the gaussian scene representation 154 to generate pixel-to-gaussian mapping data and / or can use saliency mapping to estimate w hich pixels of input pixel data are most important to training of particular gaussians 156 of the gaussian scene representation 154 to generate gaussian-to-pixel mapping data.

[0101] Saliency mapping generally involves at least one forward pass through rendering operations (e.g., projection and rasterization) to generate gradient values used to determine the pixel-to-gaussian mapping data and at least one backward pass through the rendering operations to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially if the gaussian scene representation 154 is large and / or updated frequently. To reduce the computational demand associated with running the gaussian scene representation 154 to generate mapping data 122, the selective model update pipeline 140 can use a staged approach to manage the mapping data 122. In the staged approach, after mapping data 122 (e.g., pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, the selective model update pipeline 140 stores the mapping data 122 in a manner that facilitates reuse of the mapping data 122. For example, after mapping data associated with a particular region of interest (e.g., a range of pixels, an object, or some other identifiable portion of the input pixel data) is generated, the mapping data 122 canQUALCOMM Ref. No. 2501501WO- 33 / 130 -be stored along with information identifying the region of interest in a searchable data store. As one example, the region of interest can be associated with an object or a portion of an object in the scene 132. In this example, the mapping data 122 can include information that maps the object (or the portion of the object) to gaussians that represent the object. The mapping data 122 can also, or alternatively, include pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both, associated with the object. In this example, object detection (which uses fewer computing resources than saliency mapping) can be used to process input pixel data, and if an object detected in the region of interest is represented in the mapping data 122 the mapping data entry associated with the object can be retrieved from the mapping data 122. In contrast, if the object is not represented in the mapping data 122, a mapping data generation process (e.g., saliency mapping) can be performed.

[0102] In some cases, the update strategy selector 144 can use a staged approach with multiple data stores that each include mapping data 122, where a first stage includes a set of the most frequently used mapping data 122, a second stage includes a set of next most frequently used mapping data 122, and so forth. In such cases, full mapping data generation is performed if none of the searchable data stores includes the desired mapping data.

[0103] In a particular aspect, the update strategy selector 144 is configured to select an update strategy based, at least in part, on a device condition associated with the device 102. In some examples, the update strategy selection decision is based on the device condition and is independent of the magnitude or type of differences detected or expected in the input pixel data. For example, irrespective of the update strategy the update strategy7selector 144 selects based on the detected or expected differences in the input pixel data, the update strategy selector 144 can select a different update strategy’ based on the device condition. In this example, the update strategy selected based on the device condition prevails or sets an upper bound on the type of update actually performed. To illustrate, if the device condition indicates that usage of computational resources is relatively low (e.g., below a first threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the full update 148, the partial update 150, or no update 152. However, if the device condition indicates that usage of computational resources is relatively highQUALCOMM Ref. No. 2501501WO- 34 / 130 - (e.g., above the first threshold and below a second threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the partial update 150 or no update 152. Further, if the device condition indicates that usage of computational resources is very' high (e.g., above the second threshold), the update strategy selector 144 selects no update 152 irrespective of the detected or expected differences in the input pixel data.

[0104] In a particular aspect, results of operations performed by the device 102 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation 154 in which one or more parameters of one or more of the gaussians 156 has been modified), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views 136 generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with the scene 132. For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of the device 102 or another device, such as to turn on or turn off a display based on an aspect of the scene 132. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory or haptic notification or a combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate aQUALCOMM Ref. No. 2501501WO- 35 / 130 -notification to alert a user to an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user's attention to an aspect of the scene 132. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene 132.

[0105] FIG. 2 is a block diagram of an illustrative system 200 that is operable to update one or more gaussian scene representations 154. in accordance with some examples of the present disclosure. Various components, features, and / or elements of the system 200 can be stored in the memory 120 of FIG. 1 as data or instructions, can be performed by the processors) 130 of FIG. 1, or both. Additionally, various components of the system 200 correspond to components of the system 100 of FIG. 1. and can be configured as described with reference to FIG. 1. For example, the system 200 includes the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, and the memory 120, one or more of which can be configured as described with reference to FIG. 1.

[0106] In the example illustrated in FIG. 2, the system 200 includes one or more buses, such as illustrated bus 216, coupled to the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, the memory’ 120, one or more other components or devices, or a combination thereof, via respective bus interfaces 131, 133, 135, 137, 139, 141, and 143. For example, the bus 216 can enable communication of data or signals (e.g., control signals, status signals, etc ), between and among the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, and the memory' 120. The sensors 110 can include one or more motion sensors 210, one or more light sensors 212, one or more device sensors 214, other sensors, or a combination thereof.QUALCOMM Ref. No. 2501501WO- 36 / 130 -

[0107] In FIG. 2, the processor(s) 130 include the selective model update pipeline 140, which in this example, includes the data analyzer 142. the update strategy selector 144. the update engines 146, an update data selector 224, and a mapper 222. The processor(s) 130 optionally also include the controller 160, the Tenderer 158, or both.

[0108] The image source(s) 106 are configured to generate (e.g., capture), receive, or otherwise obtain image data 147 representing a scene. As an example, the image source(s) 106 can include one or more cameras, a memory storing images, etc. In a particular aspect, the image source(s) 106 can provide image data 147 to the bus 216 via the bus interface 131 for communication to the processor(s) 130 (e.g., via the bus interface 143), to the memory 120 (e.g., via the bus interface 141), or both. In FIG. 2, the image data 147 is illustrated as provided to the data analyzer 142; however, in some cases, the image data 147 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142. In some cases, the controller 160 can provide control signals to the image source(s) 106 (e.g., via the bus 216 and the bus interfaces 143, 131 ) to control when images are captured or provided to the memory' 120 and / or the processor(s) 130. When the image source(s) 106 provide the image data 147 to the memory 120, the image data 147 can be stored in an image buffer (e.g., one of the buffer(s) 118 of FIG. 1).

[0109] The sensor(s) 110 are optional, and if present, are configured to generate sensor data 145. For example, the motion sensor(s) 210 can generate motion data indicating movement of a camera of the image source(s) 106 relative to the scene or movement of an object in the scene. As another example, the light sensor(s) 212 can generate light data indicative of changes in lighting of a portion of the scene. As yet another example, the device sensor(s) 214 can generate device condition data. The device condition data can indicate a condition associated with a device that includes the selective model update pipeline 140. To illustrate, the device condition data can indicate a battery level, w hether the device is connected to an external power source, a temperature of the device, a level of usage of a computing resource of the device (such as a processor utilization metric or a memory utilization metric), or other similar device condition information. In FIG. 2. the sensor data 145 is illustrated as provided to the data analyzer 142; however, in some cases, the sensor data 145 can be provided to other componentsQUALCOMM Ref. No. 2501501WO- 37 / 130 -of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142.

[0110] In a particular aspect, the data analyzer 142 is configured to determine differences between input pixel data of the image data 147 and reference pixel data (e.g., pixel data of one or more of the reference images 124 of FIG. 1) and to provide information related to such differences to the update strategy selector 144 and optionally to an update data selector 224. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and the reference pixel data based on a pixel-by-pixel comparison and / or using optical flow methods. As another example, the data analyzer 142 can be configured to estimate or predict differences between the input pixel data and the reference pixel data based on motion indicated in motion data, based on lighting changed indicated in light data, or both.

[0111] The data analyzer 142 may be configured to generate one or more difference metrics associated with the differences between the input pixel data and the reference pixel data. For example, the difference metrics can indicate a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.

[0112] The update strategy selector 144 is configured to select an update strategy based on output of the data analyzer 142. Thus, the update strategy selector 144 can select the update strategy based, at least in part, on the differences between input pixel data and the reference pixel data. For example, based at least in part on data provided by the data analyzer 142, the update strategy selector 144 can signal to the update engines 146 to perform no update 152, the partial update 150, or the full update 148.

[0113] When the update strategy selector 144 indicates that the partial update 150 is to be performed, the update data selector 224 can identify which pixels of the input pixel data are to be used for the partial update 150, which gaussians 156 of the gaussian scene representation 154 are to be updated, or both. For example, the update data selector 224 can use the mapping data 122 to identify' a subset of the gaussians 156 that correspond to (e.g., influence rendering of) pixels of a difference region. The pixels of the difference region can correspond to or include pixels of a region of interest, in whichQUALCOMM Ref. No. 2501501WO- 38 / 130 -case pixel-to-gaussian mapping data can be used to identify the subset of the gaussians 156. The pixel-to-gaussian mapping data can include, for example, a data structure that associates particular pixels of a view of the scene with gaussians of the gaussian scene representation 154 that influence rendering of the particular pixels. For example, when a particular view is rendered (as described with reference to FIG. 3B), the pixel-to-gaussian mapping data can be updated or generated to indicate which of the gaussians 156 influenced rendering of each pixel of the rendered view. Alternatively, the pixels of the difference region can correspond to or include pixels associated with a particular object, in which case object mapping data can be used to identify the subset of the gaussians 156. The object mapping data can include, for example, a data structure that associates particular objects detected in the scene with gaussians 156 that represent the particular objects in the gaussian scene representation 154. For example, the data analyzer 142 can include an object detector that identifies objects based on pixel data. In this example, when a particular view7that includes a detected object is rendered (as described with reference to FIG. 3B), the object mapping data can be updated or generated to indicate w hich of the gaussians 156 influenced rendering of each pixel associated with the detected object.

[0114] The update data selector 224 can also, or alternatively, use gaussian-to-pixel mapping data of the mapping data 122 to identify pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156. Because each gaussian 156 can influence rendering of more than one pixel, the identified subset of gaussians can influence more pixels than the pixels used to identify the subset of gaussians. Thus, for example, the pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156 can include pixels within the region of interest and one or more nearby pixels that are also influenced by the identified subset of the gaussians 156.

[0115] In some cases, the selective model update pipeline 140 includes a mapper 222 that is configured to generate or control the mapping data 122. For example, in FIG. 2, the mapper 222 includes a mapping data generator 270 that is configured to generate the mapping data 122 (or portions thereof), to update the mapping data 122, to determine whether new mapping data 122 is to be generated, or a combination thereof. As a particular example, the mapper 222 can perform operations associated with the staged approach to managing the mapping data 122 described with reference to FIG. 1.QUALCOMM Ref. No. 2501501WO- 39 / 130 -

[0116] In a particular aspect, the mapper 222 uses a staged approach to manage the mapping data 122. In the staged approach, after the mapping data generator 270 generates an entry of the mapping data 122 (e.g., object mapping data, pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or a combination thereof), the mapper 222 stores the mapping data 122 in a manner that facilitates reuse of the mapping data 122. For example, after mapping data associated with a particular region of interest (e.g., a range of pixels, an object, or some other identifiable portion of the input pixel data) is generated, the mapping data 122 can be stored along with information identify ing the region of interest in one of several searchable data stores. For example, in FIG. 2, the mapping data 122 includes first mapping data 272, second mapping data 274, and Nth mapping data 276 (where N is an integer greater than 2). The searchable data stores can be staged such that a set of the most frequently or most recently used mapping data is stored in the first mapping data 272, a set of the next most frequently or next most recently used mapping data is stored in the second mapping data 274, and so forth to the Nth mapping data 276. In some cases, each successive searchable data store is allocated more memory than the previous searchable data store to enable faster search for more recently or more frequently used data.

[0117] The mapper 222 is configured to search for particular mapping data in a sequential manner starting from the first mapping data 272 and progressing through additional sets of mapping data (e.g., the second mapping data 274, the Nth mapping data 276, and possibly one or more additional sets between the second and the Nth mapping data 274, 276) until the particular mapping data is found, or until a final set of mapping data is searched.

[0118] If the particular mapping data is not found at any stage (e.g., in any of the sets of mapping data 272, 274, 276), the mapper 222 can cause the mapping data generator 270 to generate the particular mapping data. The generated mapping data can be stored for subsequent reference. For example, after mapping data associated with a particular region of interest is generated, the mapping data can be stored in a searchable data store (e.g., the first mapping data 272). In this example, when new input image data is to be used for a partial update of the gaussian scene representation 154. a new region of interest in the new input image data is identified, and the searchable data store is searched to determine whether mapping data corresponding to pixels of the new regionQUALCOMM Ref. No. 2501501WO- 40 / 130 -of interest is available (e.g., has already been generated and stored). If the mapping data is available, the mapping data can be retrieved from the searchable data store and used without the need to perform saliency mapping operations to generate the mapping data.

[0119] The region of interest used in the staged approach can be associated with a particular set of pixels, with a particular spatial area within the scene, with a particular object (or a portion of an object) identified in the scene, or with another identifiable feature of the scene. As one example, objects within the scene can be identified and saliency mapping can be used to associate one or more of the objects with gaussians of the gaussian scene representation 154. In this example, when new input image data is obtained, object detection can be performed to identify objects in the new input image data. If a particular object identified in the new input image data is associated with a change that triggers a partial update of the gaussian scene representation 154, the searchable data store is searched to determine whether mapping data is available for the particular object. If the mapping data for the particular object is available in the searchable data store, the mapping data is used for the partial update of the gaussian scene representation 154. If the mapping data for the particular object is not available in the searchable data store, the mapping data can be generated using saliency mapping. Alternatively, if particular mapping data is not available in a first searchable data store (e.g., in the first mapping data 272), a second searchable data store (e.g., second mapping data 274) can be searched for the particular mapping data.

[0120] The mapper 222 can apply an eviction policy to each searchable data. The eviction policies are arranged such that a set of most recently used or most frequently used mapping data is retained at the first searchable data store, and each subsequent searchable data store retains successively less recently used or less frequently accessed mapping data.

[0121] In some cases, the staged approach can use a single searchable data store (e.g., the first mapping data 272), rather than several stages of searchable data stores. In such cases, the single searchable data store can evict the oldest or the least recently accessed mapping data when new mapping data is added. In such cases, mapping data generation is performed if the single searchable data store does not include the desired mapping data. A technical advantage of the staged approach is that significant computingQUALCOMM Ref. No. 2501501WO- 41 / 130 -resources can be preserved by avoiding recreating mapping data when new input image data is to be used for a partial update of the gaussian scene representation.

[0122] In some embodiments, the update strategy selector 144 is configured to select the update strategy based on the differences between input pixel data and the reference pixel data and additional information. For example, the data analyzer 142 can analyze the sensor data 145 and provide analysis results to the update strategy selector 144. In this example, the update strategy7selector 144 can select the update strategy7based, at least in part, on the sensor data 145.

[0123] As one example, the update strategy selector 144 may select to perform no update 152 when the differences between input pixel data and the reference pixel data are insignificant (e.g., less than a first threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are insignificant, and / or when the sensor data 145 indicate that computing resources are strained (e.g., battery level is low, available processing capacity is low, etc.). Continuing this example, the update strategy7selector 144 may select to perform the full update 148 when the differences between input pixel data and the reference pixel data are large (e.g., greater than a second threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are significant, and / or when the sensor data 145 indicate that significant computing resources are available. Further, the update strategy7selector 144 may select to perform the partial update 150 when the differences between the input pixel data and the reference pixel data are constrained to one or more identified regions of interest (based on the image data 147, the sensor data 145, or both) and / or when the sensor data 145 indicate that computing resources are available but may7be constrained.

[0124] Although the example above describes selecting an update strategy from among no update 152, the partial update 150, and the full update 148 based on relative states (e.g., significant, insignificant, large, constrained, etc.), in practice, the selection can be based on comparisons of various metrics to respective thresholds. For example, an '“insignificant” difference or value can correspond to a value of a metric that is less than a first threshold, a ‘“large” difference or value can correspond to a value of the metric that is greater than a second threshold, and a “significant” difference can correspond to aQUALCOMM Ref. No. 2501501WO- 42 / 130 -value of the metric between the first threshold and the second threshold. In some cases, more than two or fewer than two thresholds can be used, depending on the specific implementation.

[0125] When the update strategy selector 144 determines to perform the full update 148, all of the gaussians 156 of the gaussian scene representation 154 are eligible to be updated based on the input pixel data, and all of the input pixel data is used to perform the full update 148. For example, the update strategy selector 144 can provide the input pixel data to the update engines 146, and the update engines 146 can render views of the gaussian scene representation 154. determine loss function values based on comparison of the rendered views to ground-truth views, use the loss function values to calculate gradient values, and use the gradient values to determine updated parameter values 149 for each of the gaussians 156.

[0126] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all) of the gaussian scene representation 154 are updated based on the input pixel data. For example, the update data selector 224 can select a first subset of gaussians 156 that are to be modified, in which case a second subset of gaussians 156 are not to be modified and are to be held constant. The update data selector 224 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and a second subset of the input pixel data that are not to be used for the partial update 150.

[0127] In a particular aspect, the update data selector 224 selects the first subset of the gaussians 156 to be updated and the first subset of the input pixel data used for the partial update 150 based on differences between the input pixel data and reference pixel data, based on the sensor data 145. or both. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data 145 can be used to estimate the difference region. To illustrate, changes in lighting, as detected by the light sensor(s) 212, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by the motion sensor(s) 210 and associated with a particular area may be expected to correspond to differencesQUALCOMM Ref. No. 2501501WO- 43 / 130 -between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data.

[0128] The update data selector 224 can select the first subset of the gaussians 156 to be updated based on the pixels of the input pixel data to be used for the partial update 150 (e.g., based on the difference region). For example, the update data selector 224 can designate a subset of the gaussians 156 that correspond to or affect the pixels to be used for the partial update 150 as eligible to be modified by the partial update 150. In this example, other gaussians 156 of the gaussian scene representation 154 are ineligible to be modified during the partial update 150.

[0129] In many cases, there may not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, particular gaussians may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g., the view 136). Thus, in some cases, the update data selector 224 can use the difference region to identify the particular gaussians associated with pixels of the difference region. In such cases, the particular gaussians can be updated based on a selected set of pixels that include pixels of the difference region and additional pixels (typically near the difference region) that are influenced by the particular gaussians. In a particular aspect, the update data selector 224 (or the mapper 222) can use saliency mapping, look-up tables, or other mapping data to identify associations between pixels and gaussians of the gaussian scene representation 154.

[0130] In a particular aspect, results of operations performed by the system 200 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation with updated gaussians 156), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with a scene (e.g., the scene 132 of FIG. 1). For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input toQUALCOMM Ref. No. 2501501WO- 44 / 130 -one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of a device (e.g., the device 102 or another device), such as to turn on or turn off a display based on an aspect of the scene. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory or haptic notification or combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending playback of media content (e.g.. audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene.

[0131] FIGS. 3A, 3B, and 3C. together, depict a system 300 that illustrates particular aspects of the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The system 300 illustrates additional details related to updating a gaussian scene representation 154, and in particular, illustrates differences between operations associated with performing a full update (e.g., the full update 148 of FIGS. 1 and 2) and a partial update (e.g., the partial update 150 of FIGS. 1 and 2). Due to the complexify ofQUALCOMM Ref. No. 2501501WO- 45 / 130 -the system 300, the system 300 is split over three pages and linked by off-page references.

[0132] The system 300 includes one or more sensors 302, such as a first sensor 302A, a second sensor 302B, and a Pth sensor 302P (where P is an integer greater than two). At least one of the sensor(s) 302 is an image sensor (e.g.. a camera) configured to generate image data. One or more others of the sensor(s) 302 can include or correspond to motion sensors, light sensors, device condition sensors, etc., such as described with reference to FIGS. 1 and 2. The sensor(s) 302 are configured to generate sensor data 303. The system 300 optionally includes a sensor controller 304 that is configured to control operation of the sensor(s) 302 and / or to select a subset of the sensor data 303 to generate sensor data 305, which is used for further processing. If the sensor controller304 is omitted, the sensor(s) 302 can be externally controlled (e.g., responsive to user input), can operate according to a schedule, or can generate the sensor data 303 responsive to detecting a condition (e.g.. in response to detecting movement). If the sensor controller 304 is omitted, the sensor data 305 is the same as the sensor data 303.

[0133] In FIG. 3A, the system 300 includes a pose estimator 306 that is configured to determine pose data (e.g., a position and / or orientation of one of the sensor(s) 302 within a scene), which can be provided along with the sensor data 305 as input 307 to an image subspace selector 312. In some cases, the sensor data 305 can include pose information (e.g., as data or metadata). In such cases, the pose estimator 306 is optional. If the pose estimator 306 is omitted, the sensor controller 304 (or the sensor(s) 302 if the sensor controller 304 is also omitted), provide the input 307 including the sensor data305 and corresponding pose data to the image subspace selector 312.

[0134] During initial generation of the gaussian scene representation 154, the pose estimator 306, the sensor controller 304, or the sensor(s) 302 provide input 311 including sensor data 305 and corresponding pose data to a model initializer 310, which uses similar operations as described with reference to FIG. 3B for the full update 148 to generate initial parameters for the gaussians 156 of the gaussian scene representation 154. After initial generation of the gaussian scene representation 154. the model initializer 310 can be bypassed, and the input 307 can be provided to an image subspace selector 312.QUALCOMM Ref. No. 2501501WO- 46 / 130 -

[0135] In the example illustrated in FIG. 3 A, the gaussian scene representation 154 includes a plurality of gaussians 156, such as: a gaussian 156A. a gaussian 156B. and a gaussian 156Q (where Q is an integer greater than 2). Each gaussian 156 is characterized by a plurality of parameters describing various properties of the gaussian 156. For example, in FIG. 3 A, the gaussian 156A is characterized by an X-axis center position coordinate ("‘center pos. (X)") 180, Y-axis center position coordinate (“center pos. (Y)”) 181, and Z-axis center position coordinate (“center pos. (Z)”) 182. Together, the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 indicate a center position of the gaussian 156A and correspond to the position vector, pf. discussed above with reference to Equations 1-10. In FIG. 3A, the gaussian 156A is also characterized by an X-axis scale parameter 183, a Y-axis scale parameter 184, and a Z-axis scale parameter 185.Together the X-axis scale parameter 183, the Y-axis scale parameter 184, and the Z-axis scale parameter 185 indicate a size and shape of the ellipsoid of the gaussian 156A and correspond to the scale vector. s;, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO) 186, rotation (ql) 187, rotation (q2) 188, rotation (q3) 189) specified as quaternions in the example illustrated in FIG. 3A. Together the rotation parameters 186-189 indicate a spatial orientation of the ellipsoid of the gaussian 156A and correspond to the orientation parameters, q(. discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by an opacity parameter 190, which indicates a maximum opacity of the gaussian 156A. The opacity parameter 190 corresponds to the opacity parameter, tt. discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by a set of color parameters, e.g., color parameters 191, 192, and 193 in FIG. 3A. The color parameter(s) 191-193 can be specified as a set of spherical harmonics (SH) coefficients, Hbas discussed above with reference to Equations 1-10. Although FIG. 3A illustrates the gaussian 156A as characterized by a set of fourteen parameters, in other examples, the gaussians 156 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters 191-193 can be used, depending on the order of the SH coefficients used to represent the colors of the gaussians 156. To illustrate, in some implementations, a reduced set of approximately 14 parameters may be used, where the color parameters 191-193 are simplified to aQUALCOMM Ref. No. 2501501WO- 47 / 130 -single base color value (e.g., an RGB triplet) rather than a full set of spherical harmonics coefficients. In such implementations, the spherical harmonics coefficients may be omitted entirely, reducing memory and computational requirements for storing and updating the gaussians 156.

[0136] The image subspace selector 312 is configured to, based on the input 307, identify one or more patches of pixels (e.g., image patches) for a region of interest (ROI), or more than one ROI, associated with the sensor data 305 to generate ROI data 313. In a particular aspect, the image subspace selector 312 can correspond to, include, or be included within the data analyzer 142 of FIGS. 1 and 2. For example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to an object that is in motion in the scene (e.g., based on motion data from a motion sensor, based on optical flow data from two or more images, etc.). In this example, the ROI can correspond to the image patch (or patches) associated with the object that is in motion. As another example, the image subspace selector 312 can identify7one or more image patches (from one or more images) that correspond to pixels of input pixel data of the sensor data 303 or 305 that differ (by at least a threshold amount) from pixels of reference pixel data. In this example, the ROI can be defined by the location of the image patch (or patches) associated with the pixels that differ between the input pixel data and the reference pixel data. The ROI data 313 identifies one or more sets of pixels of one or more images. As another example, the image subspace selector 312 can identify one or more image patches (from one or more images) based on user input that designates or identifies a region of interest.

[0137] The update strategy selector 144 of the system 300 is configured to select an update strategy7based on strategy' selection data 320. The strategy' selection data 320 can include the sensor data 303 or 305, the input 307, the ROI data 313, other data, or a combination thereof. For example, the update strategy selector 144 can select the update strategy using any of the techniques described with reference to FIGS. 1 and 2 and / or any of the techniques described with reference to FIGS. 4-6. The update strategy selector 144 is configured to provide a selection indication 321 to the update engines 146, where the selection indication 321 causes the update engines 146 to initiate operations associated with the selected update strategy. For example, if the selection indication 321 indicates that a full update strategy is selected, the update engines 146QUALCOMM Ref. No. 2501501WO- 48 / 130 -initiate the full update 148. and if the selection indication 321 indicates that a partial update strategy is selected, the update engines 146 initiate the partial update 150. If no update is selected, the update strategy selector 144 can omit generation of the selection indication 321 or can generate a “no update” selection indication.

[0138] When the update engines 146 select the full update 148. the update engines 146 cause a pose selector 324 of FIG. 3B (as indicated by off-page reference A) to begin operations associated with the full update 148. The pose selector 324 is configured to select a pose 325 from training data 360 that includes a set of training images (e.g., images 361A-361K) and pose data (e.g.. poses 362A-362K) associated with each image 361 A-361K (where K is an integer greater than two). An image associated with the selected pose 325 is used as a ground-truth image 349 for loss calculation after a view associated with the pose 325 is rendered.

[0139] The pose 325 describes the position and orientation of a virtual camera 330 relative to the gaussians 156 of the gaussian scene representation 154 to be rendered as a view. Due to the pose 325 and intrinsic parameters (e.g., focal length, principal point, etc.) of the virtual camera 330, some of the gaussians (e.g., gaussians 331) are visible to the virtual camera 330. and some of the gaussians (e.g., gaussians 333) are not visible to the virtual camera 330. For example, for a pose 325A, a virtual camera 330A can see gaussians 331 A; however, gaussians 333A are not visible to the virtual camera 330A in the pose 325 A. As another example, for a pose 325K, a virtual camera 330K can see gaussians 33 IK; however, gaussians 333K are not visible to the virtual camera 330K in the pose 325K.

[0140] To ensure that the full update 148 updates each gaussian 156 based on all relevant data, the full update 148 generally entails rendering a view for every available pose (e.g., poses 362A to 362K) and comparing each rendered view7to the ground-truth image (e.g., each of images 361 A to 361K) associated with the pose 362 used to render the view7. Thus, performing the full update 148 is similar to operations performed by the model initializer 310 to generate an initial version of the gaussian scene representation 154 based on a set of images except that the full update 148 is initialized from a preexisting gaussian scene representation 154.QUALCOMM Ref. No. 2501501WO- 49 / 130 -

[0141] A projector 326 identifies the gaussians 331 that are visible to the virtual camera 330 for a pose 325. The projector 326 performs gaussian splatting, which entails projecting the visible gaussians 331, which are three dimensional, onto a 2D image plane 332 based on the pose 325 of the virtual camera 330 and the intrinsic parameters of the virtual camera 330. Each gaussian 331 is represented as an ellipse 335 in the 2D image plane 332. The ellipses 335 can overlap one another and can have various sizes and orientations depending on parameters of the gaussians 331 that are projected and the pose 325 of the virtual camera 330. For example, the gaussians 331 A that are visible to the virtual camera 330A in the pose 325A are projected on to a 2D image plane 332A as ellipses 335A, and the gaussians 33 IK that are visible to the virtual camera 330K in the pose 325K are projected on to a 2D image plane 332K as ellipses 335K.

[0142] Data 334 describing the ellipses 335 on the 2D image plane 332 is provided to a differentiable tile rasterizer 340. For example, data 334A describes the ellipses 335A associated with the pose 325A, and data 334B describes the ellipses 335B associated with the pose 325K. The differentiable tile rasterizer 340 is configured to perform operations to blend the ellipses 335 to generate pixel data representing a 2D image that would be captured by the virtual camera 330 for the particular selected pose 325. To generate the pixel data, the differentiable tile rasterizer 340 divides the 2D image plane 332 into tiles (represented in FIG. 3B by bold lines and different shading in the 2D image planes 332A and 332K of the differentiable tile rasterizer 340). For purposes of this description, each tile is treated as corresponding to a single pixel; however, the exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel. Additionally, to simplify the drawings, in FIG. 3B, the 2D image plane 332 includes some regions that are not associated with any gaussian; however, it will generally be the case that each tile is associated with at least one gaussian 156 to provide color information for the tile.

[0143] The differentiable tile rasterizer 340 obtains depth information 342 for the visible gaussians 331 of the pose 325, where depth refers to the distance in the scene of a gaussian from the virtual camera 330. For example, the depth information 342 can be calculated for each of the visible gaussians 331 based on the location of the virtual camera 330 in the scene (as indicated by data describing the pose 325) and the location of the center of the gaussian in the scene (e.g., as indicated by the X-axis center positionQUALCOMM Ref. No. 2501501WO- 50 / 130 -coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 of each gaussian). For example, depth information 342A indicates distances in the scene between each of the gaussians 331 A and the virtual camera 330A, and depth information 342K indicates distances in the scene between each of the gaussians 33 IK and the virtual camera 330K.

[0144] The differentiable tile rasterizer 340 can perform tile-depth association 344 to, for each tile of the 2D image plane 332, sort the ellipses 335 that at least partially overlap the tile based on the depth information 342. Since the gaussians 156 are generally not opaque, a first gaussian that is more distant from the virtual camera 330 in the scene can be partially visible through a second gaussian that is closer to the virtual camera 330. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 340 accounts for the colors of each gaussian of a set of gaussians that influence the particular pixel, the opacities of the gaussians of the set, and the order of the gaussians of the set as viewed from the virtual camera 330.

[0145] To parallelize generation of the pixel data, the differentiable tile rasterizer 340 can perform thread association 346 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources, such computing resources can include one or more cores of one or more central processing unit (CPU), one or more cores of one or more graphics processing unit (GPU), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colors of the gaussians (as determined based on the pose 325 of the virtual camera 330 and color parameters specified by spherical harmonic color coefficients, opaci ty of the gaussians, and the depth information 342).

[0146] Blending the contributions of the gaussians results in pixel data 341 representing an image (e.g., a view) captured by the virtual camera 330 for the particular pose 325. The pixel data 341 is provided to a loss calculator 348, which is configured to compare the pixel data 341 to the ground-truth image 349 corresponding to the pose 325. The loss calculator 348 determines a loss value for each pixel (e.g., a photometric L2 loss) toQUALCOMM Ref. No. 2501501WO- 51 / 130 -quantify how the pixel data 341 differs from the ground-truth image 349. Gradient values 343. 345 based on the loss values are backpropagated through the differentiable tile rasterizer 340 and the projector 326 to estimate updated parameter values 347 for the visible gaussians 331 of the pose 325.

[0147] The full update 148 continues by iteratively repeating the processes of pose selection, projection, rasterization, loss calculation, and backpropagation of gradients until all of the poses 362 have been processed. At one or more points during the full update 148, an adaptive density controller 350 performs operations that change the number of gaussians 156 of the gaussian scene representation 154, such as pruning one or more gaussians 156 (e.g., the gaussian 156B) or splitting one or more gaussians 156. Thus, in some senses, a different gaussian scene representation 154 exists after the full update 148 than existed before the full update 148 since a different number and arrangement of gaussians 156 can be present, and any gaussian 156 that is carried over can have different parameters. For example, in FIG. 3B, the gaussian scene representation 154 is illustrated as including a first gaussian 156 A, a second gaussian 156B, and one or more additional gaussians up to a Qth gaussian 156Q (where Q is an integer greater than 2). After the full update 148, the gaussian scene representation 154 can include more than Q or fewer than Q gaussians 156. Further, the full update 148 is a feedback-based process since, during each iteration, the visible gaussians 331 for a pose 325 are updated based on loss values indicating differences between an image (e.g., the pixel data 341) of a view rendered for the pose 325 and a reference image (e.g., the ground-truth image 349 associated with the pose 325).

[0148] When the update engines 146 of FIG. 3A select the partial update 150, the update engines 146 cause a gaussian selector 354 of FIG. 3C (as indicated by off-page reference B) to select a set of gaussians (e.g., gaussians 156A and 156B in the example of FIG. 3C) that are to be updated based on the RO1 data 313 and the pixel-to-gaussian mapping data of the mapping data 122. As an example, for illustrative purposes, the sensor data 303 of FIG. 3A can include an image 351 of FIG. 3C. In this illustrative example, the image subspace selector 312 can determine that the ROI includes a ball 365 depicted in the image 351. in which case the ROI data 313 can include data identifying the ball 365. The data identifying the ball 365 can include object identification data (e.g., a text label or a numerical designator), a set of pixels thatQUALCOMM Ref. No. 2501501WO- 52 / 130 -represent the ball 365 in the image 351, identifiers of the set of pixels that represents the ball 365 in the image 351, or other data that enables the gaussian selector 354 to identify a set of pixels associated with the ball 365 and select the set of gaussians based on the identified set of pixels and the pixel-to-gaussian mapping data. In a particular aspect, the image subspace selector 312, the gaussian selector 354, or both, are integrated within the update data selector 224 of FIG. 2.

[0149] The gaussian selector 354 uses the ROI data 313 and the mapping data 122 to generate data 353 identifying the selected set of gaussians, and provides the data 353 to a partial updater 356. The data 353 identifying the selected set of gaussians can include the gaussians themselves, memory addresses of the selected set of gaussians, indices of the selected set of gaussians, or other information that enables the partial updater 356 to obtain the selected set of gaussians.

[0150] The partial updater 356 is configured to determine updates 357 to the selected set of gaussians (leaving remaining gaussians 156 of the gaussian scene representation 154 unchanged) to generate updated gaussians 156’. In some implementations, the partial updater 356 can use a feedforward approach to determine the updates 357. For example, the partial updater 356 can determine the updates 357 based on the sensor data 305 of FIG. 3 A. As one example, the sensor data 303 can include motion data that can be used to determine how far an object (e.g., the ball 365) in the scene moved, and the partial updater 356 can determine the updates 357 based on the determined movement of the object. As another example, optical flow can be used to compare the image 351 to one or more reference images to identify movement of one or more objects in the scene, and the partial updater 356 can determine the updates 357 based on the identified movement. As yet another example, the sensor data 303 can include light data which can be used to determine changes in lighting or color in a portion of the scene, and the partial updater 356 can determine the updates 357 based on the changes in lighting or color.

[0151] The partial updater 356 can apply the updates 357 to the selected set of gaussians and store updated versions of the selected set of gaussians (e.g., updated gaussians 15 A’ and 156B’ in FIG. 3C) in an update buffer 358. After all gaussians ofQUALCOMM Ref. No. 2501501WO- 53 / 130 -the selected set of gaussians have been updated, the updated gaussians 156' can be written to a memory where the gaussian scene representation 154 is stored.

[0152] Optionally, in some embodiments, a rendering-based error calculator 359 can be used to check the updated gaussians 156’ before the gaussian scene representation 154 is modified. The rendering-based error calculator 359 is configured to perform projection, rendering, and error calculation, as described with reference to the projector 326, the differentiable tile rasterizer 340, and the loss calculator 348, respectively, of FIG. 3B to generate a rendered view of the gaussian scene representation 154. During the partial update 150, the rendering-based error calculator 359 can determine (e.g. obtain from the sensor data 305 or from the pose estimator 306 of FIG. 3A) a pose associated with the image 351. The rendering-based error calculator 359 can render a view of the gaussian scene representation 154 (with the updated gaussians 156') from the pose associated with the image 351, and compare the rendered view to the image 351 to determine whether the updated gaussians 156’ are acceptable or whether additional updates 357 should be determined. For example, the rendering-based error calculator 359 can determine loss value(s) (e.g., using similar operations as described with reference to the loss calculator 348 of FIG. 3B) based on differences between the image 351 and the rendered view. In this example, if the loss value(s) satisfy a threshold, the updated gaussians 156’ are used to update the gaussian scene representation 154. However, if the loss value(s) do not satisfy a threshold, the partial updater 356 generates a second set of updated gaussians 156’ and repeats the process above to determine whether loss value(s) for the second set of updated gaussians 156’ satisfy the threshold. This process can be repeated iteratively until a termination condition is reached. The termination condition can include identification of a set of updated gaussians 156’ that satisfies the threshold, or performance of a specified number of iterations. If no set of updated gaussians 156’ that satisfies the threshold is identified after the specified number of iterations, the partial updater 356 can change to another approach to generate updates 357, or the update strategy selector 144 of FIG.3A can select a different update strategy (e.g., no update, a different type of partial update, or the full update 148). In some examples, rather than process one set of updated gaussians 156’ at a time, as described above, the partial updater 356 can generateQUALCOMM Ref. No. 2501501WO- 54 / 130 -multiple candidate sets of updated gaussians 156’ in parallel, and check which, if any, best match the image 351.

[0153] In this example, the ROI data 313 can include object data identifying an object (e.g., the ball 365) that has changed or moved (as indicated by the sensor data 305), and the mapping data 122 can include the object-based mapping data, which identifies a set of the gaussians 156 that represent the object. The gaussian selector 354 can select the set of gaussians that represent the object as the set of gaussians that are to be updated. In the specific example shown in FIG. 3C, the gaussians 156A and 156B represent a ball 367 in the scene; thus, the data 353 includes the gaussians 156A and 156B.

[0154] As another example, the ROI data 313 can identify a first set of pixels that have changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the pixel-based mapping data, which identifies a set of the gaussians 156 that correspond to the first set of pixels. The gaussian selector 354 can select the set of gaussians that correspond to the first set of pixels as the set of gaussians that are to be updated. The selection data 353 can include the selected set of gaussians or information indicating how the partial updater 356 can access the selected set of gaussians, such as memory addresses of the selected set of gaussians. indices of the selected set of gaussians, etc.

[0155] The partial updater 356 is configured to modify parameters of the selected set of gaussians (e.g.. the gaussians 156A and 156B in this example) based on the sensor data 305, the input 307, and optionally other data. For example, when the ROI data 313 indicates that an object in the scene has moved, the partial updater 356 can change location parameters and / or orientation parameters of the gaussians 156A and 156B that represent the object in the gaussian scene representation 154 in order to update the gaussian scene representation 154 to reflect movement of the object. In other cases, the partial updater 356 can change color parameters (e.g., spherical harmonics coefficients) and / or opacify parameters of the gaussians 156A and 156B to represent lighting changes in the scene.

[0156] A technical benefit of performing the partial update 150 rather than the full update 148 is that significant computer resources can be conserved. For example, when the update to be performed can be calculated or estimated based on the sensor data 305,QUALCOMM Ref. No. 2501501WO- 55 / 130 -the rendering operations associated with the projector 326 and the differentiable tile rasterizer 340, the loss calculations performed by the loss calculator 348, and backpropagation of gradient values 343, 345 based on the loss values can be omitted during the partial update 150. Further, even if the partial update 150 uses the renderingbased error calculator 359 to check one or more candidate updated gaussians 156’, only one or a few views are rendered, rather than views associated with all of the images 361 of training data 360 of FIG. 3B, saving significant rendering time and resources.Further, backpropagation of gradients based on the loss values can be omitted during the partial update 150, which also results in conservation of significant computing resources.

[0157] As another non-limiting example, the partial updater 356 may be configured to perform a partial update of a gaussian scene representation by applying spatial transformations to one or more selected gaussians of the gaussian scene representation. For example, in FIG. 3D, a gaussian scene representation 370C is shown. In an aspect, the gaussian scene representation 370C may be the same as the gaussian scene representation 370B of FIG. 3C. As explained above, the gaussian scene representation 370C includes a plurality of gaussians and may be used to render an image depicting a portion of a soccer pitch. In FIG. 3D, an input image 373 is shown and may represent a current state of the scene. FIG. 3D also shows several reference images (including a reference image 372 and a reference image 372n). In this example, the reference image 372 is a rendered view before a partial update based on the input image 373, and reference image 372n is a rendered view after the partial update based on the input image 373. In the example of FIG. 3D, the soccer ball 365 is positioned in the right comer of the goalkeeper box in the reference image 372, but is located towards the left edge of the goalkeeper box in the input image 373. After the partial update, the soccer ball 365 may be rendered in a position that is more closely located towards the left edge of the goalkeeper box, as shown in the reference image 372n.

[0158] To perform the partial update, the partial updater 356 may select one or more gaussians to be updated from among a plurality of gaussians of the gaussian scene representation 370C. As explained above, the gaussians may be selected based on obtained pixel data, e.g., based on mapping data 122, image subspace selector 312, the mapper 222, the gaussian selector 354, or other techniques disclosed herein. In someQUALCOMM Ref. No. 2501501WO- 56 / 130 -embodiments, the selected gaussians may correspond to portions of the gaussian scene representation 370C that have changed over time, such as movement of the soccer ball 365. The partially updated gaussian scene representation after the partial update may more accurately represent a state of the scene at a particular point in time, e.g., due to movement of one or more objects in the scene. It is noted that the particular point in time may be a current point in time, a future point in time, or another historic point in time.

[0159] In the example of FIG. 3D, the partial updater 356 may be configured to spatially transform the selected gaussians of the gaussian scene representation to produce an updated gaussian scene representation 370D. For example, a spatial transformation 371 may be applied to the selected gaussians, e.g., gaussians associated with the soccer ball 365. In the updated gaussian scene representation 370D the selected gaussians may be in different locations based on the spatial transformation 371.

[0160] In some embodiments, the partial updater 356 may be configured to spatially transform the selected gaussians by adjusting parameters of the selected gaussians. For example, the gaussian scene representation 370C represents a scene in three dimensions using a plurality of gaussians from which the selected gaussians may be selected. Each gaussian of the gaussian scene representation may be associated with parameters including a three-dimensional coordinate location. The spatial transformation 371 applied by the partial updater 356 may modify three-dimensional coordinate locations associated with the selected gaussians to adjust the parameters of the selected gaussians, thereby spatially transforming the selected gaussians, e.g., in 3D space.

[0161] In some additional, or alternative embodiments, the plurality of gaussians may each be associated with one or more orientation parameters. The spatial transformation 371 applied by the partial updater 356 may be configured to modify one or more orientation parameters associated with the selected gaussians to adjust parameters of the selected gaussians to spatially transform the selected gaussians, e.g., in 3D space. In some embodiments, the spatial transformation 371 applied by the partial updater 356 may be configured to modify’ both location and orientation parameters of the selected gaussians.QUALCOMM Ref. No. 2501501WO- 57 / 130 -

[0162] Optionally, at least one view of the scene may be rendered based on the gaussian scene representation 370D. The at least one rendered view may be compared with the input pixel data, e.g., a ground truth image of the scene, such as the input image 373. An error metric may be determined based on the comparison of the at least one rendered view. In some embodiments, the error metric may be determined by the rendering-based error calculator 359. The partial updater 356 may receive the error metric from the rendering-based error calculator 359 and may determine whether to further update the gaussian scene representation based on the error metric. For example, the partial updater 356 may apply a spatial transformation 371’ to the gaussian scene representation to produce an updated gaussian scene representation 370E. In some embodiments, the spatial transformation 371’ applied based on the error metric may modify three-dimensional coordinate locations associated with the selected gaussians. In some additional, or alternative, embodiments, the spatial transformation 371’ applied based on the error metric may modify one or more orientation parameters associated with the selected gaussians. In some additional, or alternative, embodiments, the spatial transformation 371’ applied to the gaussian scene representation based on the error metric may modify three-dimensional coordinate locations associated with the selected gaussians and one or more orientation parameters associated with the selected gaussians.

[0163] In some embodiments, the rendering of one or more views of the scene based on the gaussian representation obtained after applying the spatial transformation and determining of the error metric(s) may be performed iteratively. For example, the spatial transformation, rendering, and calculating of the error metric(s) may be performed until the error metric(s) satisfies a threshold error. To illustrate, the threshold error may be indicative of a percentage difference between the view rendered based on the updated gaussian scene representation and the input image 373. To illustrate, a gaussian scene representation 370D may be obtained by applying a first spatial transformation, e.g., spatial transformation 371, to the gaussian scene representation 370C. An error metric may be calculated based on a view of the scene rendered based on the gaussian scene representation 370D and the input image 373. The partial updater 356 may determine, based on the error metric(s), that further updating of the gaussian scene representation is required, e.g., because the threshold error is not satisfied, and may apply a secondQUALCOMM Ref. No. 2501501WO- 58 / 130 -spatial transformation 371' to the gaussian scene representation to obtain a gaussian scene representation 370E. This process may continue until the partial updater obtains an / ?-th gaussian scene representation, shown in FIG. 3D as gaussian scene representation 370F. The partial updater 356 may determine an error metric based on an image 372n, e.g., a view of the scene rendered based on the gaussian scene representation 370F, and the input image 373 and determine to stop partially updating the gaussian scene representation based on the error metric satisfying the threshold error.

[0164] In some embodiments, one or more error metrics may be determined before adjusting parameters of the selected gaussians or performing additional adjustments of the selected gaussians. For example, a first error metric(s) may be determined based on input pixel data and reference pixel data. In this example, the input pixel data may correspond to an image representing a current view of the scene, such as input image 373, and the reference pixel data may correspond to an image of the scene as rendered based on the gaussian scene representation model, such as reference image 372. In some embodiments, the partial updater 356 may not adjust any parameters of the gaussian scene representation if the first error metric determined prior to performing any parameter adjustments satisfies a first threshold error, e.g., within 5% difference of the input pixel data. If the first error metric does not satisfy the first threshold error, the partial updater 356 may adjust parameters of the selected gaussians, e.g., location parameters, orientation parameters, or both, and then render at least one view of the scene using the updated gaussian scene representation. The at least one view of the scene rendered using the updated gaussian scene representation may then be compared with the input pixel data, e.g., the image 373, to determine a second error metric. The partial updater 356 may further adjust the parameters of the selected gaussians based on a comparison of the first error metric and the second error metric.

[0165] To illustrate, the gaussian scene representation 370C may be a baseline gaussian scene representation when the partial update process begins. A view of the baseline gaussian scene representation may be rendered to generate the reference image 372, and an error metric determined based on the reference image 372 and the input image 373. If the partial updater 356 determines to update the gaussian scene representation 370C based on the error metric, one or more spatial transformations may be applied to theQUALCOMM Ref. No. 2501501WO- 59 / 130 -gaussian scene representation 370C to obtain the gaussian scene representation 370D. A second error metric may be determined based on a view of the scene rendered based on the gaussian scene representation 370D and the input image 373, and the second error metric may be compared to the error metric determined based on input image 373 and the view rendered from the gaussian scene representation 370C, i.e., the reference image 372. If the second error metric obtained from the view rendered from the gaussian scene representation 370D is lower than the error metric obtained from the view rendered from the gaussian scene representation 370C, the partial updater 356 may determine that the gaussian scene representation 370D more accurately represents the state of the scene as shown in the input image 373 and the gaussian scene representation 370D may become the new baseline gaussian scene representation. That is, any further updates to the gaussian scene representation may be made to the gaussian scene representation 370D, rather than to the gaussian scene representation 370C since the error metrics indicate the gaussian scene representation 370D more accurately represents the state of the scene.

[0166] Continuing with this example, the error metric determined based on the image rendered based on the gaussian scene representation 370D may not satisfy the threshold error and the partial updater 356 may determine to make further updates to the gaussian scene representation 370D. Accordingly, the partial updater 356 may apply additional spatial transformations to the gaussian scene representation 370D to obtain the gaussian scene representation 370E. An image may be rendered based on gaussian scene representation 370E and a third error metric may be determined and used by the partial updater 356 to evaluate the state of the scene as represented by the gaussian scene representation 370E. If the error metric associated with the gaussian scene representation 370E is lower than the error metric associated with the view rendered based on the gaussian scene representation 370D, the partial updater 356 may make the gaussian scene representation 370E the new baseline.

[0167] This iterative process may continue until a termination condition is satisfied. The termination condition can include a specified number of iterations having been performed or determination that an error metric for a gaussian scene representation satisfies the threshold error. To illustrate, in FIG. 3D, an error metric associated with an image rendered based on the gaussian scene representation 370F satisfies the thresholdQUALCOMM Ref. No. 2501501WO- 60 / 130 -error. In this example, the gaussian scene representation 370F is the final version and the iterative process stops. If the specified number of iterations are performed without identifying a gaussian scene representation associated with an error metric that satisfies the threshold error, the iterative process may be stopped and a new update strategy can be selected (e.g., by the update strategy selector 144 of FIG. 1).

[0168] In some instances a spatial transformation applied to the baseline gaussian scene representation in a particular iteration may increase the error metric, indicating the last partial update of the gaussian scene representation resulted in a reference image that was less similar to the input image 373. In such instances, the partial updater 356 may not update the baseline gaussian scene representation and may make different spatial transformations in an attempt to obtain a gaussian scene representation that satisfies the threshold error.

[0169] As described above, the partial updater 356 may apply spatial updates to parameters, e.g., location parameters, orientation parameters, or both, of the selected gaussians included in the gaussian scene representation 370C until a / 7thgaussian scene representation 370F is obtained. An image 372n rendered based on the gaussian scene representation 370F may more closely approximate a view or state of the scene at a target point in time, such as a past historical point in time, a current point in time, or a future point in time. As explained above, the view or state of the scene at the target point in time may correspond to the input image 373.

[0170] A technical benefit of performing the partial update 150 rather than the full update 148 in the manner illustrated and described with reference to FIG. 3D is that significant computer resources can be conserved. For example, the partial updater 356 may only need to update the selected gaussians, rather than the entire gaussian scene representation, which enables updating of only those portions of the gaussian scene representation that have changed, e.g., due to movement or alteration of objects in the scene or a change in the point of view of a camera. Performing such partial updates consumes considerably less memory and computational resources as compared to performing a full update of the gaussian scene representation or generating a new gaussian scene representation. Additionally, iteratively applying the spatial transformations enables the partial updater 356 to refine the gaussian sceneQUALCOMM Ref. No. 2501501WO- 61 / 130 -representation to more accurately represent a target state of the scene represented by the gaussian scene representation. The error metrics obtained via the rendering-based error calculator 359 may be used by the partial updater 356 to control the iterative updating of the gaussian scene representation, thereby providing a technique for determining when to stop adjusting the selected gaussians of the gaussian scene representation and / or whether to perform adjustments at all.

[0171] As another non-limiting example, the partial updater 356 may be configured to perform a partial update of a gaussian scene representation based on rendering a set of candidate views based on the gaussian scene representation, where each candidate view of the set of candidate views is associated with a different spatial transformation of selected gaussians of the gaussian scene representation. For example, in FIG. 3E the gaussian scene representation 370C of FIG. 3D is show n. As explained above, the gaussian scene representation 370C may be used to render an image based on a plurality of gaussians of the gaussian scene representation 370C. e.g., images showing a portion of a soccer pitch, as explained above. It is noted that the partial update strategy illustrated in FIG. 3E obtains and evaluates multiple partial updates of the gaussian scene representation in parallel, rather than in series as is performed in the example of FIG. 3D.

[0172] In the example of FIG. 3E, the partial updater 356 may perform a partial update of the gaussian scene representation 370C so that the gaussian scene representation 370C more closely matches the input image 373. To illustrate, in the example of FIG.3E the gaussian scene representation 370C includes gaussians corresponding to the soccer ball 365, which is positioned in the right comer of the goalkeeper box. How ever, the soccer ball 365 may have changed positioned since a last time the gaussian scene representation 370C was updated, e.g., via a full or partial update, as shown in the input image 373 where the soccer ball 365 may be rendered in a position located towards the left edge of the goalkeeper box.

[0173] To perform a partial update of the gaussian scene representation 370C according to the example of FIG. 3E. the partial updater 356 may select one or more gaussians to be updated from among a plurality of gaussians of the gaussian scene representation 370C. As explained above, the gaussians may be selected based on obtained pixel data,QUALCOMM Ref. No. 2501501WO- 62 / 130 -e.g., based on mapping data 122. image subspace selector 312, or other techniques disclosed herein. In some embodiments, the selected gaussians may correspond to portions of the gaussian scene representation 370C that have changed over time, such as movement of the soccer ball 365. The updated gaussian scene representation after the partial update may more accurately represent a state of the scene at a particular point in time, e.g., due to movement of one or more objects in the scene. It is noted that the particular point in time may be a current point in time, a future point in time, or another historic point in time.

[0174] In the example of FIG. 3E, the partial updater 356 may determine a set of parameter adjustment candidates for parameters of the selected gaussians. It is noted that a parameter adjustment candidate of the set of parameter adjustment candidates represents or may be referred to as a candidate. In some embodiments, the partial updater 356 may determine the set of candidate parameter adjustments using dithering based on one or more distribution parameters associated with the selected gaussians. To illustrate, the partial updater 356 may introduce noise into each parameter adjustment candidate. As a result of the noise, each parameter adjustment candidate applies a different spatial transformation, e.g., different location, orientation, and the like, to the selected gaussians. which increases the likelihood that at least one of the parameter adjustment candidates will reduce and / or minimize an error metric.

[0175] In some embodiments, the set of parameter adjustment candidates may include adjustments to a plurality of parameters corresponding to the selected gaussians. For example, the set of parameter adjustment candidates may include parameter adjustments 374A, 374B, 374C, 374D, as shown in FIG. 3E. In some embodiments, the parameter adjustments 374A, 374B, 374C, 374D may apply different spatial transformations to location parameters of the selected gaussians. To illustrate, the parameter adjustment 374A may apply a first spatial transformation to the location parameters of the selected gaussians, the parameter adjustment 374B may apply a second spatial transformation to the location parameters of the selected gaussians, the parameter adjustment 374C may apply a third spatial transformation to the location parameters of the selected gaussians, and the parameter adjustment 374D may apply a fourth spatial transformation to the location parameters of the selected gaussians. In some embodiments, the parameter adjustments 374A, 374B, 374C, 374D may apply spatial transformations that includeQUALCOMM Ref. No. 2501501WO- 63 / 130 -different modifications to orientation parameters of the selected gaussians. To illustrate, the parameter adjustment 374A may apply a first modification to orientation parameters of the selected gaussians, the parameter adjustment 374B may apply a second modification to orientation parameters of the selected gaussians, the parameter adjustment 374C may apply a third modification to orientation parameters of the selected gaussians. and the parameter adjustment 374D may apply a fourth modification to orientation parameters of the selected gaussians. In some embodiments, the parameter adjustments 374A, 374B, 374C, 374D may be associated with both location and orientation parameter adjustments for the selected gaussians. As additional non-limiting examples, the candidate parameter adjustments can include rotation of the selected gaussians, translation of the selected gaussians, color transformation of at least one of the selected gaussians, modification of an opacity parameter of at least one of the selected gaussians, or a combination thereof.

[0176] The partial updater 356 may update the selected gaussians based on the set of parameter adjustment candidates. In some embodiments, updating the selected gaussians may include generating or obtaining a set of candidate gaussian scene representations. The set of candidate gaussian scene representations may correspond to instances of the gaussian scene representation 370C in which the parameters of the selected gaussians have been modified according to the set of parameter adjustment candidates. For example, each candidate gaussian scene representation may include adjustments to parameters of the selected set of gaussians based on a particular parameter adjustment candidate of the set of parameter adjustment candidates. It is noted that because only a partial update is being performed by the partial updater 356, each of the candidate gaussian scene representation may include gaussians that are not associated with any parameter adjustments, such as gaussians not included in the selected gaussians.

[0177] The partial updater 356 may render a set of candidate views based on the set of candidate gaussian scene representations. For example, in FIG. 3E, a set of candidate gaussian scene representations is shown to include candidate view 376a, 376b, 376c, 376M. As can be seen in FIG. 3E, the soccer ball 365 is shown in a different location in each of the set of candidate views 376a, 376b. 376c, 376n. The different location in each view may be based on the different adjustments associated with a corresponding candidate gaussian scene representation. As a non-limiting example, the spatialQUALCOMM Ref. No. 2501501WO- 64 / 130 -transformation related to parameter adjustment 374A may be associated with the candidate view 376n (as indicated by the arrow shown in association with parameter adjustment 374A in the gaussian scene representation 370C of FIG. 3E), the spatial transformation related to parameter adjustment 374B may be associated with the candidate view 376c (as indicated by the arrow shown in association with parameter adjustment 374B in the gaussian scene representation 370C of FIG. 3E), the spatial transformation related to parameter adjustment 374C may be associated with the candidate view 376b (as indicated by the arrow shown in association with parameter adjustment 374C in the gaussian scene representation 370C of FIG. 3E), and the spatial transformation related to parameter adjustment 374D may be associated with the candidate view 376a (as indicated by the arrow shown in association with the parameter adjustment 374D in the gaussian scene representation 370C of FIG. 3E).

[0178] The set of candidate views may be compared to the input image 373 to determine which candidate view is most similar to the input image 373. In some embodiments, the comparison of the candidate view s to the input image 373 may correspond to a difference between each candidate view and the input image 373, such as a pixel-by -pixel difference between the candidate view and the input image 373. In some embodiments, a set of error metric values may be determined based on the set of candidate views. For example, the error metrics 378 may be determined based on the input pixel data, e.g., the image 373, and a set of rendered candidate view s. As explained above, each candidate view of the set of rendered candidate views corresponds to a candidate partial update of the gaussian scene representation based on a respective parameter adjustment candidate of the set of parameter adjustment candidates.

[0179] In some embodiments, the partial updater 356 may update the selected gaussians of the gaussian scene representation 370C based on the error metric values. To illustrate, in the example of FIG. 3E, the error metric value associated with the candidate view 376a may indicate that the candidate view 376a has the smallest difference relative to the input image 373, e.g., because the soccer ball 365 in the candidate view 376a is closest to the position of the soccer ball 365 in the input image 373. The adjustment of the gaussian scene representation 370C may then be updated based on the parameterQUALCOMM Ref. No. 2501501WO- 65 / 130 -adjustments corresponding to the candidate view 370a to produce an updated gaussian scene representation.

[0180] In some embodiments, the partial updater 356 may update the selected gaussians based on a particular candidate partial update of the gaussian scene representation associated with a smallest value among the set of error metric values. As explained above, because dithering is used to generate the parameter adjustment candidates used to generate the rendered candidate views, e.g., 376a-376n, at least one of the parameter adjustment candidates may minimize the error metric as compared to others, ensuring the gaussian scene representation more closely represents the scene depicted in the input image 373 during each update. For example, the candidate view 376a may be associated with the smallest error metric value and the parameter adjustments associated with candidate view 376a may be applied to the gaussian scene representation 370C to produce an updated gaussian scene representation.

[0181] In some embodiments, the partial updater 356 may determine whether to further update the gaussian scene representation 370C after updating the selected gaussians. As anon-limiting example, the partial updater 356 may determine whether to further update the selected gaussians of the gaussian scene representation 370C based, at least in part, on whether a termination condition is satisfied. For example, the termination condition may be based on a threshold difference between the candidate views and the input image 373 and the partial updater 356 may stop updating the gaussian scene representation 370C when a difference between one of the candidate views and the input image 373 satisfies the threshold difference, e.g., the difference is below 1%, 3%, 5%, or another difference value. It is noted that the threshold difference may be specified as a value other than a percentage if desired and that percentage-based threshold differences have been described by way of illustration, rather than by way of limitation. As another non-limiting example, the termination condition may be based on the error metrics 378, and the partial updater 356 may stop updating the gaussian scene representation 370C when one or more values of the set of error metric values satisfy a termination condition. For example, the termination condition may correspond to an error tolerance and updating the gaussian scene representation 370C may be terminated when the error metric(s) satisfy the error tolerance, e.g., the error metrics are below 1% error, 3% error, 5% error, or another error tolerance value. It is noted that the errorQUALCOMM Ref. No. 2501501WO- 66 / 130 -tolerance may be specified as a value other than a percentage if desired and that percentage-based error tolerances have been described by way of illustration, rather than by way of limitation.

[0182] In some embodiments, more than one termination condition can be specified, and updating of the gaussian scene representation 370C can be terminated when any of the termination conditions are satisfied. For example, in some cases, one of the termination conditions can be based on a number of candidates that have been generated, a number of iterations, or another limit on computing capacity used for the partial update. In these cases, if no acceptable partial update has been identified when such a termination condition is satisfied, the selective model update pipeline 140 of FIG. 1 may cause the update strategy selector 144 to select a different update strategy (e.g., a different partial update strategy, no update, or a full update).

[0183] As can be appreciated from the foregoing, the partial updater 356 may perform partial updates of the gaussian scene representation 370C and determine whether to perform further partial updates, e.g., by determining additional sets of parameter adjustment candidates, based on whether the termination condition is satisfied. In some embodiments, the partial updater 356 may be configured to determine the additional sets of parameter adjustment candidates based on the parameter adjustments and one or more metrics, e.g., the difference and / or error metrics, associated with each of the candidate views. For example, in FIG. 3E the parameter adjustment 374B may be associated with the candidate view 376c and the parameter adjustment 374D may be associated with the candidate view 376a. The partial updater 356 may determine the spatial transformation related to the parameter adjustment 374D resulted in the best performance, e.g., based on the candidate view 376a having the smallest difference metric and / or lowest error metric. Based on determining the parameter adjustment 374D resulted in the best performance, the partial updater 356 may determine a new set of candidate parameter adjustments that are similar to parameter adjustment 374D, such as spatial transformations configured to move the soccer ball 365 further in the same direction as the parameter adjustment 374D but further left in the Y direction. It is noted that other types of parameter adjustments besides location parameters of the selected gaussians may also be determined for each new set of candidate parameter adjustments in accordance with the concepts described herein.QUALCOMM Ref. No. 2501501WO- 67 / 130 -

[0184] A technical benefit of performing the partial update 150 rather than the full update 148 in the manner illustrated and described with reference to FIG. 3E is that significant computer resources can be conserved. For example, the partial updater 356 may only need to update the selected gaussians, rather than the entire gaussian scene representation, which enables updating of only those portions of the gaussian scene representation that have changed, e.g.. due to movement or alteration of objects in the scene or a change in the point of view of a camera. Performing such partial updates consumes considerably less memory and computational resources as compared to performing a full update of the gaussian scene representation or generating a new gaussian scene representation. Additionally, iteratively determining the set of candidate parameter adjustments enables the partial updater 356 to refine the gaussian scene representation to more accurately represent a target state of the scene represented by the gaussian scene representation. The difference and / or error metrics obtained via the rendering-based error calculator 359 may be used by the partial updater 356 to control the iterative updating of the gaussian scene representation, thereby providing a technique for determining when to stop adjusting the selected gaussians of the gaussian scene representation and / or whether to perform adjustments at all. Additionally, the technique illustrated and described with reference to FIG. 3E saves memory' resources, which may be used to store other data (e.g. gradient data, image data) and reduces the computational cost while enabling the partial update to be performed quickly, e.g. because gradient data does not need to be calculated and simple comparison, such as a pixel comparison, may be used.

[0185] As another non-limiting example, the partial updater 356 may be configured to perform a partial update of a gaussian scene representation based on two or more prior images of a scene corresponding to the gaussian scene representation. For example, in FIG. 3F the gaussian scene representation 370C of FIG. 3D is shown. As explained above, the gaussian scene representation 370C may be used to render an image based on a plurality of gaussians of the gaussian scene representation 370C, e g., images showing a portion of a soccer pitch, as explained above.

[0186] In the example of FIG. 3F, the partial updater 356 may perform a partial update of the gaussian scene representation 370C so that the gaussian scene representation 370C more closely matches the input image 373. To illustrate, in the example of FIG.QUALCOMM Ref. No. 2501501WO- 68 / 130 - 3F the gaussian scene representation 370C includes gaussians corresponding to the soccer ball 365, which is positioned in the right comer of the goalkeeper box. However, the soccer ball 365 may have changed positioned since a last time the gaussian scene representation 370C was updated, e.g., via a full or partial update, as shown in the input image 373 where the soccer ball 365 is show n as located towards the left edge of the goalkeeper box.

[0187] To perform a partial update of the gaussian scene representation 370C according to the example of FIG. 3F, the partial updater 356 may select one or more gaussians to be updated from among a plurality of gaussians of the gaussian scene representation 370C. As explained above, the gaussians may be selected based on obtained pixel data, e.g., based on mapping data 122, image subspace selector 312, or other techniques disclosed herein. In some embodiments, the selected gaussians may correspond to portions of the gaussian scene representation 370C that have changed over time, such as movement of the soccer ball 365. The updated gaussian scene representation after the partial update may more accurately represent a state of the scene at a particular point in time, e.g., due to movement of one or more objects in the scene, such as the state of the scene shown in the input image 373. It is noted that the particular point in time may be a current point in time, a future point in time, or another historic point in time.

[0188] In the example of FIG. 3E, the partial updater 356 may obtain motion data 384 based on n prior images of the scene, where n is an integer greater than or equal to 2. In some embodiments, the partial updater 356 may obtain the motion data 384 from the motion calculator 382, as described above. In some embodiments, the motion data 384 may be obtained from a source other than the motion calculator 382, such as the modem 114 of FIG. 1 or another source. In some embodiments, the motion data 384 indicates a direction (and optionally a magnitude) of movement associated with an object in the scene. The selected gaussians may be associated with the object, as in the soccer ball 365 in the example of FIG. 3F. As anon-limiting example, the motion data 384 is determined based on the input image 373 and the images 380a - 380 / 7 of FIG. 3F. For example, the soccer ball 365 appears to be moving in both the X and Y directions in the images 380a - 380 / 7. The partial updater 356 may adjust parameters of the selected gaussians of the gaussian scene representation 370C based on the motion data. In some embodiments, the parameter adjustments determined by the partial updater 356 based onQUALCOMM Ref. No. 2501501WO- 69 / 130 -the motion data 384 may adjust one or more location parameters of the selected gaussians to spatially transform the selected gaussians based on the direction of movement. In some embodiments, the motion data 384 indicate a speed of movement associated with the object. The partial updater 356 may be configured to adjust the one or more location parameters by an amount that is based on the speed of movement. In some embodiments, the motion data 384 may indicate rotation associated with an object in the scene. The partial updater 356 may be configured to adjust one or more orientation parameters of the selected gaussians to spatially transform the selected gaussians based on the rotation. In some embodiments, other parameter adjustments may be performed by the partial updater 356, such as to adjust opacity parameters, color parameters, and the like of the selected gaussians.

[0189] In some embodiments, the partial updater 356 is configured to iteratively refine the updated gaussian scene representation 370C based on the motion data 384 and the input image 373. For example, after performing an initial adjustment of the gaussian scene representation 370C the partial updater 356 may render an image based on the updated gaussian scene representation 370C, and the error calculator 359 may utilize the rendered image and the input image 373 to determine one or more error metrics. In some embodiments, the partial updater 356 may determine whether to further adjust parameters of the updated gaussian scene representation based on the one or more error metrics. When the partial updater 356 determines to further adjust parameters of the updated gaussian scene representation, the partial updater 356 may make additional adjustments to the parameters of the selected gaussians to refine the updated gaussian scene representation. The additional adjustments can be determined based on the motion data 384, the one or more error metrics, or both and applied to the parameters of the selected gaussians to update the gaussian scene representation. This process may continue iteratively until a termination criterion is satisfied, which may be a threshold number of iterations or a threshold error. For example, in FIG. 3F, the updated gaussian scene representation 370G shows the soccer ball 365 has been updated from a location 386 in the gaussian scene representation 370C to a location 386’ in the gaussian scene representation 370G, w hich more closely approximates a state of the scene depicted in the input image 373. If the partial updater 356 determines an error metric(s) associatedQUALCOMM Ref. No. 2501501WO- 70 / 130 -with the gaussian scene representation 370G satisfies the termination criterion, no further updates of the gaussian scene representation 370G are made.

[0190] A technical benefit of performing the partial update 150 rather than the full update 148 in the manner illustrated and described with reference to FIG. 3F is that significant computer resources can be conserved. For example, the partial updater 356 may only need to update the selected gaussians, rather than the entire gaussian scene representation, which enables updating of only those portions of the gaussian scene representation that have changed, e.g., due to movement or alteration of objects in the scene or a change in the point of view of a camera. Additionally, the partial adjustment of parameters of the selected gaussians based on the motion data 384 may reduce a number of iterations required to obtain a satisfactory state of the gaussian scene representation, e.g., because the motion data 384 may be used to calculate a distance, direction, and other spatial transformation parameters. Performing such partial updates consumes considerably less memory and computational resources as compared to performing a full update of the gaussian scene representation or generating a new gaussian scene representation. In some embodiments, the partial updater 356 may also iteratively refine the gaussian scene representation based on the motion data 384 and difference and / or error metrics may be used by the partial updater 356 to control a number of iterations, thereby providing a technique for determining when to stop adjusting the selected gaussians of the gaussian scene representation and / or whether to perform adjustments at all.

[0191] It is noted that while FIGs. 3D-3F each describe different ways in which the partial updater 356 may perform partial updates of a gaussian scene representation, in some embodiments the partial update techniques of FIGs. 3D-3F may be used in combination. For example, the motion data 384 described with reference to FIG. 3F may be utilized by the partial updater 356 to calibrate or configure the parameter adjustments described with reference to FIGs. 3D and 3E. Utilizing the motion data 384 in this manner may reduce the number of iterations performed by the partial updater 356 by eliminating parameter adjustments that would result in poor results. To illustrate, having knowledge of the direction the soccer ball 365 is travelling may enable the partial updater 356 to not generate candidate views based on the parameter adjustments 374A-374C, which spatially transform the location of the soccer ball 365 in directionsQUALCOMM Ref. No. 2501501WO- 71 / 130 -different than may be indicated by the motion data 384. A technical benefit of utilizing the motion data in combination with the techniques of FIGs. 3D and 3E is that fewer parameter adjustments may be made by the partial updater 356 in order to achieve an updated gaussian scene representation that more closely resembles or approximates a target state of the scene, which may reduce the computational resources and memory requirements needed to perform a partial update while also enabling partial updates to be performed more quickly, e.g., due to fewer iterations being required.

[0192] FIG. 4 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure. FIG. 4 illustrates an example of the gaussian scene representation 154, the gaussian selector 354, the update data selector 224, the mapper 222 and the mapping data 122. In FIG. 4, the mapper 222 includes the mapping data generator 270, and the mapping data 122 includes the first mapping data 272, the second mapping data 274, and the Nth mapping data 276. FIG. 4 also illustrates an image 430 based on reference pixel data (e.g., one of the reference images 124 of FIG. 1), and an input image 420 based on input image data (e.g., an image from the image data 108 of FIG. 1).

[0193] In FIG. 4, the gaussian scene representation 154 includes a plurality of gaussians, which includes an illustrative gaussian 402. The gaussian 402 corresponds to one of the gaussians 156 of FIG. 1. The gaussian 402 is described by a set of parameters, including for example, location parameters indicating a location of a center 404 of the gaussian 402 in scene space as well as other parameters, as described above. In the example illustrated in FIG. 4, the gaussian scene representation 154 represents a scene that includes at least portion of a soccer pitch, including a goal 421, a net 423, markings 422, and a ball 424, as annotated in the input image 420. For example, gaussians 410 of the gaussian scene representation 154 represent the ball 424, gaussians 412 of the gaussian scene representation 154 represent the goal 421, gaussians 413 of the gaussian scene representation 154 represent the markings 422, and gaussians 414 of the gaussian scene representation 154 represent the net 423.

[0194] In FIG. 4, for simplicity of illustration, the input image 420 and the reference image 430 are identical other than the location of the ball 424. Thus, the data analyzer 142 of FIGS. 1-2 has identified pixels representing the ball 424 as difference pixels of aQUALCOMM Ref. No. 2501501WO- 72 / 130 -region of interest 465 in the input image 420. As explained above, the data analyzer 142 can identify the region of interest 465 based on pixel-by-pixel comparison, object detection, other image comparison operations (e.g., optical flow), or a combination thereof.

[0195] In a particular aspect, when the update strategy selector 144 of FIGS. 1-3A determines to perform the partial update 150, the update data selector 224 requests the mapper 222 to identify a subset of the gaussians of the gaussian scene representation 154 to be updated to generate updated parameters. To illustrate, in the example of FIG.4, the update data selector 224 requests the mapper 222 to identify gaussians associated with the region of interest 465. For example, the update data selector 224 can send, to the mapper 222, a query identifying the region of interest 465. The query can identify the region of interest 465 by indicating a spatial location in the scene, a position in the input image 420 along with pose information associated with a camera that captured the input image 420. an identifier of an object associated with the region of interest 465 (e.g., the ball 424), etc.

[0196] The mapper 222 can search the mapping data 122 to identify gaussians associated with the region of interest 465. In a particular aspect, the mapping data 122 is grouped into multiple sets of mapping data, such as the first mapping data 272, the second mapping data 274, and the Nth mapping data 276. The various sets of mapping data are grouped based on how recently or how frequently the mapping data are used. For example, the first mapping data 272 can include mapping data that is most recently or most frequently used, the second mapping data 274 can include mapping data that is less recently or less frequently used, and the Nth mapping data 276 can include even less recently or even less frequently used mapping data. This staged approach to storing the mapping data 122 can reduce the time and / or computing resources used to identify the gaussians associated with the region of interest 465. To illustrate, in the example illustrated in FIG. 4, the ball 424 is likely to move more frequently than the net 423; accordingly, the first mapping data 272 can include mapping data associated with the ball 424, and the second mapping data 274 can include mapping data associated with the net 423. Continuing this example, the goal 421 and the markings 422 may be permanently installed and unlikely to move; however, the markings 422 may wear dow n overtime and need to be refreshed. Accordingly, the Nth mapping data 276 can includeQUALCOMM Ref. No. 2501501WO- 73 / 130 -mapping data associated with the markings 422, the goal 421. both, or neither. For example, since the goal 421 in this illustrative example is substantially unchanging, the mapping data 122 may not include mapping data associated with the goal 421. In this example, in the unlikely event that the goal 421 changes, the mapper 222 can use a mapping data generator 270 to generate mapping data associated with the goal 421.

[0197] The mapping data 122 can include image-to-model mapping data, model -to-image mapping data, or both. Image-to-model mapping data associates features or regions of images (e.g., rendered views) with the gaussians 156 of the gaussian scene representation 154 that represent the features or regions. The image-to-model mapping data can include object-based mapping data, pixel-based mapping data, or both. Model-to-image mapping data associates gaussians 156 of the gaussian scene representation 154 with features or regions they represent in various views. The mapper 222 can use the image-to-model mapping data to identify the gaussians associated with the region of interest 465 and / or use model-to-image mapping data to identify which pixels are affected by particular gaussians.

[0198] Object-based mapping data associates particular objects of the scene, some of which are depicted in an image (e.g., the input image 420), with respective gaussians of the gaussian scene representation 154. For example, in FIG. 4, the ball 424 is depicted in the input image 420, and the gaussian scene representation 154 includes a set of gaussians 410 that represents the ball 424 in the scene. In this example, object-based mapping data can include an object identifier, such as “ball.” and identifiers of the set of gaussians 410. Similarly, the object-based mapping data can include other object identifiers (e.g., object identifiers for the goal 421, the net 423, the markings 422, etc.) and corresponding gaussians. When the mapping data 122 includes object-based mapping data 122, the data analyzer 142 of FIGS. 1-2 or the mapping data generator 270 can include an object detector that uses computer vision techniques to identify objects depicted in the image data 108.

[0199] Pixel-based mapping data associates particular sets of pixels of an image for a particular view (e.g.. the reference image 430) with a set of the gaussians of the gaussian scene representation 154 that influence rendering of the particular set of pixels. For example, since the reference image 430 represents a portion of the sceneQUALCOMM Ref. No. 2501501WO- 74 / 130 -represented by the gaussian scene representation 154, extrinsic and intrinsic data associated with the camera can be used to correlate the reference image 430 with a view of the scene in the gaussian scene representation 154. The extrinsic data associated with the camera relates to the camera’s position in space (e.g., camera pose, such as location and orientation), and the intrinsic data associated with the camera relates to the camera’s photographic properties, such as focal length, principal point, skew, and scaling factors, etc. As one example, for a camera with fixed extrinsic and intrinsic properties, such as a stationary security camera, each pixel of an image (e.g., the input image 420) captured by the camera can be mapped to a specific location in the scene, which can be associated with a set of gaussians to generate pixel-based mapping data. In this example, when an object in the scene (e.g., the ball 424) moves, pixels associated with the object change, and the pixel-based mapping data of the mapping data 122 can be used to identify a subset of the gaussians to be updated. The subset of gaussians are identified based on which pixels differ between the input image 420 and the reference image 430.

[0200] Optionally, the mapping data 122 can also include model-to-image mapping data that associates gaussians of the gaussian scene representation 154 with pixels of a rendered image of the gaussian scene representation 154 that are influenced by the gaussians. For example, the ball 424 of the input image 420 is represented in the gaussian scene representation 154 by the gaussians 410, each of which is distributed in scene space. To illustrate, parameters specifying the gaussian 402 indicate the location of the center 404 of the gaussian 402 and include parameters (e.g., scale and rotation parameters) that describe the distribution of the gaussian 402 about the center 404. In FIG. 4, for example, the gaussian 402 can influence rendering of pixels for rays passing through the center 404, an end 406, an end 408, or any point therebetween. Thus, the gaussian 402 can influence rendering of more than one pixel due to the spatial distribution of the gaussian 402. As one example, in the input image 420, the ball 424 is represented by a set of pixels within a region of interest 465 (shown slightly larger than the ball 424 in FIG. 4, though the region of interest 465 may be defined by edges of the ball 424). In this illustrative example, rendering of a pixel corresponding to the region of interest 465 may be influenced due to a ray passing through or near the end 408 of the gaussian 402. Moreover, changes to the gaussian 402 can influence rendering of pixels corresponding to an area outside (e.g., a surrounding area of) the region ofQUALCOMM Ref. No. 2501501WO- 75 / 130 -interest 465. For example, changing parameters of the gaussian 402 can influence rendering of a pixel associated with a ray passing through the end 406 of the gaussian 402. The mapper 222 can use the model-to-image mapping data to identify the pixels in a rendered image that correspond to a second region 467 of the input image 420. The pixels of the second region 467 include the pixels of the region of interest 465 and one or more additional pixels; the gaussians that influence rendering of first pixels corresponding to the region of interest 465 also influence rendering of second pixels corresponding to the one or more additional pixels. The model-to-image mapping data indicates that the gaussians influence rendering of the first pixels and second pixels of the rendered image.

[0201] In a particular aspect, the mapper 222 is configured to generate the mapping data 122 and store the mapping data 122 in a staged arrangement. For example, when a new mapping data entry is generated (e.g., during initial population of the mapping data 122 or when the mapping data generator 270 is invoked because mapping data associated with a particular object or a particular set of pixels is not found in the mapping data 122), the mapper 222 can cause the mapping data generator 270 to generate a new mapping data entry. The mapper 222 can provide the new mapping data entry to the update data selector 224 for use in updating the gaussian scene representation 154 and can also store the new mapping data entry in the first mapping data 272, the second mapping data 274, or the Nth mapping data 276. To illustrate, the mapper 222 can control where a particular mapping data entry is stored based on recency information. For example, the recency information can indicate how recently the gaussians associated with the mapping data entry have been updated, how recently the mapping data entry has been updated, how recently the mapping data entry has been used to update the gaussian scene representation 154, or a combination thereof. In this illustrative example, when the gaussians associated with a mapping data entry are updated, the mapping data entry is moved to the first mapping data 272. The mapper 222 can apply an eviction policy to the first mapping data 272 to remove mapping data entries that have not been updated within a particular period or to remove a least recently updated mapping data entry w hen a new mapping data entry is added to the first mapping data 272. Thus, in this illustrative example, the first mapping data 272 also includes a set of the most recently updated mapping data 272. The mapper 222 canQUALCOMM Ref. No. 2501501WO- 76 / 130 -add mapping data entries evicted from the first mapping data 272 to the second mapping data 274. The mapper 222 applies another eviction policy to the second mapping data 274 to remove mapping data entries from the second mapping data 274 to the Nth mapping data 276, and so forth.

[0202] In some embodiments, the mapper 222 adds mapping data entries to particular stages of the mapping data 122 (e g., to the first mapping data 272, the second mapping data 274, the Nth mapping data 276, etc.) based on frequency information, or based on a weighted combination of recency and frequency. The frequency information can indicate how frequently the gaussians associated with the mapping data entry have been updated, how frequently the mapping data entry has been updated, how frequently the mapping data entry has been used to update the gaussian scene representation 154, or a combination thereof. In such embodiments, eviction policies applied for each stage of the mapping data 122 can be based on recency, frequency, or a weighted combination thereof.

[0203] Although the mapping data 122 of FIG. 4 are illustrated as arranged in at least three stages, including the first mapping data 272, the second mapping data 274, and the Nth mapping data 276. in some embodiments, the mapping data 122 can be arranged in fewer than three stages (e g., the first mapping data 272 and the second mapping data 274), in which case the Nth mapping data 276 is omitted.

[0204] During operation, the update data selector 224 can provide the mapper 222 with a query identifying the region of interest 465 or an object of interest (e g., the ball 424). The mapper 222 uses a staged search to search the mapping data 122 based on the query. For example, the mapper 222 first searches the first mapping data 272 based on the query since the first mapping data 272 includes information for the most recently or most frequently updated parameters of the gaussian scene representation 154. If the first mapping data 272 includes information responsive to the query, the mapper 222 returns the information to the update data selector 224. Optionally, the mapper 222 can also update data indicating the frequency or recency of use of the information responsive to the query. For example, the mapper 222 can rearrange one or more pointers of the first mapping data 272 to indicate that the information responsive to the query is the most recently updated information in the first mapping data 272. As another example, the firstQUALCOMM Ref. No. 2501501WO- 77 / 130 -mapping data 272 can include or be associated with metadata that includes the information indicating when particular portions of the first mapping data 272 was used to update the gaussian scene representation 154 or how frequently the particular portions of the first mapping data 272 have been used to update the gaussian scene representation 154.

[0205] If the information responsive to the query is not identified in the first mapping data 272, the mapper 222 searches the second mapping data 274 based on the query. If the second mapping data 274 includes information responsive to the query, the mapper 222 returns the information to the data analyzer 142 or the update data selector 224 and updates the mapping data 122. In this situation, updating the mapping data 122 can include updating the second mapping data 274 to indicate when the information was used to update the gaussian scene representation 154 or how frequently the information has been used to update the gaussian scene representation 154. Alternatively, updating the mapping data 122 in this situation can include moving the information from the second mapping data 274 to the first mapping data 272. For example, if the mapper 222 arranges the mapping data 122 in stages based on how recently particular mapping data was used to update the gaussian scene representation 154, then the information responsive to the query is moved from the second mapping data 274 to the first mapping data 272. However, if the mapper 222 arranges the mapping data 122 in stages based on how frequently particular mapping data is used to update the gaussian scene representation 154, then the mapper 222 can modify the update frequency associated with the information, and determine whether the information should remain in the second mapping data 274 or be moved to the first mapping data 272 based on the new update frequency associated with the information.

[0206] The mapper 222 continues operation in this manner for each additional stage of the mapping data 122 if any are present. If no stage of the mapping data 122 includes information responsive to the query, the mapper 222 invokes the mapping data generator 270 to generate the information, and the information is optionally added to a particular stage of the mapping data 122 based on recency or frequency of use of the information to update the gaussian scene representation 154.QUALCOMM Ref. No. 2501501WO- 78 / 130 -

[0207] A technical advantage of the model update operations described with reference to FIG. 4 is that they enable updating the gaussian scene representation 154 in a manner that is efficient in terms of computing resources utilized. For example, arranging the mapping data 122 as described enables the mapper 222 to access the mapping data based on recency and / or frequency of use, which reduces computing resources needed to access desired mapping data in most cases.

[0208] FIG. 5 is a flow chart illustrating particular aspects of a method 500 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 500 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the selective model update pipeline 140.

[0209] In particular, the method 500 illustrates aspects of operations associated with a process for selecting an update mode for performing a partial update of a gaussian scene representation. An '‘update mode” in this context refers to a particular type of partial update strategy, which may be selected by the update strategy selector 144 of FIG. 1. The method 500 includes obtaining input data 502. The input data 502 may include, for example, input pixel data, motion data, mapping data, information identifying pixels of a region of interest, data identifying one or more objects detected in an input image, or a combination thereof.

[0210] The method 500 includes, at block 510, selecting gaussians from among a plurality of gaussians of a gaussian scene representation. As explained above, the gaussians may be selected based on mapping data, input from a user indicating one or more regions of interest within a scene associated with the gaussian scene representation, or other techniques.

[0211] At block 520, the method 500 includes selecting an update mode. As explained above, systems according to some embodiments of the present disclosure may perform full or partial updates of gaussian scene representations. In the example of FIG. 5, a partial update strategy is selected, and selecting the update mode includes selecting a particular type of partial update to be used. For example, each of the techniques described above with reference to FIGs. 3D-3E can correspond to an update mode for a partial update strategy. At block 530, a partial update of selected gaussians may beQUALCOMM Ref. No. 2501501WO- 79 / 130 -performed in accordance with the concepts described and illustrated with reference to FIG. 3D. For example, the gaussian scene representation may be partially updated by spatially transforming the selected gaussians, while other gaussians of the gaussian scene representation are held constant or unchanged. A view 532 of the scene may be rendered based on the gaussian scene representation after the spatial transformation of the selected gaussians is applied.

[0212] At block 540, a partial update of selected gaussians may be performed in accordance with the concepts described and illustrated with reference to FIG. 3E. For example, the gaussian scene representation may be partially updated by determining a set of parameter adjustment candidates for parameters of the selected gaussians and updating the selected gaussians based on the set of parameter adjustment candidates. As explained above, each candidate of the set of parameter adjustment candidates may spatially transform at least a portion of the selected gaussians. while other gaussians of the gaussian scene representation are held constant or unchanged. A set of candidate views 542 may be rendered based on a state of the gaussian scene representation, as modified by each of the set of parameter adjustment candidates.

[0213] At block 550. a motion-based partial update of the gaussian scene representation may be performed in accordance with the concepts described and illustrated with reference to FIG. 3F. For example, when the input data 502 includes motion data, parameters of the selected gaussians may be adjusted to spatially transform the selected gaussians based on the motion data. A view 552 of the scene may be rendered based on the gaussian scene representation after the spatial transformation of the selected gaussians is applied.

[0214] At block 560. the method 500 includes validating an output of the selected update mode determined in block 520. To illustrate, when the selected update mode is the update mode associated with block 530, the view 532 may be compared to the input pixel data and one or more metrics determined based on the comparing. Similarly, when the selected update mode is the update mode associated with block 540, the one or more metrics may be determined based on the set of candidate views 542, and when the selected update mode is the update mode associated with block 550, the one or more metrics may be determined based on the view 552. As explained above, the metrics mayQUALCOMM Ref. No. 2501501WO- 80 / 130 -be pixel-by -pixel difference metrics, an error metric, or other metrics that may be used to validate a state of the scene, as expressed in one or more rendered views, e.g., the view 532, the candidate views 542, or the view 552, relative to the input data 502. As explained above, if the state of the scene, as quantified by the one or more metrics, satisfies a threshold, the gaussian scene representation, as modified by one of blocks 530, 540. 550 may be output as an updated gaussian scene representation 562.

[0215] However, if the state of the scene, as quantified by the one or more metrics, does not satisfy the threshold, processing of the method 500 may return to the selected update mode. For example, when the selected update mode is associated with the block 530. feedback 534 may be provided to the selected update mode and used to refine the spatial transformations, as described above with reference to FIG. 3D. When the selected update mode is associated with the block 540, feedback 544 may be provided to the selected update mode and used to refine the set of candidate parameter adjustments used to apply spatial transformations to the gaussian scene representation, as described above with reference to FIG. 3E. When the selected update mode is associated with the block 550, feedback 554 may be provided to the selected update mode and used to refine the motion data-based adjustments to parameters of the selected gaussians, as described above with reference to FIG. 3F.

[0216] In some embodiments, selection of the update mode may be based on the input data 502. For example, if the input data does not contain motion data, the update mode associated with block 550 may not be selected. As another example, if the input data does not contain prior image data, the update mode associated with block 540 may not be selected. In some embodiments, selection of the update mode may be based on information other than the input data. For example, selection of the update mode may be based on a state of the device, such as based on available computational resources, e.g., memory, computational resources of a processor, and the like. When available computational resources are low, selection of the update mode at block 520 may be the partial update mode associated with block 530 or block 550, e.g., because the update mode associated with block 540 requires rendering candidate views which may consume more computational resources, such as memory, than the update modes associated with blocks 530, 550. It is noted that while the update mode associated with block 540 may consume more computational resources than the update modesQUALCOMM Ref. No. 2501501WO- 81 / 130 -associated with blocks 530, 550, the update mode associated with block 540 consumes less computational resources than performing a full update of the gaussian scene representation. In some embodiments, selection of the update mode may be based on the input data 502 and the state of the device. For example, when available computational resources are low, selection of the update mode at block 520 may select the partial update mode associated with block 530 when the input data does not include motion data and may select the partial update mode associated with block 550 when the input data includes motion data. It is noted other factors and considerations with respect to the contents of the available input data and / or state of the device may be accounted for during update mode selection at block 520 and the various examples described above have been provided by way of illustration, rather than by way of limitation.

[0217] FIG. 6 is a flow chart illustrating particular aspects of a method 600 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. I, in accordance with some examples of the present disclosure. For example, the method 600 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140.

[0218] The method 600 includes, at block 602, obtaining input pixel data. In some embodiments, a region of interest may be associated with the input pixel data. For example, the input pixel data can correspond to, include, or be included within the image data 108 of FIG. 1, the sensor data 303 of FIG. 3A, the input image 373 of FIG.3D. the input image 420 of FIG. 4. or the input data 502 of FIG. 5. The input pixel data can be obtained from the image source(s) 106, the modem 114, or the data analyzer 142 of FIG. 1, the update data selector 224 of FIG. 2 or FIG. 4, or the image subspace selector 312 of FIGS. 3A and 3C. In a particular example, the input pixel data can be obtained periodically or occasionally (e.g., based on a schedule or based on the sensor data) or can be obtained based on user input.

[0219] The method 600 includes, at block 604, selecting based on the input pixel data, gaussians of a gaussian scene representation to be updated. For example, the selected gaussians may be selected by the mapper 222 of FIG. 4 based on mapping data. As one example, the mapping data can correspond to the mapping data 122, such as the first mapping data 272 of FIG. 2.QUALCOMM Ref. No. 2501501WO- 82 / 130 -

[0220] The method 600 includes, at block 606, spatially transforming the selected gaussians of the gaussian scene representation. For example, the partial updater 356 may apply spatial transformations to the selected gaussians to update the gaussian scene representation. In an aspect, the spatial transformations may adjust location parameters, orientation parameters, or both. Optionally, in some cases, other parameters of the selected gaussians may also be adjusted, as explained above with reference to FIG. 3D. In some embodiments, a view may be rendered based on the updated gaussian scene representation, and one or more metrics may be determined based on the view. If the one or more metrics satisfy a threshold, the updated gaussian scene representation may replace the prior version of the gaussian scene representation, e g., before the spatial transformations applied at block 606. If the one or more metrics do not satisfy the threshold, additional spatial transformations may be applied to the gaussian scene representation until a termination criterion is satisfied, as explained above with reference to FIG. 3D.

[0221] FIG. 7 is a flow chart illustrating particular aspects of a method 700 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 700 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140.

[0222] The method 700 includes, at block 702, obtaining input pixel data. For example, the input pixel data can correspond to. include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1. As another example, the input pixel data can correspond to, include, or be included within image data 147 from the image source(s) 106 of FIG. 2. As another example, the input pixel data can correspond to. include, or be included within the sensor data 305, the ROI data 313 of FIG. 3A, the input image 373 of FIG. 3E, the input image 420 of FIG. 4, or the input data 502 of FIG. 5.

[0223] The method 700 includes, at block 704, selecting based on the input pixel data, gaussians of a gaussian scene representation to be updated. For example, the selected gaussians may be selected by the mapper 222 of FIG. 4 based on mapping data. As oneQUALCOMM Ref. No. 2501501WO- 83 / 130 -example, the mapping data can correspond to the mapping data 122, such as the first mapping data 272 of FIG. 2.

[0224] The method 700 includes, at block 706, determining a set of parameter adjustment candidates for parameters of the selected Gaussians. The parameter adjustment candidates may adjust location parameters, orientation parameters, opacity parameters, color parameters, or other parameters of the selected gaussians, as explained above with reference to FIG. 3E. In some embodiments, the set of parameter adjustment candidates may be determined by the partial updater 356 of FIGs. 3C and 3E. In some embodiments, the set of parameter adjustment candidates may be determined based on dithering.

[0225] The method 700 includes, at block 708, updating the selected Gaussians based on the set of parameter adjustment candidates. For example, the partial updater 356 may create instances of the gaussian scene representation and different parameter adjustment candidates may be applied to each instance of the gaussian scene representation. In some embodiments, a set of candidate views, where each candidate view is rendered based on a different instance of the updated gaussian scene representation. One or more metrics may be determined based on the set of candidate views. If the one or more metrics for a particular candidate view satisfy a threshold, the updated gaussian scene representation used to generate the candidate view may replace the prior version of the gaussian scene representation. If the one or more metrics do not satisfy the threshold, additional parameter adjustment candidates may be determined and additional candidate views generated until a termination criterion is satisfied, as explained above with reference to FIG. 3E.

[0226] FIG. 8 is a flow chart illustrating particular aspects of a method 800 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 800 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140.

[0227] The method 800 includes, at block 802, obtaining input pixel data corresponding to an input image depicting a portion of a scene. The input pixel data can correspond to, include, or be included within the image data 108 from the image source(s) 106, theQUALCOMM Ref. No. 2501501WO- 84 / 130 -modem 114, or the buffer(s) 118 of FIG. 1; image data 147 from the image source(s) 106 of FIG. 2; or the sensor data 305 or the ROI data 313 of FIG. 3 A, the input image 373 of FIG. 3F, the input image 420 of FIG. 4, or the input data 502 of FIG. 5.

[0228] At block 804. the method 800 includes obtaining motion data based on two or more prior images of the scene. In some embodiments, the partial updater 356 may obtain the motion data, e.g., the motion data 384 of FIG. 3F, from the motion calculator 382, as described above. In some embodiments, the motion data 384 may be obtained from a source other than the motion calculator 382, such as the modem 114 of FIG. 1 or another source. In some embodiments, the motion data 384 indicates a direction of movement associated with an object in the scene. In some embodiments, the motion data may be determined based on the two or more prior images, such as by tracking movement of objects in the images over time.

[0229] The method 800 includes, at block 806, selecting, based, at least in part, on the input image, gaussians of a gaussian scene representation to be updated. For example, the selected gaussians may be selected by the mapper 222 of FIG. 4 based on mapping data. As one example, the mapping data can correspond to the mapping data 122. such as the first mapping data 272 of FIG. 2.

[0230] The method 800 includes, at block 808, adjusting parameters of the selected gaussians to spatially transform the selected gaussians based on the motion data. As explained above, in some embodiments, the parameter adjustments determined based on the motion data may adjust one or more location parameters of the selected gaussians to spatially transform the selected gaussians based on the direction of movement. In some embodiments, the motion data 384 may indicate a speed of movement associated with the object. In some embodiments, the one or more location parameters may be adjusted by an amount, e.g., a distance, a direction, a height, and the like, that is based on the speed of movement. In some embodiments, the motion data may indicate rotation associated with an object in the scene and one or more orientation parameters of the selected gaussians may be adjusted to spatially transform the selected gaussians based on the rotation. Optionally, in some embodiments, other parameters of the gaussians may also be adjusted, such as opacity parameters, color parameters, and the like.QUALCOMM Ref. No. 2501501WO- 85 / 130 -

[0231] In some embodiments, a view may be rendered based on the updated gaussian scene representation. One or more metrics may be determined based on the rendered view. If the one or more metrics satisfy a threshold, the updated gaussian scene representation used to generate the view may replace the prior version of the gaussian scene representation. If the one or more metrics do not satisfy the threshold, additional parameter adjustments may be determined and additional views generated until a termination criterion is satisfied, as explained above with reference to FIG. 3F.

[0232] Technical advantages of the methods 500, 600, 700, and 800 include enabling updating of gaussian scene representations in a manner that is efficient in terms of computing resources utilized. For example, performing the full update uses significant computing resources and generally uses all of the input pixel data to perform calculations that account for all of the gaussians of the gaussian scene representation. In contrast, the methods 500, 600, 700, and 800 enable selection of an update strategy (e.g., a partial update) that uses fewer calculations under particular circumstances, such as when the differences detected affect only a subset of the gaussians of the gaussian scene representation. Thus, fewer calculations can be used to keep the gaussian scene representation updated based on the input pixel data, resulting in savings of computing resources (e.g.. processor time, power, and memory). Further, when the differences associated with the input pixel data are small, the methods 500, 600, 700, and 800 can decide to perform no update, in which case even the computing resources associated with performing the partial update are conserved.

[0233] FIG. 9 depicts an implementation 900 of the device 102 of FIG. 1 as an integrated circuit 902 that includes the one or more processors 130. The integrated circuit 902 also includes one or more input connectors 906 (e.g., pins, contacts, leads, solder balls, etc.) to enable input data 904 to be received for processing. For example, the input data 904 can correspond to, include, or be included within, the image data 108, the sensor data 112, the data 116, the reference images 124, the mapping data 122, the gaussian scene representation 154, or other data described with reference to FIG. 1; the reference pixel data or other data described with reference to FIG. 2; the sensor data 303, the sensor data 305. the input 307, the ROI data 313. the strategy selection data 320, the mapping data 122, the gaussians 156, the ground-truth image 349 of FIGS. 3A-3C; or a combination thereof.QUALCOMM Ref. No. 2501501WO- 86 / 130 -

[0234] The integrated circuit 902 also includes one or more output connectors 908 (e.g., pins, contacts, leads, solder balls, etc.) to enable output data 910 to be sent to another device or component. For example, the output data 910 can correspond to, include, or be included within, the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the mapping data 122, the view 136, another output of the selective model update pipeline 140. or other data described with reference to FIG. 1; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154 or other data described with reference to FIG. 2; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the selection indication 321, the pixel data 341, the gradient values 343, the updated parameter values 347, or other data described with reference to FIGS. 3A-3C; or a combination thereof.

[0235] The processor(s) 130 of the integrated circuit 902 are configured to initiate, perform, or control various operations described with reference to FIGS. 1-8. For example, in FIG. 9, the processor(s) 130 of the integrated circuit 902 include the selective model update pipeline 140 (or components thereof) and the gaussian scene representation 154 (or components thereof, e.g., a subset of the gaussians 156 of FIG.1). Optionally, the processor(s) 130 can also include the renderer(s) 158, the view selector 162, or both.

[0236] The integrated circuit 902 enables implementation of selective partial gaussian scene representation updating as a component in a system that includes or is coupled to one or more image sources, such as a mobile phone or tablet as described with reference to FIG. 10; a mixed reality or augmented reality glasses device as described with reference to FIG. 11; a wearable electronic device as described with reference to FIG.12; a voice-controlled speaker system described with reference to FIG. 13; a camera described with reference to FIG. 14; a virtual reality, mixed reality, or augmented reality headset described with reference to FIG. 15; or a vehicle described with reference to FIG. 16 or FIG. 17.

[0237] FIG. 10 depicts an implementation 1000 in which the device 102 includes a mobile device 1002, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 1002 includes one or more microphones 1010, one or more speakersQUALCOMM Ref. No. 2501501WO- 87 / 130 - 1006. one or more cameras 1008, and a display screen 1004. Components of the integrated circuit 902, including the processor(s) 130. the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the mobile device 1002 and are illustrated using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1002.

[0238] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1008 of the mobile device 1002 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1008 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as whether the mobile device 1002 is connected to an external power source, a battery state of the mobile device 1002, availability of computing resources onboard the mobile device 1002, etc.

[0239] The mobile device 1002 can use the gaussian scene representation to generate one or more views, which can be displayed to a user via the display screen. For example, the view(s) can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the mobile device 1002 and subsequently sent to another device for rendering to generate one or more views. In some cases, the mobile device 1002 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a remote computing device, such as a server). In some cases, the mobile device 1002 can be configured to perform partial updates or full updates of the gaussian scene representation based on input pixel data received fromQUALCOMM Ref. No. 2501501WO- 88 / 130 -another device, such as augmented reality or mixed reality glasses, a headset device, a body cam, or another wearable electronic device.

[0240] FIG. 11 depicts an implementation 1100 in which the device 102 includes a portable electronic device that corresponds to augmented reality or mixed reality glasses 1102. The glasses 1102 include a holographic projection unit 1112 configured to project visual data onto a surface of a lens 1114 or to reflect the visual data off of a surface of the lens 1114 and onto the wearer’s retina. The glasses 1102 also include one or more cameras 1108, and optionally one or more microphones 1110, one or more speakers 1106. or both. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the glasses 1102.

[0241] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1108 of the glasses 1102 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1108 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the glasses 1102.

[0242] The glasses 1102 can use the gaussian scene representation to generate one or more views, which can be projected by the holographic projection unit 1112 onto the lens 1114. For example, the view can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the glasses 1102 and subsequently sent to another device for rendering to generate one or more views. In some cases, the glasses 1102 can be configured to perform partial updates of theQUALCOMM Ref. No. 2501501WO- 89 / 130 -gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the glasses 1102 or a remote computing device, such as a server).

[0243] FIG. 12 depicts an implementation 1200 in which the device 102 includes a wearable electronic device 1202, illustrated as awearable camera device (sometimes referred to as a “body cam”). A front 1222 and a back 1220 of the wearable electronic device 1202 are shown in FIG. 12.

[0244] In the example illustrated, the wearable electronic device 1202 includes one or more microphones 1210, one or more speakers 1206, and one or more cameras 1208. The wearable electronic device 1202 may also include a body 1224 with an integrated attachment mechanism, such as clip 1226. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the wearable electronic device 1202.

[0245] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1208 of the wearable electronic device 1202 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1208 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or pow er onboard the wearable electronic device 1202.

[0246] The wearable electronic device 1202 can use the gaussian scene representation to facilitate user interaction with the scene. For example, the wearable electronic device 1202 can generate verbal output (e.g., instructions or information about the scene) based on the gaussian scene representation and can provide the verbal output to the user viaQUALCOMM Ref. No. 2501501WO- 90 / 130 -the speaker(s) 1206. In some cases, the gaussian scene representation can be updated at the wearable electronic device 1202 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wearable electronic device 1202 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wearable electronic device 1202 or a remote computing device, such as a server).

[0247] FIG. 13 is an implementation 1300 in which the device 102 includes a wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 can have wireless network connectivity and is configured to execute an assistant operation. In the example illustrated, the wireless speaker and voice activated device 1302 includes one or more microphones 1310, one or more speakers 1306, one or more cameras 1308, and a display 1304. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the wireless speaker and voice activated device 1302.

[0248] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1308 of the wireless speaker and voice activated device 1302 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1308 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the wireless speaker and voice activated device 1302.QUALCOMM Ref. No. 2501501WO- 91 / 130 -

[0249] In a particular aspect, the wireless speaker and voice activated device 1302 can provide assistant operations based at least in part on the gaussian scene representation. For example, the wireless speaker and voice activated device 1302 can respond to verbal queries from the user based on the gaussian scene representation, such as by providing directions to an object or person in the scene. In some cases, the gaussian scene representation can be updated at the wireless speaker and voice activated device 1302 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wireless speaker and voice activated device 1302 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wireless speaker and voice activated device 1302 or a remote computing device, such as a server).

[0250] FIG. 14 depicts an implementation 1400 in which the device 102 includes a portable electronic device that corresponds to a camera device 1402. In the example illustrated, the camera device 1402 includes one or more microphones 1410, one or more speakers 1406, and one or more cameras 1408. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the camera device 1402.

[0251] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1408 of the camera device 1402 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1408 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the camera device 1402.QUALCOMM Ref. No. 2501501WO- 92 / 130 -

[0252] The camera device 1402 can use the gaussian scene representation to facilitate user interaction with the scene, documentation of the scene (e.g., by object or location tagging), etc. In some cases, the gaussian scene representation can be updated at the camera device 1402 and subsequently sent to another device for rendering to generate one or more views. In some cases, the camera device 1402 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the camera device 1402 or a remote computing device, such as a server).

[0253] FIG. 15 depicts an implementation 1500 in which the device 102 includes a portable electronic device that corresponds to a virtual reality, mixed reality, or augmented reality headset 1502. In the example illustrated, the headset 1502 includes one or more microphones 1510, one or more speakers 1506, and one or more cameras 1508. The headset 1502 also includes a display 1504 positioned to be visible to a user when the headset 1502 is worn by the user. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the headset 1502.

[0254] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1508 of the headset 1502 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1508 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the headset 1502.QUALCOMM Ref. No. 2501501WO- 93 / 130 -

[0255] In some cases, the headset 1502 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to generate a virtual reality or augmented reality view for a user. In some cases, the gaussian scene representation can be updated at the headset 1502 and subsequently sent to another device for rendering to generate one or more views. In some cases, the headset 1502 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the headset 1502 or a remote computing device, such as a server).

[0256] FIG. 16 depicts an implementation 1600 in which the device 102 corresponds to, or is integrated within, a vehicle 1602, illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). In the example illustrated, the vehicle 1602 includes one or more microphones 1610, one or more speakers 1606, and one or more cameras 1608. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140. the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1602.

[0257] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1608 of the vehicle 1602 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1608 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select betw een performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the vehicle 1602.

[0258] In some cases, the vehicle 1602 can use the gaussian scene representation (e.g.. the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1602 andQUALCOMM Ref. No. 2501501WO- 94 / 130 -subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1602 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1602 or a remote computing device, such as a server).

[0259] FIG. 17 depicts another implementation 1700 in which the device 102 corresponds to, or is integrated within, a vehicle 1702, illustrated as a car. In the example illustrated, the vehicle 1702 includes one or more displays 1704, one or more microphones 1708. one or more speakers 1706, and one or more cameras 1710. For example, the cameras 1710 can include one or more interior cameras 1710A (which can be positioned to capture interior or exterior images), one or more front cameras 1710B, one or more rear cameras 1710C, one or more side cameras 1710D and 1710E, or a combination thereof. Components of the integrated circuit 902, including the processor(s) 130. the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1702.

[0260] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1710 of the vehicle 1702 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1710 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as availability of computing resources and / or power onboard the vehicle 1702.

[0261] In some cases, the vehicle 1702 can use the gaussian scene representation (e.g, the updated gaussian scene representation) to facilitate navigation or obstacle avoidance.QUALCOMM Ref. No. 2501501WO- 95 / 130 - In some cases, the gaussian scene representation can be updated at the vehicle 1702 and subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1702 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1702 or a remote computing device, such as a server).

[0262] Referring to FIG. 18, a block diagram of a particular illustrative implementation of a device is depicted and generally designated 1800. In various implementations, the device 1800 may have more or fewer components than illustrated in FIG. 18. In an illustrative implementation, the device 1800 may correspond to the device 102. In an illustrative implementation, the device 1800 may perform one or more operations described with reference to FIGS. 1-17.

[0263] In a particular implementation, the device 1800 includes a processor 1806 (e.g., a CPU). The device 1800 may include one or more additional processors 1810 (e.g., one or more DSPs). In a particular aspect, the processor(s) 130 of FIG. 1 corresponds to the processor 1806, the processors 1810, or a combination thereof. The processors 1810 may include a speech and music coder-decoder (CODEC) 1808 that includes the selective model update pipeline 140, the gaussian scene representation 154, a voice coder (“vocoder”) encoder 1836, a vocoder decoder 1838, or a combination thereof.

[0264] The device 1800 may include a memory 1886 and a CODEC 1834. The memory 1886 may include instructions 1856, that are executable by the one or more additional processors 1810 (or the processor 1806) to implement the functionality described with reference to the selective model update pipeline 140, the gaussian scene representation 154, or both. The device 1800 may include a modem 1870 coupled, via a transceiver 1850, to an antenna 1852. In a particular aspect, the modem 1870 corresponds to the modem 114 of FIG. 1.

[0265] The device 1800 may include a display 1828 coupled to a display controller 1826. One or more speakers 1892 and one or more microphones 1894 may be coupled to the CODEC 1834. The CODEC 1834 may include a digital-to-analog converter (DAC) 1802, an analog-to-digital converter (ADC) 1804, or both. In a particular implementation, the CODEC 1834 may receive analog signals from the microphone(s)QUALCOMM Ref. No. 2501501WO- 96 / 130 - 1894. convert the analog signals to digital signals using the analog-to-digital converter 1804. and provide the digital signals to the speech and music codec 1808. In a particular implementation, the speech and music codec 1808 may provide digital signals to the CODEC 1834. The CODEC 1834 may convert the digital signals to analog signals using the digital -to-analog converter 1802 and may provide the analog signals to the one or more speakers 1892.

[0266] In a particular implementation, the device 1800 may be included in a system-in-package or system-on-chip device 1822. In a particular implementation, the memory 1886. the processor 1806, the processors 1810, the display controller 1826, the CODEC 1834, and the modem 1870 are included in the system-in-package or system-on-chip device 1822. In a particular implementation, an input device 1830 (e.g., one or more cameras) and a power supply 1844 are coupled to the system-in-package or the systemon-chip device 1822. Moreover, in a particular implementation, as illustrated in FIG. 18, the display 1828. the input device 1830. the one or more speakers 1892, the microphone(s) 1894, the antenna 1852, and the power supply 1844 are external to the system-in-package or the system-on-chip device 1822. In a particular implementation, each of the display 1828, the input device 1830, the one or more speakers 1892, the microphone(s) 1894. the antenna 1852, and the power supply 1844 may be coupled to a component of the system-in-package or the system-on-chip device 1822, such as an interface, or a controller.

[0267] The device 1800 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a computing device, a communication device, an intemet-of-things (loT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.QUALCOMM Ref. No. 2501501WO- 97 / 130 -

[0268] In conjunction with the described implementations, an apparatus includes means for obtaining input pixel data associated with a region of interest. For example, the means for obtaining input pixel data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106, the modem 114, the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108, the camera(s) 1208, the camera(s) 1308, the camera(s) 1408, the camera(s) 1508, the camera(s) 1608, the camera(s) 1710, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870. one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0269] The apparatus also includes means for determining whether first mapping data includes data associated with the region of interest. For example, the means determining whether the first mapping data includes the data associated with the region of interest can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the update data selector 224, the gaussian selector 354, the integrated circuit 902. the processor 1806, the processor(s) 1810, one or more other circuits or components configured to determine whether mapping data includes data associated with a region of interest, or any combination thereof.

[0270] The apparatus also includes means for obtaining, based on a determination that the first mapping data does not include the data associated with the region of interest, the data associated with the region of interest from second mapping data, where the first mapping data is associated with a first set of gaussians of a gaussian scene representation and the second mapping data is associated with a second set of gaussians of the gaussian scene representation, and where the second set of gaussians includes more gaussians of the gaussian scene representation than the first set of gaussians. For example, the means for obtaining the data associated with the region of interest from second mapping data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140. the update data selector 224. the gaussian selector 354, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or moreQUALCOMM Ref. No. 2501501WO- 98 / 130 -other circuits or components configured to obtaining the data associated with a region of interest, or any combination thereof.

[0271] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 1886) includes instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g.. the one or more processors 1810 or the processor 1806), cause the one or more processors to obtain input pixel data associated with a region of interest; determine whether first mapping data includes data associated with the region of interest; and based on a determination that the first mapping data does not include the data associated with the region of interest, obtain the data associated with the region of interest from second mapping data, wherein the first mapping data is associated with a first set of gaussians of a gaussian scene representation and the second mapping data is associated with a second set of gaussians of the gaussian scene representation, and wherein the second set of gaussians includes more gaussians of the gaussian scene representation than the first set of gaussians.

[0272] In conjunction with the described implementations, an apparatus includes means to store a gaussian scene representation. For example, the means for storing the gaussian scene representation can correspond to the device 102, the memory 120, the buffer(s) 118, the processor(s) 130, the selective model update pipeline 140, the update strategy selector 144, the partial updater 356, the integrated circuit 902, the input device 1830, the processor 1806, the processor(s) 1810. the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0273] The apparatus also includes means to obtain input pixel data. For example, the means for obtaining input pixel data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106, the modem 114, the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108, the camera(s) 1208, the camera(s) 1308, the camera(s) 1408, the camera(s) 1508, the camera(s) 1608, the camera(s) 1710, the input device 1830, the processor 1806, theQUALCOMM Ref. No. 2501501WO- 99 / 130 -processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0274] The apparatus also includes means to select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. For example, the means for selecting gaussians of the gaussian scene representation based on the input pixel data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the mapper 222, the gaussian selector 354, the integrated circuit 902, the processor 1806, the processor(s) 1810. one or more other circuits or components configured to select gaussians of a gaussian scene representation to be updated, or any combination thereof.

[0275] The apparatus also includes means to spatially transform the selected gaussians of the gaussian scene representation. For example, the means for spatially transforming the selected gaussians of the gaussian scene representation can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the updater 356, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to spatial transform gaussians, or any combination thereof.

[0276] In some implementations, a non-transitory computer-readable storage medium (e.g., a computer-readable storage device, such as the memory 1886) stores instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g., the one or more processors 1810 or the processor 1806), cause the one or more processors to perform operations to obtain input pixel data; select, based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation representing a scene; and spatially transform the selected gaussians of the gaussian scene representation.

[0277] In conjunction with the described implementations, an apparatus includes means to store a gaussian scene representation. For example, the means for storing the gaussian scene representation can correspond to the device 102, the memory 120, the buffer(s) 118, the processor(s) 130, the selective model update pipeline 140, the update strategy selector 144, the partial updater 356, the integrated circuit 902, the input device 1830,QUALCOMM Ref. No. 2501501WO- 100 / 130 -the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0278] The apparatus includes means to obtain input pixel data. For example, the means for obtaining input pixel data can correspond to the device 102. the memory 120. the buffer(s) 118, the image source(s) 106, the modem 114, the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108, the camera(s) 1208, the camera(s) 1308, the camera(s) 1408, the camera(s) 1508, the camera(s) 1608, the camera(s) 1710, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof. The apparatus includes means to select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. For example, the means for selecting gaussians of the gaussian scene representation based on the input pixel data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the mapper 222, the gaussian selector 354. the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to select gaussians of a gaussian scene representation to be updated, or any combination thereof.

[0279] The apparatus includes means to select, based on the input pixel data, gaussians of the gaussian scene representation to be updated. For example, the means for selecting gaussians of the gaussian scene representation based on the input pixel data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the mapper 222. the gaussian selector 354. the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to select gaussians of a gaussian scene representation to be updated, or any combination thereof.

[0280] The apparatus includes means to determine a set of parameter adjustment candidates for parameters of the selected gaussians. For example, the means forQUALCOMM Ref. No. 2501501WO- 101 / 130 -determining a set of parameter adjustment candidates for parameters of the selected gaussians can correspond to the device 102, the processor(s) 130. the selective model update pipeline 140, the updater 356, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to spatial transform gaussians, or any combination thereof.

[0281] The apparatus includes means to update the selected gaussians based on the set of parameter adjustment candidates. For example, the means for updating the selected gaussians based on the set of parameter adjustment candidates can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the updater 356, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to spatial transform gaussians, or any combination thereof.

[0282] In some implementations, a non-transitory computer-readable storage medium (e.g., a computer-readable storage device, such as the memory 1886) stores instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g., the one or more processors 1810 or the processor 1806), cause the one or more processors to perform operations to obtain input pixel data; select based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation associated with a scene; determine a set of parameter adjustment candidates for parameters of the selected gaussians; and updating the selected gaussians based on the set of parameter adjustment candidates.

[0283] In conjunction with the described implementations, an apparatus includes means to store a gaussian scene representation representing a scene. For example, the means for storing the gaussian scene representation can correspond to the device 102, the memory 120, the buffer(s) 118, the processor(s) 130, the selective model update pipeline 140, the update strategy selector 144, the partial updater 356, the integrated circuit 902, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0284] The apparatus includes means to obtain input pixel data corresponding to an input image depicting a portion of the scene. For example, the means for obtaining inputQUALCOMM Ref. No. 2501501WO- 102 / 130 -pixel data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106. the modem 114, the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108, the camera(s) 1208, the camera(s) 1308, the camera(s) 1408. the camera(s) 1508, the camera(s) 1608. the camera(s) 1710, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain input pixel data associated with a region of interest, or any combination thereof.

[0285] The apparatus includes means to obtain motion data based on two or more prior images of the scene. For example, the means for obtaining motion data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the motion calculator 382, the integrated circuit 902, the processor 1806, the processor(s) 1810. one or more other circuits or components configured to select gaussians of a gaussian scene representation to be updated, or any combination thereof.

[0286] The apparatus includes means to select, based, at least in part, on the input image, gaussians of the gaussian scene representation to be updated. For example, the means for selecting gaussians of the gaussian scene representation based on the input pixel data can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the mapper 222, the gaussian selector 354, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to select gaussians of a gaussian scene representation to be updated, or any combination thereof.

[0287] The apparatus includes means to adjust parameters of the selected gaussians to spatially transform the selected gaussians of the gaussian scene representation based on the motion data. For example, the means for adjusting parameters of the selected gaussians representation can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140. the updater 356, the integrated circuit 902, the processor 1806, the processor(s) 1810. one or more other circuits or components configured to spatial transform gaussians, or any combination thereof.QUALCOMM Ref. No. 2501501WO- 103 / 130 -

[0288] In some implementations, a non-transitory computer-readable storage medium (e.g., a computer-readable storage device, such as the memory 1886) stores instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g., the one or more processors 1810 or the processor 1806), cause the one or more processors to perform operations to store a gaussian scene representation representing a scene: obtain input pixel data corresponding to an input image depicting a portion of the scene; obtain motion data based on two or more prior images of the scene; select, based, at least in part, on the input image, gaussians of the gaussian scene representation to be updated; and adjust parameters of the selected gaussians to spatially transform the selected gaussians of the gaussian scene representation based on the motion data.

[0289] Particular aspects of the disclosure are described below in sets of interrelated Examples:

[0290] According to example 1, a device includes a memory configured to store a gaussian scene representation; and one or more processors coupled to the memory and configured to: obtain input pixel data; select, based on the input pixel data, gaussians of the gaussian scene representation to be updated; and spatially transform the selected gaussians of the gaussian scene representation.

[0291] Example 2 includes the device of example 1, wherein the one or more processors are configured to spatially transform the selected gaussians by adjusting parameters of the selected gaussians.

[0292] Example 3 includes the device of example 2, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, the plurality of gaussians including the selected gaussians, wherein each gaussian of the gaussian scene representation is associated with a three-dimensional coordinate location, and wherein the one or more processors are configured to modify three-dimensional coordinate locations associated with the selected gaussians to adjust the parameters of the selected gaussians to spatially transform the selected gaussians.

[0293] Example 4 includes the device of example 2, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, each gaussian associated with one or more orientation parameters, and wherein the one orQUALCOMM Ref. No. 2501501WO- 104 / 130 -more processors are configured to modify orientation parameters associated with the selected gaussians to adjust parameters of the selected gaussians to spatially transform the selected gaussians.

[0294] Example 5 includes the device of any of examples 1 to 4, wherein, to adjust parameters of the selected gaussians to spatially transform the selected gaussians, the one or more processors are configured to: render at least one view of the scene using the gaussian scene representation; compare the at least one view of the scene with the input pixel data to determine an error metric; and determine whether to further update the gaussian scene representation based on the error metric.

[0295] Example 6 includes the device of example 5, wherein the one or more processors are configured to modify three-dimensional coordinate locations associated with the selected gaussians based on the error metric.

[0296] Example 7 includes the device of any of examples 1 to 6, wherein the one or more processors are configured to: before adjusting parameters of the selected gaussians, determine a first error metric based on the input pixel data and reference pixel data; and after adjusting parameters of the selected gaussians: render at least one view of the scene using the gaussian scene representation; compare the at least one view of the scene with the input pixel data to determine a second error metric; and further adjust the parameters of the selected gaussians based on a comparison of the first error metric and the second error metric.

[0297] Example 8 includes the device of any of examples 1 to 7, wherein the one or more processors are configured to: determine whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest; and select the selected gaussians based on the mapping data.

[0298] Example 9 includes the device of example 8, wherein the one or more processors are configured to identify the region of interest in the input pixel data based on user input.

[0299] Example 10 includes the device of any of examples 1 to 9, wherein the one or more processors are configured to obtain the input pixel data based on user input.QUALCOMM Ref. No. 2501501WO- 105 / 130 -

[0300] Example 11 includes the device of any of examples 1 to 10, wherein the one or more processors are configured to select an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0301] Example 12 includes the device of example 11, wherein the plurality of update strategies comprise at least a partial update strategy to perform a partial update of the gaussian scene representation and a full update strategy to perform a full update of the gaussian scene representation, and wherein the one or more processors are configured to modify the selected gaussians of the gaussian scene representation and hold constant remaining gaussians of the gaussian scene representation in the partial update strategy.

[0302] Example 13 includes the device of example 11 or example 12, further comprising one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data, and wherein the one or more processors are configured to select the update strategy based, at least in part, on the sensor data.

[0303] Example 14 includes the device of any of examples 1 to 13, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

[0304] Example 15 includes the device of any of examples 1 to 14, wherein the memory includes an image buffer configured to store the input pixel data.

[0305] Example 16 includes the device of any of examples 1 to 15, further comprising a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the input pixel data, mapping data, or both, via the modem.

[0306] Example 17 includes the device of any of examples 1 to 16, further comprising a display device configured to display a view of the scene rendered subsequent to spatial transformation of the selected gaussians.

[0307] Example 18 includes the device of any of examples 1 to 17, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.QUALCOMM Ref. No. 2501501WO- 106 / 130 -

[0308] According to example 19, a method includes: obtaining, by one or more processors, input pixel data; selecting, by the one or more processors, based on the input pixel data, gaussians of a gaussian scene representation to be updated; and spatially transforming, by the one or more processors, the selected gaussians of the gaussian scene representation.

[0309] Example 20 includes the method of example 19, wherein spatially transforming the selected gaussians comprises adjusting parameters of the selected gaussians.

[0310] Example 21 includes the method of example 20. wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, the plurality of gaussians including the selected gaussians, wherein each gaussian of the gaussian scene representation is associated with a three-dimensional coordinate location, and wherein the method comprises modifying three-dimensional coordinate locations associated with the selected gaussians to adjust the parameters of the selected gaussians to spatially transform the selected gaussians.

[0311] Example 22 includes the method of example 20, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, each gaussian associated with one or more orientation parameters, and wherein the method comprises modifying orientation parameters associated with the selected gaussians to adjust parameters of the selected gaussians to spatially transform the selected gaussians.

[0312] Example 23 includes the method of any of examples 19 to 22, wherein adjusting parameters of the selected gaussians to spatially transform the selected gaussians comprises: rendering at least one view of the scene using the gaussian scene representation; comparing the at least one view of the scene with the input pixel data to determine an error metric; and determining whether to further update the gaussian scene representation based on the error metric.

[0313] Example 24 includes the method of example 23. further comprising modifying three-dimensional coordinate locations associated with the selected gaussians based on the error metric.QUALCOMM Ref. No. 2501501WO- 107 / 130 -

[0314] Example 25 includes the method of any of examples 19 to 24, further comprising: before adjusting parameters of the selected gaussians, determining a first error metric based on the input pixel data and reference pixel data; and after adjusting parameters of the selected gaussians: rendering at least one view of the scene using the gaussian scene representation; comparing the at least one view of the scene with the input pixel data to determine a second error metric; and further adjusting the parameters of the selected gaussians based on a comparison of the first error metric and the second error metric.

[0315] Example 26 includes the method of any of examples 19 to 25. further comprising: determining whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest; and selecting the selected gaussians based on mapping data.

[0316] Example 27 includes the method of example 26, further comprising identifying the region of interest in the input pixel data based on user input.

[0317] Example 28 includes the method of any of examples 19 to 27. wherein the input pixel data is obtained based on user input.

[0318] Example 29 includes the method of any of examples 19 to 28, further comprising selecting an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0319] Example 30 includes the method of example 29, wherein the plurality of update strategies comprise at least a partial update strategy to perform a partial update of the gaussian scene representation and a full update strategy to perform a full update of the gaussian scene representation, and wherein the method comprises modifying the selected gaussians of the gaussian scene representation and holding constant remaining gaussians of the gaussian scene representation in the partial update strategy.

[0320] Example 31 includes the method of example 29, further comprising generating, via one or more sensors coupled to the one or more processors, sensor data associated with the input pixel data; and selecting the update strategy based, at least in part, on the sensor data.QUALCOMM Ref. No. 2501501WO- 108 / 130 -

[0321] Example 32 includes the method of any of examples 19 to 31, further comprising capturing one or more images using one or more cameras coupled to the one or more processors, wherein the input pixel data is generated based on the one or more images.

[0322] Example 33 includes the method of any of examples 19 to 32. further comprising storing the input pixel data in an image buffer of a memory.

[0323] Example 34 includes the method of any of examples 19 to 33, further comprising receiving the input pixel data, mapping data, or both, via a modem coupled to the one or more processors.

[0324] Example 35 includes the method of any of examples 19 to 34, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0325] According to example 36, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: obtaining input pixel data; selecting, based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation representing a scene; and spatially transforming the selected gaussians of the gaussian scene representation.

[0326] Example 37 includes the non-transitory computer-readable storage medium of example 36, wherein spatially transforming the selected gaussians comprises adjusting parameters of the selected gaussians.

[0327] Example 38 includes the non-transitory computer-readable storage medium of example 37, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, the plurality of gaussians including the selected gaussians, wherein each gaussian of the gaussian scene representation is associated with a three-dimensional coordinate location, and wherein the operations comprise modifying three-dimensional coordinate locations associated with the selectedQUALCOMM Ref. No. 2501501WO- 109 / 130 -gaussians to adjust the parameters of the selected gaussians to spatially transform the selected gaussians.

[0328] Example 39 includes the non-transitory computer-readable storage medium of example 38, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, each gaussian associated with one or more orientation parameters, and wherein the operations comprise modifying orientation parameters associated with the selected gaussians to adjust parameters of the selected gaussians to spatially transform the selected gaussians.

[0329] Example 40 includes the non-transitory computer-readable storage medium of any of examples 36 to 39, wherein adjusting parameters of the selected gaussians to spatially transform the selected gaussians comprises: rendering at least one view of the scene using the gaussian scene representation; comparing the at least one view of the scene with the input pixel data to determine an error metric; and determining whether to further update the gaussian scene representation based on the error metric.

[0330] Example 41 includes the non-transitory computer-readable storage medium of example 40, the operations further comprising modifying three-dimensional coordinate locations associated with the selected gaussians based on the error metric.

[0331] Example 42 includes the non-transitory computer-readable storage medium of any of examples 36 to 41, the operations further include: before adjusting parameters of the selected gaussians, determining a first error metric based on the input pixel data and reference pixel data; and after adjusting parameters of the selected gaussians: rendering at least one view of the scene using the gaussian scene representation; comparing the at least one view of the scene with the input pixel data to determine a second error metric; and further adjusting the parameters of the selected gaussians based on a comparison of the first error metric and the second error metric.

[0332] Example 43 includes the non-transitory computer-readable storage medium of any of examples 36 to 42, the operations further comprising: determining whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest; and selecting the selected gaussians based on mapping data.QUALCOMM Ref. No. 2501501WO- 110 / 130 -

[0333] Example 44 includes the non-transitory computer-readable storage medium of example 43, the operations further comprising identifying the region of interest in the input pixel data based on user input.

[0334] Example 45 includes the non-transitory computer-readable storage medium of any of examples 36 to 44, wherein the input pixel data is obtained based on user input.

[0335] Example 46 includes the non-transitory computer-readable storage medium of any of examples 36 to 45, the operations further comprising selecting an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0336] Example 47 includes the non-transitory computer-readable storage medium of example 46, wherein the plurality of update strategies comprise at least a partial update strategy’ to perform a partial update of the gaussian scene representation and a full update strategy to perform a full update of the gaussian scene representation, and wherein the operations comprise modifying the selected gaussians of the gaussian scene representation and holding constant remaining gaussians of the gaussian scene representation in the partial update strategy.

[0337] Example 48 includes the non-transitory computer-readable storage medium of example 46, the operations further comprising generating sensor data associated with the input pixel data using one or more sensors coupled to the one or more processors and selecting the update strategy based, at least in part, on the sensor data.

[0338] Example 49 includes the non-transitory computer-readable storage medium of any of examples 36 to 48, the operations further comprising capturing one or more images using one or more cameras coupled to the one or more processors, wherein the input pixel data is generated based on the one or more images.

[0339] Example 50 includes the non-transitory computer-readable storage medium of any of examples 36 to 49, further comprising storing the input pixel data in an image buffer of a memory.

[0340] Example 51 includes the non-transitory computer-readable storage medium of any of examples 36 to 50, the operations further comprising receiving the input pixel data, mapping data, or both, via a modem coupled to the one or more processors.QUALCOMM Ref. No. 2501501WO- 111 / 130 -

[0341] Example 52 includes the non-transitory computer-readable storage medium of any of examples 36 to 51, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0342] According to example 53, a device includes: a memory configured to store a gaussian scene representation; and one or more processors coupled to the memory and configured to: obtain input pixel data; select, based on the input pixel data, gaussians of the gaussian scene representation to be updated; determine a set of parameter adjustment candidates for parameters of the selected gaussians; and update the selected gaussians based on the set of parameter adjustment candidates.

[0343] Example 54 includes the device of example 53, wherein the one or more processors are configured to: determine a set of error metric values based on the input pixel data and a set of rendered candidate views, wherein each candidate view of the set of rendered candidate views corresponds to a candidate partial update of the gaussian scene representation based on a respective parameter adjustment candidate of the set of parameter adjustment candidates, and wherein the update of the selected gaussians is based on the set of error metric values.

[0344] Example 55 includes the device of example 54, wherein one or more processors are configured to update the selected gaussians based on a particular candidate partial update of the gaussian scene representation associated with a smallest value among the set of error metric values.

[0345] Example 56 includes the device of example 54, wherein the one or more processors are configured to, after said update of the selected gaussians, determine whether to further update the gaussian scene representation based, at least in part, on whether one or more values of the set of error metric values satisfy a termination condition.

[0346] Example 57 includes the device of any of examples 53 to 56, wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate spatial transformation of the selected gaussians.QUALCOMM Ref. No. 2501501WO- 112 / 130 -

[0347] Example 58 includes the device of example 57, wherein the spatial transformation includes translation of the selected gaussians.

[0348] Example 59 includes the device of example 57, wherein the spatial transformation includes rotation of the selected gaussians.

[0349] Example 60 includes the device of example 59, wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate color transformation of at least one of the selected gaussians.

[0350] Example 61 includes the device of any of examples 53 to 60, wherein the one or more processors are configured to use dithering to determine the set of candidate parameter adjustments based on one or more distribution parameters associated with the selected gaussians.

[0351] Example 62 includes the device of any of examples 53 to 61, wherein the one or more processors are configured to determine whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest.

[0352] Example 63 includes the device of any of examples 53 to 62, wherein the one or more processors are configured to obtain the input pixel data based on user input.

[0353] Example 64 includes the device of any of examples 53 to 63, wherein the one or more processors are configured to identify a region of interest in the input pixel data based on user input.

[0354] Example 65 includes the device of any of examples 53 to 64, wherein the one or more processors are configured to select an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0355] Example 66 includes the device of example 65, wherein, when the selected update strategy is a partial update of the gaussian scene representation, the one or more processors are configured to modify the selected gaussians of the gaussian scene representation and hold constant remaining gaussians of the gaussian scene representation.QUALCOMM Ref. No. 2501501WO- 113 / 130 -

[0356] Example 67 includes the device of example 65, further comprising one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data, and wherein the one or more processors are configured to select the update strategy based, at least in part, on the sensor data.

[0357] Example 68 includes the device of any of examples 53 to 67, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

[0358] Example 69 includes the device of any of examples 53 to 68, wherein the memory includes an image buffer configured to store the input pixel data.

[0359] Example 70 includes the device of any of examples 53 to 69, further comprising a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the input pixel data, mapping data, or both, via the modem.

[0360] Example 71 includes the device of any of examples 53 to 70, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0361] According to example 72, a method includes: obtaining, by one or more processors, input pixel data; selecting, by the one or more processors, based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation associated with a scene; determining, by the one or more processors, a set of parameter adjustment candidates for parameters of the selected gaussians; and updating, by the one or more processors, the selected gaussians based on the set of parameter adjustment candidates.

[0362] Example 73 includes the method of example 72, further comprising: determining a set of error metric values based on the input pixel data and a set of rendered candidate views, wherein each candidate view of the set of rendered candidate views corresponds to a candidate partial update of the gaussian scene representation based on a respective parameter adjustment candidate of the set of parameter adjustment candidates, and wherein the update of the selected gaussians is based on the set of error metric values.QUALCOMM Ref. No. 2501501WO- 114 / 130 -

[0363] Example 74 includes the method of example 73, further comprising updating the selected gaussians based on a particular candidate partial update of the gaussian scene representation associated with a smallest value among the set of error metric values.

[0364] Example 75 includes the method of example 74, further comprising, after the update of the selected gaussians. determining whether to further update the gaussian scene representation based, at least in part, on whether one or more values of the set of error metric values satisfy a termination condition.

[0365] Example 76 includes the method of any of examples 72 to 75. wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate spatial transformation of the selected gaussians.

[0366] Example 77 includes the method of example 76. wherein the spatial transformation includes translation of the selected gaussians.

[0367] Example 78 includes the method of example 76, wherein the spatial transformation includes rotation of the selected gaussians.

[0368] Example 79 includes the method of example 78, wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate color transformation of at least one of the selected gaussians.

[0369] Example 80 includes the method of any of examples 72 to 79, wherein the set of candidate parameter adjustments are determined using dithering based on one or more distribution parameters associated with the selected gaussians.

[0370] Example 81 includes the method of any of examples 72 to 80, further comprising determining whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest.

[0371] Example 82 includes the method of any of examples 72 to 81, further comprising obtaining the input pixel data based on user input.

[0372] Example 83 includes the method of any of examples 72 to 82, further comprising identifying a region of interest in the input pixel data based on user input.QUALCOMM Ref. No. 2501501WO- 115 / 130 -

[0373] Example 84 includes the method of any of examples 72 to 83, further comprising selecting an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0374] Example 85 includes the method of example 84, wherein, when the selected update strategy is a partial update of the gaussian scene representation, the method comprises modifying the selected gaussians of the gaussian scene representation and holding constant remaining gaussians of the gaussian scene representation.

[0375] Example 86 includes the method of example 84. further comprising: obtaining, via one or more sensors coupled to the one or more processors, sensor data associated with the input pixel data; and selecting the update strategy based, at least in part, on the sensor data.

[0376] Example 87 includes the method of any of examples 72 to 86, further comprising capturing one or more images using one or more cameras coupled to the one or more processors, wherein the input pixel data is generated based on the one or more images.

[0377] Example 88 includes the method of any of examples 72 to 87, further comprising storing the input pixel data at an image buffer of a memory.

[0378] Example 89 includes the method of any of examples 72 to 88, further comprising obtaining the input pixel data, mapping data, or both, via a modem coupled to the one or more processors.

[0379] Example 90 includes the method of any of examples 72 to 89, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0380] According to example 91, a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processor to perform operations comprising: obtaining input pixel data; selecting based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation associated with a scene; determining a set ofQUALCOMM Ref. No. 2501501WO- 116 / 130 -parameter adjustment candidates for parameters of the selected gaussians; and updating the selected gaussians based on the set of parameter adjustment candidates.

[0381] Example 92 includes the non-transitory computer-readable storage medium of example 91, the operations further comprising: determining a set of error metric values based on the input pixel data and a set of rendered candidate views, wherein each candidate view of the set of rendered candidate views corresponds to a candidate partial update of the gaussian scene representation based on a respective parameter adjustment candidate of the set of parameter adjustment candidates, and wherein the update of the selected gaussians is based on the set of error metric values.

[0382] Example 93 includes the non-transitory computer-readable storage medium of example 92, the operations further comprising updating the selected gaussians based on a particular candidate partial update of the gaussian scene representation associated with a smallest value among the set of error metric values.

[0383] Example 94 includes the non-transitory computer-readable storage medium of example 93, the operations further comprising, after the update of the selected gaussians, determining whether to further update the gaussian scene representation based, at least in part, on whether one or more values of the set of error metric values satisfy a termination condition.

[0384] Example 95 includes the non-transitory computer-readable storage medium of any of examples 91 to 94, wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate spatial transformation of the selected gaussians.

[0385] Example 96 includes the non-transitory computer-readable storage medium of example 95, wherein the spatial transformation includes translation of the selected gaussians.

[0386] Example 97 includes the non-transitory computer-readable storage medium of example 95, wherein the spatial transformation includes rotation of the selected gaussians.QUALCOMM Ref. No. 2501501WO- 117 / 130 -

[0387] Example 98 includes the non-transitory computer-readable storage medium of example 97, wherein a parameter adjustment candidate of the set of parameter adjustment candidates represents a candidate color transformation of at least one of the selected gaussians.

[0388] Example 99 includes the non-transitory computer-readable storage medium of any of examples 91 to 98, the operations comprise determining the set of candidate parameter adjustments using dithering based on one or more distribution parameters associated with the selected gaussians.

[0389] Example 100 includes the non-transitory computer-readable storage medium of any of examples 91 to 99, the operations further comprising determining whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest.

[0390] Example 101 includes the non-transitory computer-readable storage medium of any of examples 91 to 100, the operations further comprising obtaining the input pixel data based on user input.

[0391] Example 102 includes the non-transitory computer-readable storage medium of any of examples 91 to 101, the operations further comprising identifying a region of interest in the input pixel data based on user input.

[0392] Example 103 includes the non-transitory computer-readable storage medium of any of examples 91 to 102, the operations further comprising selecting an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0393] Example 104 includes the non-transitory computer-readable storage medium of example 103. wherein, when the selected update strategy is a partial update of the gaussian scene representation, the operations comprise modifying the selected gaussians of the gaussian scene representation and holding constant remaining gaussians of the gaussian scene representation.

[0394] Example 105 includes the non-transitory computer-readable storage medium of example 103, the operations further comprising generating sensor data associated withQUALCOMM Ref. No. 2501501WO- 118 / 130 -the input pixel data via one or more sensors coupled to the one or more processors, and selecting the update strategy based, at least in part, on the sensor data.

[0395] Example 106 includes the non-transitory computer-readable storage medium of any of examples 91 to 105, the operations further comprising capturing, via one or more cameras coupled to the one or more processors, one or more images to generate the input pixel data.

[0396] Example 107 includes the non-transitory computer-readable storage medium of any of examples 91 to 106, the operations further comprising storing the input pixel data at an image buffer of a memory.

[0397] Example 108 includes the non-transitory computer-readable storage medium of any of examples 91 to 107, the operations further comprising obtaining the input pixel data, mapping data, or both, via a modem coupled to the one or more processors.

[0398] Example 109 includes the non-transitory computer-readable storage medium of any of examples 91 to 108, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0399] According to example 110. a device includes: a memory configured to store a gaussian scene representation representing a scene; and one or more processors coupled to the memory and configured to: obtain input pixel data corresponding to an input image depicting a portion of the scene; obtain motion data based on two or more prior images of the scene; select, based, at least in part, on the input image, gaussians of the gaussian scene representation to be updated; and adjust parameters of the selected gaussians to spatially transform the selected gaussians of the gaussian scene representation based on the motion data.

[0400] Example 111 includes the device of example 109, wherein the motion data indicates a direction of movement associated with an object in the scene, wherein the selected gaussians are associated with the object, and wherein the one or more processors are configured to adjust one or more location parameters of the selectedQUALCOMM Ref. No. 2501501WO- 119 / 130 -gaussians to spatially transform the selected gaussians based on the direction of movement.

[0401] Example 112 includes the device of example 111, wherein the motion data indicates a speed of movement associated with the object, and wherein the one or more processors are configured to adjust the one or more location parameters by an amount that is based on the speed of movement.

[0402] Example 113 includes the device of any of examples 110 to 112, wherein the motion data indicates rotation associated with an object in the scene, wherein the selected gaussians are associated with the object, and wherein the one or more processors are configured to adjust one or more orientation parameters of the selected gaussians to spatially transform the selected gaussians based on the rotation.

[0403] Example 114 includes the device of any of examples 110 to 113, wherein the one or more processors are configured to determine a region of interest based on mapping data and select the selected gaussians based on pixels associated with the region of interest.

[0404] Example 115 includes the device of any of examples 110 to 114, wherein the one or more processors are configured to obtain the input pixel data based on user input.

[0405] Example 116 includes the device of any of examples 110 to 115, wherein the one or more processors are configured to identify a region of interest in the input pixel data based on user input.

[0406] Example 117 includes the device of any of examples 110 to 116, wherein the one or more processors are configured to select an update strategy for the gaussian scene representation from among a plurality of update strategies.

[0407] Example 118 includes the device of example 117, wherein, when the selected update strategy is a partial update of the gaussian scene representation, the one or more processors are configured to modify a selected subset of gaussians of the gaussian scene representation and hold constant remaining gaussians of the gaussian scenerepresentation.QUALCOMM Ref. No. 2501501WO- 120 / 130 -

[0408] Example 119 includes the device of example 117, further comprising one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data, and wherein the one or more processors are configured to select the update strategy based, at least in part, on the sensor data.

[0409] Example 120 includes the device of any of examples 110 to 119, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

[0410] Example 121 includes the device of any of examples 110 to 120, wherein the memory includes an image buffer configured to store the input pixel data.

[0411] Example 122 includes the device of any of examples 110 to 121, further comprising a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the input pixel data, mapping data, or both, via the modem.

[0412] Example 123 includes the device of any of examples 110 to 122, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.

[0413] According to example 124. a method includes: obtaining, by one or more processors, input pixel data corresponding to an input image depicting a portion of a scene; obtaining, by the one or more processors, motion data based on two or more prior images of the scene; selecting, by the one or more processors, based, at least in part, on the input image, gaussians of a gaussian scene representation to be updated; and adjusting, by the one or more processors, parameters of the selected gaussians to spatially transform the selected gaussians based on the motion data.

[0414] Example 125 includes the method of example 4. wherein the motion data indicates a direction of movement associated with an object in the scene, wherein the selected gaussians are associated with the object, and wherein the method comprises adjusting one or more location parameters of the selected gaussians to spatially transform the selected gaussians based on the direction of movement.QUALCOMM Ref. No. 2501501WO- 121 / 130 -

[0415] Example 126 includes the method of example 125, wherein the motion data indicates a speed of movement associated with the object, and wherein the method comprises adjusting the one or more location parameters by an amount that is based on the speed of movement.

[0416] Example 127 includes the method of any of examples 124 to 126. wherein the motion data indicates rotation associated with an object in the scene, wherein the selected gaussians are associated with the...

Claims

QUALCOMM Ref. No. 2501501WO- 126 / 130 - WHAT IS CLAIMED IS:

1. A device comprising:a memory configured to store a gaussian scene representation; andone or more processors coupled to the memory and configured to:obtain input pixel data;select, based on the input pixel data, gaussians of the gaussian scene representation to be updated; andspatially transform the selected gaussians of the gaussian scene representation.

2. The device of claim 1, wherein the one or more processors are configured to spatially transform the selected gaussians by adjusting parameters of the selected gaussians.

3. The device of claim 2, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, the plurality of gaussians including the selected gaussians, wherein each gaussian of the gaussian scene representation is associated with a three-dimensional coordinate location, and wherein the one or more processors are configured to modify three-dimensional coordinate locations associated with the selected gaussians to adjust the parameters of the selected gaussians to spatially transform the selected gaussians.

4. The device of claim 2, wherein the gaussian scene representation represents a scene in three dimensions using a plurality of gaussians, each gaussian associated with one or more orientation parameters, and wherein the one or more processors are configured to modify orientation parameters associated with the selected gaussians to adjust parameters of the selected gaussians to spatially transform the selected gaussians.QUALCOMM Ref. No. 2501501WO- 127 / 130 - 5. The device of claim 1, wherein, to adjust parameters of the selected gaussians to spatially transform the selected gaussians, the one or more processors are configured to:render at least one view of the scene using the gaussian scene representation; compare the at least one view of the scene with the input pixel data to determine an error metric; anddetermine whether to further update the gaussian scene representation based on the error metric.

6. The device of claim 5, wherein the one or more processors are configured to modify three-dimensional coordinate locations associated with the selected gaussians based on the error metric.

7. The device of claim 1, wherein the one or more processors are configured to: before adjusting parameters of the selected gaussians, determine a first error metric based on the input pixel data and reference pixel data; and after adjusting parameters of the selected gaussians:render at least one view of the scene using the gaussian scene representation;compare the at least one view of the scene with the input pixel data to determine a second error metric; andfurther adjust the parameters of the selected gaussians based on a comparison of the first error metric and the second error metric.

8. The device of claim 1, wherein the one or more processors are configured to: determine whether mapping data includes data associated with a region of interest based on pixels associated with the region of interest; andselect the selected gaussians based on the mapping data.

9. The device of claim 8, wherein the one or more processors are configured to identify the region of interest in the input pixel data based on user input.

10. The device of claim 1, wherein the one or more processors are configured to obtain the input pixel data based on user input.QUALCOMM Ref. No. 2501501WO- 128 / 130 - 11. The device of claim 1, wherein the one or more processors are configured to select an update strategy for the gaussian scene representation from among a plurality of update strategies.

12. The device of claim 11, wherein the plurality of update strategies comprise at least a partial update strategy to perform a partial update of the gaussian scene representation and a full update strategy to perform a full update of the gaussian scene representation, and wherein the one or more processors are configured to modify the selected gaussians of the gaussian scene representation and hold constant remaining gaussians of the gaussian scene representation in the partial update strategy.

13. The device of claim 11, further comprising one or more sensors coupled to the one or more processors and configured to generate sensor data associated with the input pixel data14. The device of claim 13, wherein the one or more processors are configured to select the update strategy based, at least in part, on the sensor data.

15. The device of claim 1, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images to generate the input pixel data.

16. The device of claim 1, wherein the memory includes an image buffer configured to store the input pixel data.

17. The device of claim 1, further comprising a display device configured to display a view of the scene rendered subsequent to spatial transformation of the selected gaussians.

18. The device of claim 1, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.QUALCOMM Ref. No. 2501501WO- 129 / 130 - 19. A method comprising:obtaining, by one or more processors, input pixel data;selecting, by the one or more processors, based on the input pixel data, gaussians of a gaussian scene representation to be updated; andspatially transforming, by the one or more processors, the selected gaussians of the gaussian scene representation.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:obtaining input pixel data;selecting, based on the input pixel data, gaussians of a gaussian scene representation to be updated, the gaussian scene representation representing a scene; and spatially transforming the selected gaussians of the gaussian scenerepresentation.