Update strategy selection for gaussian scene representation based on sensor data

Selective update strategies for gaussian scene representations address computational challenges by reducing resource consumption through partial updates based on scene changes and resource availability, enhancing efficiency in applications like autonomous vehicle control.

WO2026240944A1PCT designated stage Publication Date: 2026-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/024832
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-27
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Gaussian scene representations require significant computing resources for updates due to frequent changes in the scene, particularly in applications like autonomous vehicle control, leading to computational challenges.

Method used

Implementing selective update strategies for gaussian scene representations based on input pixel data, sensor data, and device conditions to reduce computational complexity by performing partial updates only when necessary, such as when changes in the scene or resource availability allow.

Benefits of technology

Reduces computing resource consumption by selectively updating only portions of the gaussian scene representation, thereby maintaining accuracy with fewer computational resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026024832_19112026_PF_FP_ABST
    Figure US2026024832_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A device includes a memory configured to store a gaussian scene representation. The device also includes one or more processors coupled to the memory and configured to obtain sensor data corresponding to a scene modeled by the gaussian scene representation, and select, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.
Need to check novelty before this filing date? Find Prior Art

Description

QUALCOMM Ref. No. 2501500WO- 1 / 98 -UPDATE STRATEGY SELECTION FOR GAUSSIAN SCENE REPRESENTATION BASED ON SENSOR DATAI. Cross-Reference to Related Applications

[0001] The present application claims priority from the commonly owned U. S.Provisional Patent Application No. 63 / 804,910, filed May 13, 2025; U. S. Provisional Patent Application No. 63 / 831,815, filed June 27, 2025; U. S. Provisional Patent Application No. 63 / 822,724, filed June 12, 2025; U. S. Provisional Patent Application No. 63 / 830,101, filed June 25, 2025; U. S. Provisional Patent Application No.63 / 830,105, filed June 25, 2025; U. S. Provisional Patent Application No. 63 / 830,112, filed June 25, 2025; U. S. Provisional Patent Application No. 63 / 825,138, filed June 17, 2025; and U. S. Provisional Patent Application No. 63 / 820,238, filed June 9, 2025, the contents of each of which are expressly incorporated herein by reference in their entirety.II. Field

[0002] The present disclosure is generally related to gaussian scene representation.III. Description of Related Art

[0003] Computer science uses various techniques to generate three dimensional (3D) models or four dimensional (4D) models (where time is a fourth dimension) of a scene based on images of the scene. One example of a scene model used in computer science is a gaussian scene representation, which uses primitive elements commonly referred to as "gaussians".

[0004] A process referred to as gaussian splatting can be used to generate views based on the gaussian scene representation. The views can include views of the scene from various perspectives, and in particular, the gaussian scene representation can be used to generate novel views of the scene (e.g., views of the scene that do not correspond directly to any particular image used to generate the gaussian scene representation). For example, a gaussian scene representation can be used to generate images or sequences of images (e.g., videos) that simulate how the scene would appear from a different vantage point, perspective, or lighting condition, gaussian scene representations can beQUALCOMM Ref. No. 2501500WO- 2 / 98 -used to generate imagery for computer vision, graphics, video games, extended reality visualization. 3D reconstruction, security and surveillance, medical imaging, computer assisted navigation, as well as other uses.

[0005] One challenge associated with using gaussian scene representations is that a large set of training data (e.g., images of the scene from a variety of viewpoints) are generally processed in order to generate an accurate gaussian scene representation. Gaussian scene representations also tend to be large (e.g., in terms of data storage used). As a result, use of a gaussian scene representation to generate a novel view consumes significant computing resources (e.g., memory utilization, processor utilization, power, etc.), and training the gaussian scene representation uses even more computer resources than view generation. Consumption of computing resources is particularly challenging in situations where a gaussian scene representation is updated frequently due to changes in the scene over time. To illustrate, in an autonomous vehicle control use case, a gaussian scene representation can be used for various computer vision tasks, such as obstacle avoidance, compliance with traffic laws, etc. In order to perform these tasks well, the gaussian scene representation may be updated frequently as the vehicle moves through the scene and / or as other vehicles, pedestrians, etc., move nearby.IV. Summary

[0006] According to one implementation of the present disclosure, a device includes a memory configured to store a gaussian scene representation. The device also includes one or more processors coupled to the memory and configured to obtain sensor data corresponding to a scene modeled by the gaussian scene representation, and select, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0007] According to one implementation of the present disclosure, a method includes obtaining sensor data corresponding to a scene modeled by a gaussian scene representation. The method also includes selecting, from among a plurality of update strategies, an update strategy' for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.QUALCOMM Ref. No. 2501500WO- 3 / 98 -

[0008] According to one implementation of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain sensor data corresponding to a scene modeled by a gaussian scene representation. The instructions also cause the one or more processors to select, from among a plurality of update strategies, an update strategy’ for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0009] According to one implementation of the present disclosure, an apparatus includes means for obtaining sensor data corresponding to a scene modeled by a gaussian scene representation. The apparatus also includes means for selecting, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.V. Brief Description of the Drawings

[0011] FIG. 1 is a block diagram of an illustrative system operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure.

[0012] FIG. 2 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0013] FIGS. 3A, 3B, and 3C, together, illustrate particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.

[0014] FIG. 4 illustrates particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501500WO- 4 / 98 -

[0015] FIGS. 5A, 5B, 5C, and 5D are flow charts illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0016] FIG. 6 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0017] FIG. 7 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0018] FIG. 8 is a flow chart illustrating particular aspects of a method associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.

[0019] FIG. 9 illustrates an example of an integrated circuit operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0020] FIG. 10 is a diagram of a mobile device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0021] FIG. 11 is a diagram of a mixed reality or augmented reality glasses device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0022] FIG. 12 is a diagram of a wearable electronic device operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0023] FIG. 13 is a diagram of a voice-controlled speaker system operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2501500WO- 5 / 98 -

[0024] FIG. 14 is a diagram of a camera operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0025] FIG. 15 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0026] FIG. 16 is a diagram of a first example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0027] FIG. 17 is a diagram of a second example of a vehicle operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.

[0028] FIG. 18 is a block diagram of a particular illustrative example of a device that is operable to perform operations associated with updating gaussian scene representations, in accordance with some examples of the present disclosure.VI. Detailed Description

[0029] Use of a gaussian scene representation in a situation in which the gaussian scene representation is periodically or occasionally updated to capture new scene information (e.g., new image(s)) presents significant challenges due to the computational complexity of updating gaussian scene representations. Aspects disclosed herein reduce the computational complexity associated with updating a gaussian scene representation by using different update strategies to update the gaussian scene representation in different situations. For example, one update strategy can include performing a full update of the gaussian scene representation, and another update strategy is to perform a partial update of the gaussian scene representation. As used herein, a "partial update” refers to updating a specified subset of the gaussians of the gaussian scene representation while holding other gaussians of the gaussian scene representation constant. The specified subset of the gaussians updated during a partial update are selected based, at least inQUALCOMM Ref. No. 2501500WO- 6 / 98 -part, on changes detected in the scene (e.g., motion, lighting changes, differences between an input image and a reference image, etc.).

[0030] Selectively updating only a portion of the gaussian scene representation when the partial update strategy is used uses significantly fewer computing resources than performing the full update. In contrast to conventional update systems, which always perform a full update when any update is performed, the disclosed aspects provide computationally efficient mechanisms for identifying the portions of the gaussian scene representation that should be updated based on available information. As a result, computing resources used to maintain an up-to-date gaussian scene representation can be significantly reduced. Thus, a technical advantage of the disclosed aspects is a reduction of computing resources used to update gaussian scene representations.

[0031] In a particular aspect, a system includes one or more processors configured to obtain input pixel data associated with a scene and to select, based on an update criterion, an update strategy for the gaussian scene representation based on the input pixel data. The update criterion can be based on the input pixel data itself. For example, the input pixel data can be analyzed to identify motion, lighting changes, areas with pixel changes, or other characteristics, where changes are relative to training data (e.g.. a training image) used to train the gaussian scene representation or synthetic data (e g., a synthetic image generated using the gaussian scene representation). In this example, the update criterion can include one or more thresholds, and results of the analysis of the input pixel data can be compared to the threshold(s) to select an update strategy’, from among a plurality' of available update strategies, to be used to update the gaussian scene representation based on the input pixel data.

[0032] As another example, the update condition can be based on sensor data. For example, the sensor data can be analyzed to identify motion, lighting changes, or other characteristics associated with updating the gaussian scene representation. In this example, the sensor data can include light data (e.g., to indicate lighting changes) or motion data (e.g., from an accelerometer or ranging sensor or based on image analysis), as illustrative examples. In this example, the update criterion can include one or more thresholds, and results of the analysis of the sensor data can be compared to the threshold(s) to determine whether to update the gaussian scene representation based onQUALCOMM Ref. No. 2501500WO- 7 / 98 -the input pixel data. To illustrate, the sensor data can include a value of a lighting measurement from a lighting sensor or based on image analysis, which can be compared to a reference lighting value to determine a lighting change value. The lighting change value can be compared to one or more thresholds to select an update strategy for the gaussian scene representation. A technical advantage of using sensor data to select the update strategy is that processing the sensor data uses fewer computing resources than performing a comparison between input pixel data and reference pixel data associated with the gaussian scene representation and / or uses fewer computing resources than performing a full update of the gaussian scene representation; accordingly, using the sensor data to detect changes in the scene in order to select an update strategy’ can save significant computing resources.

[0033] As another example, the update condition can be based on a device condition. In this example, the device condition can be indicative of usage of a computational resource. To illustrate, the device condition can include or indicate a battery level, a processor load, a bus load, a cache load, a memory usage, an available network bandwidth, and / or a memory' access latency, as non-limiting examples. In this example, the update strategy for the gaussian scene representation can be selected based on levels of available computing resources. A technical advantage of using the device condition to select the update strategy for the gaussian scene representation is that the gaussian scene representation can be maintained using appropriate updates (e.g., full or partial) when sufficient computing resources are available, and more computationally expensive updates (e.g.. full updates or partial updates using more gaussians) can be deferred or limited at other times, thereby limiting the impact of updating the gaussian scene representation on other processes.

[0034] The available update strategies that can be selected among can include at least a full update, a partial update, and no update. Further, in some cases, the available update strategies can include various types of partial updates, such as object-based updates, pixel-based updates, updates that are limited to specified parameters of a selected subset of gaussians, staged search updates, etc. In a full update, all of the gaussians of the gaussian scene representation are processed (generally, based on all of the available views). For example, during a full update, a large number of camera poses are used, and a rendered view is generated for each. Each rendered view is compared to a ground-QUALCOMM Ref. No. 2501500WO- 8 / 98 -truth view (e.g., an image of the scene captured by a camera) to calculate a loss function. A value or values of the loss function are used to determine gradients based on the differences between an output of the gaussian scene representation (e.g., a rendered view) and an expected output of the gaussian scene representation (e.g., a ground-truth view), and parameters of one or more of the gaussians of the gaussian scene representation are updated based on the gradients. In this example, during a full update, gradients are determined for every parameter of every gaussian of the gaussian scene representation, which is very computationally expensive.

[0035] In contrast to a full update, during a partial update only a selected subset of the gaussians of the gaussian scene representation are updated and remaining gaussians are not modified. For example, a first set of gaussians of the gaussian scene representation are selected for update and the remaining gaussians of the gaussian scene representation are held constant during the partial update. In some cases, only selected parameters of the first set of gaussians are updated. For example, each gaussian is specified by a set of parameters including location parameters indicating a location of a center of the gaussian in a scene space, an opacity parameter indicating an opacity (e.g., a maximum opacity) of the gaussian, one or more orientation parameters indicating an orientation of the gaussian in the scene space, a set of color parameters indicating spherical harmonics coefficients for defining a color of the gaussian as viewed from various directions, and so on. In this example, for some partial updates, a first subset of the parameters of each selected gaussian are updated and a second subset of the parameters of each selected gaussian are held constant. To illustrate, the location parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant. As another illustrative example, one or more color parameters of the selected gaussians may be updated while the remaining parameters of the selected gaussians are held constant.

[0036] The processor(s) can be configured to select an update strategy to be used, from among a plurality of available update strategies, based on the input pixel data, based on sensor data, based on a device condition, based on other factors (e.g., user input), or based on a combination thereof. For example, the processor(s) can be configured to perform no update when a metric associated with the input pixel data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first thresholdQUALCOMM Ref. No. 2501500WO- 9 / 98 -but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. In this example, the metric can be based on an amount of difference between the input pixel data and reference pixel data, a size or dimensions of a region that differs between the input pixel data and reference pixel data, an amount of motion detected in the input pixel data, an amount of lighting change detected in the input pixel data, or other factors that are indicative of whether the gaussian scene representation accurately captures information in the input pixel data. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated with the sensor data fails to satisfy a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. As another example, the processor(s) can, additionally or alternatively, be configured to perform no update when a metric associated w ith the device condition fails to satisfy' a first threshold, to perform a partial update when the metric satisfies the first threshold but fails to satisfy a second threshold, and to perform a full update when the metric satisfies the second threshold. Metrics based on separate threshold comparisons for each factor are provided as an illustrative example; in other examples a metric can be based on a comparison of a combination of factors and corresponding thresholds. In addition to selecting between no update, a partial update, and a full update, in some cases, the processor(s) can select among several available partial update strategies.

[0037] The processor(s) can be configured to select which portions of the gaussian scene representation (e.g., which gaussians) are eligible to be modified during a partial update based on the input pixel data, based on sensor data, or both. For example, the processor(s) can identify' a region of interest associated with differences between the input pixel data and the reference pixel data. In this example, the processor(s) can use mapping data to identify gaussians of the gaussian scene representation that are associated with the region of interest. In this example, the identified gaussians are selected to be updated (e.g., are eligible to be modified) as part of the partial update. In some cases, the selected gaussians can affect pixels associated with the region of interest as well as other pixels. In such cases, mapping data can be used to determineQUALCOMM Ref. No. 2501500WO- 10 / 98 -which pixels of the input pixel data are to be used as training data to update the selected gaussians.

[0038] In a particular aspect, the mapping data can be generated based on gradient data. For example, saliency mapping can be used to estimate which gaussians most affect particular output pixel data from the gaussian scene representation to generate pixel-to-gaussian mapping data. Likewise, saliency mapping can be used to estimate which pixels of input data are most important to training of particular gaussians to generate gaussian-to-pixel mapping data.

[0039] Saliency mapping generally involves at least one forward pass through the gaussian scene representation (e.g., rendering one or more views) to generate gradients used to determine the pixel-to-gaussian mapping data, and at least one backpropagation pass to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially when a gaussian scene representation is updated frequently. To reduce the computational demand associated with running the gaussian scene representation to generate mapping data, particular aspects disclosed herein use a staged approach to manage the mapping data. In the staged approach, after mapping data (e.g.. pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, as described above, the mapping data can be stored for subsequent reference. For example, after mapping data associated with a particular region of interest is generated, the mapping data can be stored in a searchable data store. In this example, when new input image data is to be used for a partial update of the gaussian scene representation, a new region of interest in the new input image data is identified, and the searchable data store is searched to determine whether mapping data corresponding to pixels of the new region of interest is available (e.g., has already been generated and stored). If the mapping data is available, the mapping data can be retrieved from the searchable data store and used without the need to perform saliency mapping operations to generate the mapping data.

[0040] The region of interest used in the staged approach can be associated with a particular set of pixels, with a particular spatial area within the scene, with a particular object (or a portion of an object) identified in the scene, or with another identifiable feature of the scene. As one example, objects within the scene can be identified andQUALCOMM Ref. No. 2501500WO- 11 / 98 -saliency mapping can be used to associate one or more of the objects with gaussians of the gaussian scene representation. In this example, when new input image data is obtained, object detection can be performed to identify objects in the new input image data. If a particular object identified in the new input image data is associated with a change that triggers a partial update of the gaussian scene representation, the searchable data store is searched to determine whether mapping data is available for the particular object. If the mapping data for the particular object is available in the searchable data store, the mapping data is used for the partial update of the gaussian scene representation. If the mapping data for the particular object is not available in the searchable data store, the mapping data can be generated using saliency mapping.

[0041] Alternatively, in some cases, if particular mapping data is not available in a first searchable data store, a second searchable data store can be searched for the mapping data. For example, a set of most frequently used mapping data can be stored at a first searchable data store, a set of next most frequently used mapping data can be stored at a second searchable data store, and so forth, to an Nth data store (where N is an integer greater than 2). In such cases, full mapping data generation (e.g., using saliency mapping) is performed if none of the searchable data stores include the desired mapping data. Further, in some cases, after full mapping data generation is performed, the resulting mapping data can be added to one or more of the searchable data stores. Each of the searchable data stores can use an eviction policy in which a set of most recently used or most frequently used mapping data is retained at the first searchable data store, and each subsequent searchable data store retains successively less recently used or less frequently accessed mapping data.

[0042] In some cases, the staged approach can use a single searchable data store, rather than several stages of searchable data stores. In such cases, the single searchable data store can evict the oldest or the least recently accessed mapping data when new mapping data is added. In such cases, full mapping data generation is performed if the single searchable data store does not include the desired mapping data. A technical advantage of the staged approach is that significant computing resources can be preserved by avoiding recreating mapping data when new input image data is to be used for a partial update of the gaussian scene representation.QUALCOMM Ref. No. 2501500WO- 12 / 98 -

[0043] In the present discussion, a “gaussian’" refers to a graphical element of a gaussian scene representation. Each gaussian is specified by a set of parameters indicating characteristics, described further below, of the gaussian in scene space. In some examples, gaussians have a generally ellipsoidal 3D shape in the scene space, with a maximum opacity at a center position of the gaussian. In some examples, the opacity diminishes following an equation similar to a Gaussian probability distribution.

[0044] In a particular aspect, the parameters specifying a set of gaussians include:• a position vector. p(, which indicates the center position of a gaussian with index z;• an opacify parameter, tLG [0,1], which indicates a maximum opacify of the gaussian;• orientation parameters, qt. which can include a quaternion defining the spatial orientation (rotation) of the gaussian;• a scale vector, sf, which defines a size and shape of the ellipsoid of the gaussian;and• a set of spherical harmonics (SH) coefficients, H,. which can include a matrix with columns composed of 3D color vectors (in portions of this disclosure H, is referred to as an SH coefficient array).

[0045] In some implementations, not all of the parameters described above are used to specify a gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacify parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a view-independent color) rather than higher-order spherical harmonics that capture view-dependent color variations. Using a reduced parameter set can decrease memory usage and computational complexify associated with storing.QUALCOMM Ref. No. 2501500WO- 13 / 98 -updating, and rendering the gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.

[0046] In some implementations, the gaussian scene representation may use two-dimensional (2D) gaussians rather than three-dimensional (3D) gaussians. A 2D gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D gaussians, which distribute density throughout a model the complete volumetric ellipsoid, 2D gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.

[0047] A 2D gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parameters (e.g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D gaussian primitive may have fewer parameters than a corresponding 3D gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.

[0048] Using 2D gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D gaussian primitives can be more tightly aligned to object surfaces than 3D gaussians. Additionally, 2D gaussians may provide more accurate geometry' representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.

[0049] In some implementations, the gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each gaussian of the gaussian scene representation can be used to represent a triangular face of a mesh,QUALCOMM Ref. No. 2501500WO- 14 / 98 -enabling conversion between gaussian representations and mesh representations. This approach provides a one-to-one mapping between gaussians and triangular mesh faces that is differentiable, enabling the gaussian scene representation to be modified in a manner similar to modifying a mesh.

[0050] To convert a 2D gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D gaussian may be performed. The covariance matrix Z can be decomposed as S = RART, where R is a matrix of eigenvectors (el, e2, e3) and A is a diagonal matrix of eigenvalues (XI, X2, X3). The two largest eigenvalues ( I, X2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (X3) corresponds to the normal direction. A triangular face can then be constructed using the mean of the gaussian as a center point and defining triangle vertices along the tangent directions. For example, triangle vertices may be defined as vl = p + alel, v2 = p + a2e2. and v3 = p - alel, where al and a2 are scale factors proportional to square roots of their respective eigenvalues XI and X2 and p is the center point corresponding to the mean of the 2D gaussian. This creates a triangle aligned with the gaussian's orientation and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the gaussian.

[0051] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the gaussian. When the mesh is modified, the gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the gaussians when the mesh is animated or otherwise modified. During rendering, the gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of gaussians of the gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between gaussians and triangular faces.

[0052] Throughout this disclosure, references to gaussians, gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D gaussians, 2D gaussians, mesh-integrated gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the type of gaussian representation used, as described herein.QUALCOMM Ref. No. 2501500WO- 15 / 98 -

[0053] The following elements may be determined based on the parameters above:• S is a 3x3 matrix similar to the covariance matrix in Gaussian probability distributions.• c is a 3D view-dependent color vector for each gaussian, computed using the set of spherical harmonics coefficients of the gaussian.

[0054] The three dimensions of the color vector c may correspond to three different color component values, such as a red value, a green value, and a blue value, that specify a color for the gaussian. The color vector c is view-dependent in the sense that the gaussian may appear to have different colors depending on a position from which the gaussian is viewed, e.g., due to reflections. The view-dependent color vector c for the gaussian may be derived from a camera position f. the center position p, of the gaussian, and the SH coefficient array H;of the gaussian. The camera position f and the center position p, may each be a vector of spherical coordinates (9, <p). Thus, a viewing direction vector comprising a Q and a coordinate value that defines a viewing direction from the camera position f to the center position p, may be described as (f -p;). Therefore, the view -dependent color vector c for a gaussian i having central position p;from camera position f may be written as c / (f - p;, H;) or simply c(f - p, H).

[0055] In certain 3D Gaussian Splatting (3DGS) processes, H, includes up to 16 spherical harmonics coefficients per color vector component (e.g., 16 spherical harmonics components for red, 16 spherical harmonics components for green, and 16 spherical harmonics components for blue). A Tenderer may calculate a value of a color component in the view -dependent color vector c based on a linear combination LC of a set of terms Ti... Ymax scaled by the spherical harmonics coefficients in H, for the color component. That is, the Tenderer may calculate the linear combination LCtj for gaussian i and color component / as:LC = H T EI + H 7Y7 + H maX^maX! j=1,2,3, (1)Equation (1) may be rewritten as follows:QUALCOMM Ref. No. 2501500WO- 16 / 98 - Cf = H; y, (2)etc.,arespherical harmonics coefficients for gaussian i in H;for the color component / . Each of the terms in vector y may be defined by the following spherical harmonics equation:In equation (3) above, the values (0, < / >) specify the viewing direction vector derived from f and pi. The values I and m are the degree and order of the spherical harmonics equation, where - I < m < I. Thus, in an example where there are 16 spherical harmonics coefficients, the maximum value of / is 3, and when I is 0. m is 0; when Z is 1, m is 1.0, - 1; when / is 2, m is 2, 1, 0, -1, -2, and so on. Thus, Ti may correspond to I being 0 and m being 0, Yi may correspond to I being 1 and m being 1, Y? may correspond to I being 1 and m being 0, Y may correspond to I being 1 and m being -1, Fs may correspond to I being 2 and m being 2, and so on. With respect to conventional RGB values, the Tenderer may calculate the value of a color component in the view-dependent color vector c by applying a sigmoid function to LC and multiplying by a dynamic range scaling factor, such as 255.

[0056] To generate a rendered image based on a set of gaussians, a Tenderer may determine color values of pixels in the rendered image. A rendered image is a type of digital image that is composed of a grid of individual pixels, each with its own color vector. Each pixel is associated with a camera ray extending from the pixel through a virtual aperture of a virtual camera. In conventional 3DGS rendering, for each pixel of the rendered image, the Tenderer may identify gaussians that intersect the camera ray associated with the pixel and sort the identified gaussians based on a depth from the virtual camera. For each of the identified gaussians, the Tenderer calculates an opacify value for a location within the gaussian based on the position vector of the gaussian, the maximum opacity of the gaussian, the orientation vector of the gaussian, and the scaleQUALCOMM Ref. No. 2501500WO- 17 / 98 -vector of the gaussian. The Tenderer calculates the opacity value (a) for 3D location x along the camera ray within a gaussian z (i. e., a;(x)) as follows:1,2, • • •, N. (4)

[0057] In equation (4) and throughout this disclosure, the exp function is ex, where e is Euler's number, (,.)Tindicates matrix transposition, indicates a matrix inversion, and N indicates the total quantity of identified gaussians that intersect the camera ray. In general terms, equation (4) scales the maximum opacity value (ti) of the gaussian based on a Gaussian distribution function according to a distance of the location x from a center pi of the gaussian. The location x may be defined as a vector of coordinates, e.g., spherical coordinates, Cartesian coordinates, etc.

[0058] When processing a location x within a gaussian, the Tenderer also calculates a view-dependent color vector c for the gaussian. The view-dependent color vector c may include color component values that define a color for the gaussian, such as red, green, and blue values. Considering a virtual camera with a focal point (i.e., virtual aperture) at position f, the Tenderer may calculate an alpha-scaled color vector r, for gaussian i. The alpha-scaled color vector specifies a contribution of a location x within the gaussian z to a color of a pixel. The alpha-scaled color vector may be calculated using the following formula:Tj(x,f) = «i(x) c(f - pi, Hi), i = 1,2, -, N. (5)

[0059] Thus, equation (5) determines an alpha-scaled color vector r for a location x viewed from position f by scaling the view-dependent color vector c of a gaussian z that contains the location x based on the previously determined opacity value for the location x (i.e., «i(x)). The view-dependent color vector c may be determined using a spherical harmonics equation based on the SH coefficient matrix Hi of the gaussian and the differences between coordinates the center position of the gaussian (f - pi) as described above. The view-dependent color vector of a gaussian may be dependent on the viewingQUALCOMM Ref. No. 2501500WO- 18 / 98 -position f because of the directions of light sources, which can cause shadows, reflections, and other optical effects.

[0060] Depending on the size and orientation of a gaussian and a discretization of the 3D space into discrete locations, there may be multiple locations along a camera ray that intersect the same gaussian. Hence, the Tenderer may calculate multiple opacity values and color vectors for the same gaussian. The Tenderer may use these opacity values and color values in a rendering equation for determining a color vector of a pixel of a rendered image. However, for ease of explanation, this disclosure and the equations thereof assume that there is only one location per gaussian.

[0061] An advantage of using the formula of equation (4) is that mapping gaussians to the 2D rendered image can be approximated by a 2D gaussian with a 2x2 covariance matrix that can be computed asS2D= J WS3DWTJT, (6)where W is a matrix defined by the camera’s image-generation transformation, and J is the Jacobian matrix defined by an affine approximation of the projective camera transformation.

[0062] For ease of explanation, this disclosure omits some details about how rendering is discretized for image pixels, and considers that, for all gaussians, there are opacity' values [m, n] and color vectors ct[m, n] defined for each image pixel with integer image coordinates [m, n]. The image coordinates of a pixel indicate a 2D location of the pixel within a rendered image. In other words, each pixel of a rendered image is identified by a set of coordinates m and n. Due to the transparency of gaussians, before a scene is rendered, the gaussians to be rendered are sorted according to their distance from the camera position f. Formally, a sorting sequence s[ / c] of indexes of gaussians is determined according to increasing distance, such thatQUALCOMM Ref. No. 2501500WO- 19 / 98 - In equation (7), the double straight brackets (i.e., ||... ||) indicate Euclidean distance.

[0063] Thus, the Tenderer processes locations (e.g., gaussians) along the camera ray associated with a pixel in an order from closest to the virtual camera to further from the virtual camera. Hence, if the opacity values of locations within the gaussians intersected by the camera ray associated with the pixel reach a level corresponding to complete opacity, later gaussians that intersect the camera ray do not contribute to the color vector of the pixel. Hence, the Tenderer can skip processing of locations along the camera ray associated with the pixel that intersect the later gaussians.

[0064] The Tenderer may compute a final color vector for a pixel based on the opacity values and the view-dependent color values for the locations along the camera ray associated with the pixel. For instance, after sorting the gaussians based on depth, the Tenderer may compute a rendered color vector r for the pixel according to the following equation, which corresponds to the computer graphics technique of alpha-blending:

[0065] In equation (8). N indicates a total quantity of graphics elements (e.g., gaussians) along the camera ray associated with the pixel at coordinates \m, n], The termcS[k] [m, n] indicates a color vector for a distance-sorted graphics element having index value k in a sorting order of the graphics elements. In an example where the graphics element is a gaussian, cs[fe] [m, n] may be a view-dependent color vector for the gaussian. The term as[fe] [m, n] indicates an opacity value for the distance-sorted graphics element having an index value k in the sorting order. Similarly, as[i] [m, n] indicates an opacity value for the distance-sorted graphics element having an index value I in the sorting order. Note that in equation (8), the term ^[^(l — asp] [m, n]) is calculated individually for each of index values k=\ to N so that the scaled color vector for the graphics element with index value k (cs[fc] [m, n] as[fe] [m, n]) is further scaled based on the opacity values of graphics elements closer to the camera position than theQUALCOMM Ref. No. 2501500WO- 20 / 98 -graphics element with index value k. Thus, a Tenderer using equation (8) sorts the graphics elements based on depth from the camera position.

[0066] In complex scenes, like those used in computer games, most of the graphics elements are assumed to be opaque (i. e.,= 1 in the object's surface), so equation (8) can be implemented in a massively parallel manner by storing, for each pixel, the minimum depth of the elements already rendered, and only updating the color vector of a pixel if the newly visited element has smaller depth (Z-buffer updates). This is what enables current graphics processing units (GPUs) to render complex scenes at high resolutions and frame rates. On the other hand, when there is transparency, equation (8) defines a strictly sequential order for processing elements, and the only way to parallelize rendering is to create sets of non-overlapping elements and implement sequential alpha-blending on each set.

[0067] This made introduction of transparency in games challenging, but transparency is needed to represent many features, like flames and smoke, clouds, “force fields,” and so on. One proposed empiric solution to avoid sorting, named Order-Independent Transparency (OIT), is to approximate equation (8) with a weighted sum. There are several OIT variants, such as to replace equation (8) with the following equation for determining a rendered color vector f for a pixel at coordinates [m, n] of a rendered image:where zLis the distance to a virtual camera, c0[m, n] is the background color for the pixel at position [m,n] in the rendered image, and w(z, a) is a function that decreases with distance, so that objects nearer to the camera are assigned larger weights in the alphanormalized weighted sum of equation (9). The Tenderer may calculate the term a[m, n] using the following equation:QUALCOMM Ref. No. 2501500WO- 21 / 98 - Note that equation (9) is a weighted sum of a first value and a second value, where the first value is the background color vector co for the camera ray associated with pixel [m, n] scaled by the accumulated opacity (a[m, n]) of the camera ray, and the second term is a composited color vector scaled by the accumulated transparency (1 — a[m, n]) of the camera ray. In the OIT rendering of equation (9), there are two summation operations ■■■ for determining a color vector for a pixel, and since addition is commutative, the terms in the summation operations can be computed in any order. However, the final color vector for a pixel can only be computed after the two sums are known, which can be done with a single pass over all pixels, performing the divisions in equation (9). In other words, the Tenderer can perform the summation operations for all pixels, and then, after the summation operations for all pixels are complete, perform the division operations for all pixels. Since OIT uses the term w(z, a) instead of the actual opacity values of closer graphics elements, OIT may eliminate the need for sorting the graphics elements based on depth during rendering. However, use of the term w(z, a) instead of the actual opacity values of closer graphics elements means that OIT only provides an approximation of alpha blending and does not model truly opaque surfaces.

[0068] Because OIT allows rendering transparency without sorting, OIT may be considered for eliminating the computationally expensive sorting phase of Gaussian splatting rendering. However, while OIT methods have been extensively used in computer games for several years, the OIT methods were developed to be used with conventional computer graphics, and only meant to represent semi-transparent elements. Furthermore, the OIT methods are not expected to generate good approximations since the OIT methods are used with elements of shape and appearance where differences are difficult to see, like flames and smoke. Finally, human intervention may be needed to identify when the OIT methods are not working correctly, and to fix problems.

[0069] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms '‘a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in otherQUALCOMM Ref. No. 2501500WO- 22 / 98 -implementations. To illustrate, FIG. 1 depicts a device 102 including one or more processors (“processor(s)” 130 of FIG. 1). which indicates that in some implementations the device 102 includes a single processor 130 and in other implementations the device 102 includes multiple processors 130. For ease of reference herein, such features are generally introduced as "one or more’' features and may subsequently be referred to in the singular or optional plural (as indicated by “(s)”) unless aspects related to multiple of the features are being described.

[0070] As used herein, the terms “comprise,” “comprises,” and “comprising” may be used interchangeably with “include,” “includes,” or “including.” Additionally, the term “wherein” may be used interchangeably with “where.” As used herein, “exemplary” indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

[0071] The phrase “corresponds to” as used herein is a relational phrase indicating correspondence, equivalence, or matching. For example, if A corresponds to B, then A is B, there is a mapping between A and B, or A matches B. The phrase “is associated with” as used herein is a broad relational phrase indicating a looser or more general relationship such as, for example, a categorical relationship (e.g., A is part of or belongs to B), a causal relationship (e.g., A causes B), a logical relationship (e.g., If A then B), correlation (e.g., when A is present B is present), a structural relationship (e.g., the B that is coupled to A), and other possible relationships. Correspondence always includes association; whereas association can, but does not always, indicate correspondence. For example, if A is associated with B, there can be a mapping betw een A and B that could be described as correspondence betw een A and B.

[0072] As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) includeQUALCOMM Ref. No. 2501500WO- 23 / 98 -any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, "directly coupled” may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0073] In the present disclosure, terms such as ‘“obtaining,” “determining,” “calculating,” “estimating.” “shifting,” “adjusting,” etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “obtaining,” “generating,” “calculating,” “estimating,” “using.” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “‘obtaining,” “generating,” “‘calculating,” “estimating,” or “determining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, receiving, or accessing the parameter (or signal) that is already generated, such as by another component or device.

[0074] As used herein, “image data” and “pixel data” should be understood according to any of their usual and customary meanings in the fields of image processing, computer vision, photography, medical imaging, and / or artificial intelligence. For example, image data includes a digital representation of visual information, such as color, intensities, and spatial arrangement of portions of an image, and may also include metadata and / or auxiliary information, such as a time, a date, and / or a location associated with the image data; field of view information; camera pose information; camera intrinsic parameters; etc. The color, intensities, and spatial arrangement information of an image isQUALCOMM Ref. No. 2501500WO- 24 / 98 -represented in pixel data, where each pixel has a corresponding set of values representing color, intensity, and position within a field of view of the image.

[0075] FIG. 1 is a block diagram of an illustrative system 100 that is operable to update one or more gaussian scene representations, in accordance with some examples of the present disclosure. In FIG. 1, the system 100 includes a device 102 that includes a memory 120 configured to store one or more gaussian scene representations, such as illustrative gaussian scene representation 154, and one or more processors 130 configured to selectively update one or more of the gaussian scene representation(s) based on input data.

[0076] FIG. 1 also illustrates a schematic representation of a scene 132 that can be represented by the gaussian scene representation 154. The scene 132 can be a 3D scene or a 4D scene, where a fourth dimension of the scene 132 is time, such that the 4D scene models the 3D scene over time. FIG. 1 also illustrates a view 136 associated with the scene 132. For example, the view 136 represents what a camera 134 with a particular pose and field of view would capture of the scene 132.

[0077] The device 102 includes or is coupled to one or more sources 104 of input data. For example, the source(s) 104 illustrated in FIG. 1 include one or more image sources 106, one or more sensors 110, a modem 114, or a combination thereof. In other examples, the source(s) 104 can also, or alternatively, include other devices or systems, such as a memory device, a media player, a game system, etc. Although the image source(s) 106, the sensor(s) 110, and the modem 114 are illustrated outside the device 102 in FIG. 1, in other examples, the image source(s) 106, the sensor(s) 110, the modem 114, or a combination thereof, are integrated within the device 102.

[0078] The image source(s) 106 are configured to provide image data 108 to the device 102. For example, the image source(s) 106 can include or correspond to one or more still cameras, one or more video cameras, or both. As another example, the image source(s) 106 can include a memory that stores the image data 108 or a computer or similar device that generates the image data 108 by executing instructions, such as a game engine that generates images to represent a virtual reality, augmented reality, or mixed reality' scene. The image source(s) 106 can be controlled manually (e.g., based onQUALCOMM Ref. No. 2501500WO- 25 / 98 -user input) or automatically (e.g., based on sensor data 112, based on position information, based on a schedule, etc.).

[0079] The sensor(s) 110 are configured to provide sensor data 112 to the device 102. For example, the sensor(s) 110 can include a light sensor that is configured to generate light data representing lighting intensity or lighting changes within a scene. As another example, the sensor(s) 110 can include a motion sensor that is configured to generate motion data representing movement of the device 102, movement of another device, or movement within a portion of the scene 132. In some examples, the sensor(s) 110 can include one or more ranging sensor(s). such as sonar, lidar, or radar sensor(s). In some cases, the image data 108 can be processed to generate motion data, light data, or both.

[0080] The modem 114 is configured to provide data 116 received from a remote device to the device 102. The data 116 received via the modem 114 can include sensor data 112, image data 108, user input, mapping data 122, reference images 124, gaussians 156 of a gaussian scene representation 154, other data, or a combination thereof.

[0081] In a particular aspect, the gaussian scene representation 154 can be stored in the memory 120. The memory 120 can also include one or more buffer(s) 118 configured to store data received from the source(s) 104 for analysis and / or use in updating the gaussian scene representation 154. For example, the buffer(s) 118 can include an image buffer configured to store the image data 108, a sensor data buffer configured to store the sensor data 112, or both. The memory 120 may also be configured to store reference images 124. The reference images 124 can include one or more training images 126 (e.g., an image of the scene 132 used to train the gaussian scene representation 154), synthetic images 128 (e.g., an image of the scene 132 generated as output by the gaussian scene representation 154), or both. The reference images 124 can be used as a reference for a current state of a gaussian scene representation 154 for purposes of determining whether and / or how to update the gaussian scene representation 154 based on newly received input pixel data.

[0082] In FIG. 1, the processor(s) 130 include a selective model update pipeline 140, a Tenderer 158, a controller 160, and a view selector 162. FIG. 1 also shows the gaussian scene representation 154 in the processor(s) 130, which may be the case during certain operations. However, in other situations, such as during a partial update of the gaussianQUALCOMM Ref. No. 2501500WO- 26 / 98 -scene representation 154, only a subset of the gaussian scene representation 154 (e.g., selected gaussians 156) may be loaded to the processor(s) 130. Similarly, the view selector 162 and the Tenderer 158 may only be loaded to the processor(s) 130 for particular operations, such as rendering the view 136 or rendering views during a full update of the gaussian scene representation 154.

[0083] As explained further below, the selective model update pipeline 140 is configured to manage updating of the gaussian scene representation 154 based on input pixel data. The Tenderer 158 is configured to generate views (e.g., the view 136) of the scene 132 based on the gaussian scene representation 154 and based on view selection data from the view selector 162. The view selector 162 is configured to provide the gaussian scene representation 154, the Tenderer 158, or both, with the view selection data, where the view selection data indicates a location, a pose, a field of view, a focal length, other information specifying a portion of the scene 132 to be rendered, or a combination thereof. In the example of FIG. 1, the Tenderer 158 is coupled to a display 138 that is configured to display the view 136; however, in other cases, the view 136 can be stored or sent to another device for use. To illustrate, the view 136 can be sent via the modem 114 to a remote device for further processing or display.

[0084] The controller 160 is configured to coordinate tasks among various components of the system 100. For example, the controller 160 can cause one or more of the source(s) 104 to provide data to the processor(s) 130 and / or the memory 120 at particular times or intervals. As another example, various components of the system 100 can provide status information associated with their respective tasks to the controller 160, and the controller 160 can use the status information to initiate or control other components of system 100. For example, when the device 102 receives new image data 108 from the image source(s), the controller 160 can be notified and can initiate the selective model update pipeline 140 or a portion thereof in response to the new image data 108. As another example, when the update strategy selector 144 selects a particular update strategy, the controller 160 can cause components associated with the particular update strategy to be initiated. To illustrate, a full update 148 includes rendering of various views of the gaussian scene representation 154, whereas a partial update 150 may not entail rendering of views. Accordingly, when the update strategy selector 144QUALCOMM Ref. No. 2501500WO- 27 / 98 -selects the full update 148, the controller 160 can be notified and can cause the processor(s) 130 to load or initiate the view selector 162 and the Tenderer 158.

[0085] In the example illustrated in FIG. 1, the selective model update pipeline 140 includes one or more data analyzers (such as illustrated data analyzer 142), an update strategy selector 144, and update engines 146. In this example, the data analyzer 142 is configured to analyze input data, such as at least a portion of the image data 108, at least a portion of the sensor data 112, or other data, to generate one or more metrics, classification data, or other data indicative of estimated, expected, or detected changes in the scene 132. For example, changes in the scene 132 can be detected based on differences between input pixel data (e.g., at least a portion of the image data 108) and reference pixel data of the reference images 124. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and reference pixel data (e.g., based on a pixel-by-pixel comparison) and generate one or more difference metrics. A difference metric can indicate, for example, a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.

[0086] The update strategy selector 144 is configured to select an update strategy to be used (if any) based on one or more update criteria. For example, the update strategy' selector 144 can select from among a full update 148, one or more different types of partial updates (including a partial update 150 in FIG. 1), or no update 152 based on an update criterion (or based on more than one update criteria). If the partial update 150 is to be performed, the update strategy selector 144 may also select a portion of the input pixel data to be used to update a corresponding set of gaussians 156 of a gaussian scene representation 154.

[0087] In a particular aspect, the update strategy selector 144 selects an update strategy based on output of the data analyzer 142 (e.g., based on one or more of the difference metrics). Additionally, or alternatively, the update strategy selector 144 can select the update strategy based on a device condition associated with the device 102 or a component thereof. For example, the sensor data 112 can include device condition dataQUALCOMM Ref. No. 2501500WO- 28 / 98 -indicating a location of the device 102, a battery level of the device 102, whether the device 102 is charging, a temperature of the device 102. etc. Additionally, or alternatively, the processor(s) 130, the memory 120, or another component of the device 102 can provide device condition data indicating a level of usage of a computing resource of the device 102. To illustrate, the device condition data can include a processor utilization metric (e.g., a processor load), a memory utilization metric (e.g., memory usage, cache load, or memory access latency), a communication metric (e.g., a bus load, an available network bandwidth, or network latency), or a similar metric. Examples of a processor utilization metric include, without limitation, user processor time (e.g., time spent executing user-space processes), system processor time (e.g., time spent on kernel-level operations), idle time (e.g., percentage of time a processor 130 is not executing any tasks), input / output wait time, load average (e.g., average number of processes waiting to execute in a given time period), a count of context switches per second, a count of processor throttling events (e.g., due to overheating or power-saving measures), or a combination thereof.

[0088] For example, the update strategy selector 144 can be configured to compare one or more difference metrics from the data analyzer 142 to corresponding thresholds to select the update strategy. To illustrate, the update strategy selector 144 can compare a value of a difference metric based on the differences between the input pixel data and the reference pixel data to one or more thresholds to determine whether to perform the partial update 150, the full update 148, or no update 152. As anon-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of the difference metric is less than a first threshold, determine to perform the partial update 150 if the value of the difference metric is greater than the first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the difference metric is greater than the second threshold. As a second non-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of a size metric is less than a first threshold, determine to perform the partial update 150 if the value of the size metric is greater than a first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the size metric is greater than the second threshold. In this illustration, the size metric can be based on dimension of a difference region, a count of pixels of the difference region, aQUALCOMM Ref. No. 2501500WO- 29 / 98 -count of gaussians 156 of the gaussian scene representation 154 associated with the difference region (also referred to herein as “difference gaussians7’), or a combination thereof. For example, the size metric can be based on a count of the difference gaussians and a count of gaussians 156 of the gaussian scene representation 154.

[0089] When the update strategy selector 144 determines to perform the full update 148. the update engines 146 perform operations that are similar to initial creation of a gaussian scene representation except that the full update 148 is initialized from the existing gaussian scene representation 154. For example, during the full update 148, all of the gaussians 156 of the gaussian scene representation 154 can be processed from various views (as described further with reference to FIGS. 3A and 3B) to update their respective parameters. In this example, processing the gaussians 156 includes rendering selected views, comparing the rendered views to ground-truth image data (e.g., images of the scene) to determine loss values, and using the loss values to update the gaussians 156 (e.g.. using backpropagation and / or other model training techniques). The full update 148 can also include operations such as adaptive density control which can generate new gaussians 156 or eliminate or merge gaussians 156. Thus, the full update 148 is very computationally intensive.

[0090] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all of the gaussians 156) of the gaussian scene representation 154 are updated, as described further with reference to FIGS. 3A and 3C. For example, the update strategy selector 144 may indicate that a first subset of gaussians 156 are selected for update and a second subset of gaussians 156 of the gaussian scene representation 154 are not selected for update and are to be held constant. The update strategy selector 144 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and optionally a second subset of the input pixel data that is not to be used for the partial update 150.

[0091] In a particular aspect, the update strategy selector 144 selects the first subset of gaussians 156 to be updated and the first subset of the input pixel data used in the partial update 150 based on differences between the input pixel data and reference pixel data of the reference images 124, based on differences between successive sets of input pixel data, based on the sensor data 112, or a combination thereof. For example, differencesQUALCOMM Ref. No. 2501500WO- 30 / 98 -between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, differences between sequential images of the scene 132 can be used to identity motion of one or more objects in the scene 132, lighting changes in one or more portions of the scene 132, or both, which may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the input pixel data to the reference pixel data. As another example, the sensor data can be used to estimate the difference region. To illustrate, changes in lighting, as detected by a light sensor, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by a motion sensor and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data.

[0092] The update strategy selector 144 can use the mapping data 122 (e.g., pixel -to-gaussian mapping data) to identify a subset of the gaussians 156 associated with pixels of the difference region. In this example, the pixels of the difference region can be used for the partial update 150, and the subset of the gaussians 156 associated with pixels of the difference region can be updated during the partial update 150. Other gaussians 156 of the gaussian scene representation 154 are not modified during the partial update 150.

[0093] It is likely that there will not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, a particular gaussian 156 may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g., the view 136). Thus, in some cases, the update strategy selector 144 is also configured to use the mapping data 122 (e.g., gaussian-to-pixel mapping data) to identify pixels that are influenced by the subset of the gaussians 156 associated with pixels of the difference region. In such cases, the pixels of the input pixel data that correspond to the pixels that are influenced by the subset of the gaussians 156 are included in the training data used for the partial update 150. Updating a gaussian in this context refers to changing one or more parameters of the gaussian.QUALCOMM Ref. No. 2501500WO- 31 / 98 -

[0094] In a particular aspect, the update strategy selector 144 is configured to generate and / or update the mapping data 122. For example, the update strategy’ selector 144 can use saliency mapping to estimate which gaussians 156 most affect particular output pixel data from the gaussian scene representation 154 to generate pixel -to-gaussian mapping data and / or can use saliency mapping to estimate which pixels of input pixel data are most important to training of particular gaussians 156 of the gaussian scene representation 154 to generate gaussian-to-pixel mapping data.

[0095] Saliency mapping generally involves at least one forward pass through rendering operations (e.g., projection and rasterization) to generate gradient values used to determine the pixel-to-gaussian mapping data and at least one backward pass through the rendering operations to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially if the gaussian scene representation 154 is large and / or updated frequently. To reduce the computational demand associated with running the gaussian scene representation 154 to generate mapping data 122, the selective model update pipeline 140 can use a staged approach to manage the mapping data 122. In the staged approach, after mapping data 122 (e.g., pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, the selective model update pipeline 140 stores the mapping data 122 in a manner that facilitates reuse of the mapping data 122. For example, after mapping data associated with a particular region of interest (e.g., a range of pixels, an object, or some other identifiable portion of the input pixel data) is generated, the mapping data 122 can be stored along with information identifying the region of interest in a searchable data store. As one example, the region of interest can be associated with an object or a portion of an object in the scene 132. In this example, the mapping data 122 can include information that maps the object (or the portion of the object) to gaussians that represent the object. The mapping data 122 can also, or alternatively, include pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both, associated with the object. In this example, object detection (which uses fewer computing resources than saliency mapping) can be used to process input pixel data, and if an object detected in the region of interest is represented in the mapping data 122 the mapping data entry associated with the object can be retrieved from the mapping data 122. In contrast, if the object isQUALCOMM Ref. No. 2501500WO- 32 / 98 -not represented in the mapping data 122, a mapping data generation process (e.g., saliency mapping) can be performed.

[0096] In some cases, the update strategy selector 144 can use a staged approach with multiple data stores that each include mapping data 122, where a first stage includes a set of the most frequently used mapping data 122. a second stage includes a set of next most frequently used mapping data 122, and so forth. In such cases, full mapping data generation is performed if none of the searchable data stores includes the desired mapping data.

[0097] In a particular aspect, the update strategy selector 144 is configured to select an update strategy based, at least in part, on a device condition associated with the device 102. In some examples, the update strategy selection decision is based on the device condition and is independent of the magnitude or type of differences detected or expected in the input pixel data. For example, irrespective of the update strategy the update strategy selector 144 selects based on the detected or expected differences in the input pixel data, the update strategy selector 144 can select a different update strategy based on the device condition. In this example, the update strategy selected based on the device condition prevails or sets an upper bound on the type of update actually performed. To illustrate, if the device condition indicates that usage of computational resources is relatively low (e.g., below a first threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the full update 148, the partial update 150, or no update 152. However, if the device condition indicates that usage of computational resources is relatively high (e.g., above the first threshold and below a second threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the partial update 150 or no update 152. Further, if the device condition indicates that usage of computational resources is very’ high (e.g., above the second threshold), the update strategy selector 144 selects no update 152 irrespective of the detected or expected differences in the input pixel data.

[0098] In a particular aspect, results of operations performed by the device 102 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation 154 in which one or more parameters of one or more of the gaussians 156 has beenQUALCOMM Ref. No. 2501500WO- 33 / 98 -modified), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views 136 generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with the scene 132. For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of the device 102 or another device, such as to turn on or turn off a display based on an aspect of the scene 132. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory' or haptic notification or a combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene 132. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stopQUALCOMM Ref. No. 2501500WO- 34 / 98 -playback of media content in order to allow a user to focus on information in the scene 132.

[0099] FIG. 2 is a block diagram of an illustrative system 200 that is operable to update one or more gaussian scene representations 154. in accordance with some examples of the present disclosure. Various components, features, and / or elements of the system 200 can be stored in the memory 120 of FIG. 1 as data or instructions, can be performed by the processors) 130 of FIG. 1, or both. Additionally, various components of the system 200 correspond to components of the system 100 of FIG. 1, and can be configured as described with reference to FIG. 1. For example, the system 200 includes the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, and the memory 120, one or more of which can be configured as described with reference to FIG. 1.

[0100] In the example illustrated in FIG. 2, the system 200 includes one or more buses, such as illustrated bus 216, coupled to the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, the memory 120, one or more other components or devices, or a combination thereof, via respective bus interfaces 131, 133, 135, 137, 139, 141, and 143. For example, the bus 216 can enable communication of data or signals (e.g., control signals, status signals, etc.), between and among the image source(s) 106. the modem 114, the sensors 110, the processor(s) 130, and the memory 120. The sensors 110 can include one or more motion sensors 210, one or more light sensors 212, one or more device sensors 214, other sensors, or a combination thereof.

[0101] In FIG. 2, the processor(s) 130 include the selective model update pipeline 140, which in this example, includes the data analyzer 142, the update strategy selector 144, the update engines 146, an update data selector 224, and a mapper 222. The processor(s) 130 optionally also include the controller 160. the Tenderer 158. or both.

[0102] The image source(s) 106 are configured to generate (e.g., capture), receive, or otherwise obtain image data 147 representing a scene. As an example, the image source(s) 106 can include one or more cameras, a memory’ storing images, etc. In a particular aspect, the image source(s) 106 can provide image data 147 to the bus 216 via the bus interface 131 for communication to the processor(s) 130 (e.g., via the bus interface 143), to the memoiy 120 (e.g., via the bus interface 141), or both. In FIG. 2, the image data 147 is illustrated as provided to the data analyzer 142; however, in someQUALCOMM Ref. No. 2501500WO- 35 / 98 -cases, the image data 147 can be provided to other components of the selective model update pipeline 140 in addition to. or instead of. the data analyzer 142. In some cases, the controller 160 can provide control signals to the image source(s) 106 (e.g., via the bus 216 and the bus interfaces 143, 131) to control when images are captured or provided to the memory 120 and / or the processor(s) 130. When the image source(s) 106 provide the image data 147 to the memory 120, the image data 147 can be stored in an image buffer (e.g., one of the buffer(s) 118 of FIG. 1).

[0103] The sensor(s) 110 are optional, and if present, are configured to generate sensor data 145. For example, the motion sensor(s) 210 can generate motion data indicating movement of a camera of the image source(s) 106 relative to the scene or movement of an object in the scene. As another example, the light sensor(s) 212 can generate light data indicative of changes in lighting of a portion of the scene. As yet another example, the device sensor(s) 214 can generate device condition data. The device condition data can indicate a condition associated with a device that includes the selective model update pipeline 140. To illustrate, the device condition data can indicate a battery level, whether the device is connected to an external power source, a temperature of the device, a level of usage of a computing resource of the device (such as a processor utilization metric or a memory utilization metric), or other similar device condition information. In FIG. 2, the sensor data 145 is illustrated as provided to the data analyzer 142; however, in some cases, the sensor data 145 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142.

[0104] In a particular aspect, the data analyzer 142 is configured to determine differences between input pixel data of the image data 147 and reference pixel data (e.g., pixel data of one or more of the reference images 124 of FIG. 1) and to provide information related to such differences to the update strategy selector 144 and optionally to an update data selector 224. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and the reference pixel data based on a pixel-by-pixel comparison and / or using optical flow methods. As another example, the data analyzer 142 can be configured to estimate or predict differences between the input pixel data and the reference pixel data based on motion indicated in motion data, based on lighting changed indicated in light data, based on differences between earlierQUALCOMM Ref. No. 2501500WO- 36 / 98 -and later-acquired sets of the input pixel data (e.g., corresponding to sequential images in the image data 147). or a combination thereof.

[0105] The data analyzer 142 may be configured to generate one or more difference metrics associated with the differences between the input pixel data and the reference pixel data. For example, the difference metrics can indicate a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.

[0106] The update strategy selector 144 is configured to select an update strategy based on output of the data analyzer 142. Thus, the update strategy selector 144 can select the update strategy based, at least in part, on the differences between input pixel data and the reference pixel data. For example, based at least in part on data provided by the data analyzer 142, the update strategy selector 144 can signal to the update engines 146 to perform no update 152, the partial update 150, or the full update 148.

[0107] When the update strategy selector 144 indicates that the partial update 150 is to be performed, the update data selector 224 can identify which pixels of the input pixel data are to be used for the partial update 150, which gaussians 156 of the gaussian scene representation 154 are to be updated, or both. For example, the update data selector 224 can use the mapping data 122 to identify a subset of the gaussians 156 that correspond to (e.g.. influence rendering ol) pixels of a difference region. The pixels of the difference region can correspond to or include pixels of a region of interest, in which case pixel-to-gaussian mapping data 274 can be used to identify7the subset of the gaussians 156. Alternatively, the pixels of the difference region can correspond to or include pixels associated with a particular object, in which case object mapping data 270 can be used to identify the subset of the gaussians 156.

[0108] The update data selector 224 can also, or alternatively, use gaussian-to-pixel mapping data 276 of the mapping data 122 to identify pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156. Because each gaussian 156 can influence rendering of more than one pixel, the identified subset of gaussians can influence more pixels than the pixels used to identify the subset of gaussians. Thus, for example, the pixels of the input pixel data that are to be used toQUALCOMM Ref. No. 2501500WO- 37 / 98 -update an identified subset of the gaussians 156 can include pixels within the region of interest and one or more nearby pixels that are also influenced by the identified subset of the gaussians 156.

[0109] In some cases, the selective model update pipeline 140 includes a mapper 222 that is configured to generate or control the mapping data 122. For example, the mapper 222 can perform operations to generate the mapping data 122 (or portions thereof), to update the mapping data 122, to determine whether new mapping data 122 is to be generated, etc. As a particular example, the mapper 222 can perform operations associated with the staged approach to managing the mapping data 122 described with reference to FIG. 1.

[0110] In some embodiments, the update strategy selector 144 is configured to select the update strategy based on the differences between input pixel data and the reference pixel data and additional information. For example, the data analyzer 142 can analyze the sensor data 145 and provide analysis results to the update strategy selector 144. In this example, the update strategy' selector 144 can select the update strategy' based, at least in part, on the sensor data 145.[OHl] As one example, the update strategy selector 144 may select to perform no update 152 when the differences between input pixel data and the reference pixel data are insignificant (e.g., less than a first threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are insignificant, and / or when the sensor data 145 indicate that computing resources are strained (e.g., battery level is low, available processing capacity7is low, etc.). Continuing this example, the update strategy selector 144 may select to perform the full update 148 when the differences between input pixel data and the reference pixel data are large (e.g., greater than a second threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are significant, and / or when the sensor data 145 indicate that significant computing resources are available. Further, the update strategy selector 144 may select to perform the partial update 150 when the differences between the input pixel data and the reference pixel data are constrained to one or more identified regions of interest (based on the image data 147, the sensor data 145, or both)QUALCOMM Ref. No. 2501500WO- 38 / 98 -and / or when the sensor data 145 indicate that computing resources are available but may be constrained.

[0112] Although the example above describes selecting an update strategy from among no update 152, the partial update 150, and the full update 148 based on relative states (e.g., significant, insignificant, large, constrained, etc.), in practice, the selection can be based on comparisons of various metrics to respective thresholds. For example, an “insignificant” difference or value can correspond to a value of a metric that is less than a first threshold, a “large” difference or value can correspond to a value of the metric that is greater than a second threshold, and a “significant” difference can correspond to a value of the metric between the first threshold and the second threshold. In some cases, more than two or fewer than two thresholds can be used, depending on the specific implementation.

[0113] When the update strategy selector 144 determines to perform the full update 148, all of the gaussians 156 of the gaussian scene representation 154 are eligible to be updated based on the input pixel data, and all of the input pixel data is used to perform the full update 148. For example, the update strategy selector 144 can provide the input pixel data to the update engines 146. and the update engines 146 can render views of the gaussian scene representation 154, determine loss function values based on comparison of the rendered views to ground-truth views, use the loss function values to calculate gradient values, and use the gradient values to determine updated parameter values 149 for each of the gaussians 156.

[0114] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all) of the gaussian scene representation 154 are updated based on the input pixel data. For example, the update data selector 224 can select a first subset of gaussians 156 that are to be modified, in which case a second subset of gaussians 156 are not to be modified and are to be held constant. The update data selector 224 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and a second subset of the input pixel data that are not to be used for the partial update 150.

[0115] In a particular aspect, the update data selector 224 selects the first subset of the gaussians 156 to be updated and the first subset of the input pixel data used for theQUALCOMM Ref. No. 2501500WO- 39 / 98 -partial update 150 based on differences between the input pixel data and reference pixel data, based on the sensor data 145. or both. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data 145 can be used to estimate the difference region. To illustrate, changes in lighting, as detected by the light sensor(s) 212, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by the motion sensor(s) 210 and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data.

[0116] The update data selector 224 can select the first subset of the gaussians 156 to be updated based on the pixels of the input pixel data to be used for the partial update 150 (e.g., based on the difference region). For example, the update data selector 224 can designate a subset of the gaussians 156 that correspond to or affect the pixels to be used for the partial update 150 as eligible to be modified by the partial update 150. In this example, other gaussians 156 of the gaussian scene representation 154 are ineligible to be modified during the partial update 150.

[0117] In many cases, there may not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, particular gaussians may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g., the view 136). Thus, in some cases, the update data selector 224 can use the difference region to identify’ the particular gaussians associated w ith pixels of the difference region. In such cases, the particular gaussians can be updated based on a selected set of pixels that include pixels of the difference region and additional pixels (typically near the difference region) that are influenced by the particular gaussians. In a particular aspect, the update data selector 224 (or the mapper 222) can use saliency mapping, look-up tables, or other mapping data to identify associations between pixels and gaussians of the gaussian scene representation 154.QUALCOMM Ref. No. 2501500WO- 40 / 98 -

[0118] In a particular aspect, results of operations performed by the system 200 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation with updated gaussians 156), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with a scene (e.g., the scene 132 of FIG. 1). For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of a device (e.g., the device 102 or another device), such as to turn on or turn off a display based on an aspect of the scene. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory or haptic notification or combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing orQUALCOMM Ref. No. 2501500WO- 41 / 98 -suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene.

[0119] FIGS. 3A, 3B, and 3C, together, depict a system 300 that illustrates particular aspects of the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The system 300 illustrates additional details related to updating a gaussian scene representation 154, and in particular, illustrates differences between operations associated with performing a full update (e.g., the full update 148 of FIGS. 1 and 2) and a partial update (e.g.. the partial update 150 of FIGS. 1 and 2). Due to the complexity of the system 300, the system 300 is split over three pages and linked by off-page references.

[0120] The system 300 includes one or more sensors 302. such as a first sensor 302A, a second sensor 302B, and a Pth sensor 302P (where P is an integer greater than two). At least one of the sensor(s) 302 is an image sensor (e.g., a camera) configured to generate image data. One or more others of the sensor(s) 302 can include or correspond to motion sensors, light sensors, device condition sensors, etc., such as described with reference to FIGS. 1 and 2. The sensor(s) 302 are configured to generate sensor data 303. The system 300 optionally includes a sensor controller 304 that is configured to control operation of the sensor(s) 302 and / or to select a subset of the sensor data 303 to generate sensor data 305, which is used for further processing. If the sensor controller 304 is omitted, the sensor(s) 302 can be externally controlled (e.g., responsive to user input), can operate according to a schedule, or can generate the sensor data 303 responsive to detecting a condition (e.g., in response to detecting movement). If the sensor controller 304 is omitted, the sensor data 305 is the same as the sensor data 303.

[0121] In FIG. 3A, the system 300 includes a pose estimator 306 that is configured to determine pose data (e.g., a position and / or orientation of one of the sensor(s) 302 within a scene), which can be provided along with the sensor data 305 as input 307 to an image subspace selector 312. In some cases, the sensor data 305 can include pose information (e.g., as data or metadata). In such cases, the pose estimator 306 is optional. If the pose estimator 306 is omitted, the sensor controller 304 (or the sensor(s) 302 if theQUALCOMM Ref. No. 2501500WO- 42 / 98 -sensor controller 304 is also omitted), provide the input 307 including the sensor data 305 and corresponding pose data to the image subspace selector 312.

[0122] During initial generation of the gaussian scene representation 154, the pose estimator 306, the sensor controller 304, or the sensor(s) 302 provide input 311 including sensor data 305 and corresponding pose data to a model initializer 310. which uses similar operations as described with reference to FIG. 3B for the full update 148 to generate initial parameters for the gaussians 156 of the gaussian scene representation 154. After initial generation of the gaussian scene representation 154, the model initializer 310 can be bypassed, and the input 307 can be provided to an image subspace selector 312.

[0123] In the example illustrated in FIG. 3 A, the gaussian scene representation 154 includes a plurality of gaussians 156, such as: a gaussian 156A, a gaussian 156B. and a gaussian 156Q (where Q is an integer greater than 2). Each gaussian 156 is characterized by a plurality of parameters describing various properties of the gaussian 156. For example, in FIG. 3 A, the gaussian 156A is characterized by an X-axis center position coordinate (‘‘center pos. (X)”) 180, Y-axis center position coordinate (“center pos. (Y)”) 181. and Z-axis center position coordinate (“center pos. (Z)”) 182. Together, the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 indicate a center position of the gaussian 156A and correspond to the position vector, p,, discussed above with reference to Equations 1-10. In FIG. 3 A, the gaussian 156A is also characterized by an X-axis scale parameter 183, a Y-axis scale parameter 184, and a Z-axis scale parameter 185.Together the X-axis scale parameter 183, the Y-axis scale parameter 184, and the Z-axis scale parameter 185 indicate a size and shape of the ellipsoid of the gaussian 156A and correspond to the scale vector, s;, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO) 186, rotation (ql) 187, rotation (q2) 188, rotation (q3) 189) specified as quaternions in the example illustrated in FIG. 3A. Together the rotation parameters 186-189 indicate a spatial orientation of the ellipsoid of the gaussian 156A and correspond to the orientation parameters, qz, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by an opacity parameter 190, which indicates a maximum opacity of the gaussian 156A. The opacity parameter 190QUALCOMM Ref. No. 2501500WO- 43 / 98 -corresponds to the opacity parameter,discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by a set of color parameters, e.g., color parameters 191, 192, and 193 in FIG. 3 A. The color parameter(s) 191-193 can be specified as a set of spherical harmonics (SH) coefficients, H as discussed above with reference to Equations 1-10. Although FIG. 3A illustrates the gaussian 156A as characterized by a set of fourteen parameters, in other examples, the gaussians 156 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters 191-193 can be used, depending on the order of the SH coefficients used to represent the colors of the gaussians 156. To illustrate, in some implementations, a reduced set of approximately 14 parameters may be used, where the color parameters 191-193 are simplified to a single base color value (e.g., an RGB triplet) rather than a full set of spherical harmonics coefficients. In such implementations, the spherical harmonics coefficients may be omitted entirely, reducing memory and computational requirements for storing and updating the gaussians 156.

[0124] The image subspace selector 312 is configured to, based on the input 307, identify one or more patches of pixels (e.g., image patches) for a region of interest (ROI), or more than one ROI, associated with the sensor data 305 to generate ROI data 313. In aparticular aspect, the image subspace selector 312 can correspond to, include, or be included within the data analyzer 142 of FIGS. 1 and 2. For example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to an object that is in motion in the scene (e.g.. based on motion data from a motion sensor, based on optical flow data from two or more images, etc.). In this example, the ROI can correspond to the image patch (or patches) associated with the object that is in motion. As another example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to pixels of input pixel data of the sensor data 303 or 305 that differ (by at least a threshold amount) from pixels of reference pixel data. In this example, the ROI can be defined by the location of the image patch (or patches) associated with the pixels that differ between the input pixel data and the reference pixel data. The ROI data 313 identifies one or more sets of pixels of one or more images.QUALCOMM Ref. No. 2501500WO- 44 / 98 -

[0125] The update strategy selector 144 of the system 300 is configured to select an update strategy based on strategy selection data 320. The strategy selection data 320 can include the sensor data 303 or 305, the input 307, the ROI data 313, other data, or a combination thereof. For example, the update strategy' selector 144 can select the update strategy' using any of the techniques described with reference to FIGS. 1 and 2 and / or any of the techniques described with reference to FIGS. 4-8. The update strategy selector 144 is configured to provide a selection indication 321 to the update engines 146, where the selection indication 321 causes the update engines 146 to initiate operations associated with the selected update strategy'. For example, if the selection indication 321 indicates that a full update strategy is selected, the update engines 146 initiate the full update 148. and if the selection indication 321 indicates that a partial update strategy is selected, the update engines 146 initiate the partial update 150. If no update is selected, the update strategy selector 144 can omit generation of the selection indication 321 or can generate a “no update"’ selection indication.

[0126] When the update engines 146 select the full update 148, the update engines 146 cause a pose selector 324 of FIG. 3B (as indicated by off-page reference A) to begin operations associated with the full update 148. The pose selector 324 is configured to select a pose 325 from training data 360 that includes a set of training images (e.g.. images 361 A-361K) and pose data (e.g., poses 362A-362K) associated with each image 361 A-361K (where K is an integer greater than two). An image associated with the selected pose 325 is used as a ground-truth image 349 for loss calculation after a view associated with the pose 325 is rendered.

[0127] The pose 325 describes the position and orientation of a virtual camera 330 relative to the gaussians 156 of the gaussian scene representation 154 to be rendered as a view. Due to the pose 325 and intrinsic parameters (e.g., focal length, principal point, etc.) of the virtual camera 330, some of the gaussians (e.g., gaussians 331) are visible to the virtual camera 330, and some of the gaussians (e.g., gaussians 333) are not visible to the virtual camera 330. For example, for a pose 325A, a virtual camera 330A can see gaussians 331 A; however, gaussians 333A are not visible to the virtual camera 330A in the pose 325 A. As another example, for a pose 325K, a virtual camera 330K can see gaussians 331K; however, gaussians 333K are not visible to the virtual camera 330K in the pose 325K.QUALCOMM Ref. No. 2501500WO- 45 / 98 -

[0128] To ensure that the full update 148 updates each gaussian 156 based on all relevant data, the full update 148 generally entails rendering a view for every available pose (e g., poses 362A to 362K) and comparing each rendered view to the ground-truth image (e.g., each of images 361 A to 361K) associated with the pose 362 used to render the view. Thus, performing the full update 148 is similar to operations performed by the model initializer 310 to generate an initial version of the gaussian scene representation 154 based on a set of images except that the full update 148 is initialized from a preexisting gaussian scene representation 154.

[0129] A projector 326 identifies the gaussians 331 that are visible to the virtual camera 330 for a pose 325. The projector 326 performs gaussian splatting, which entails projecting the visible gaussians 331, which are three dimensional, onto a 2D image plane 332 based on the pose 325 of the virtual camera 330 and the intrinsic parameters of the virtual camera 330. Each gaussian 331 is represented as an ellipse 335 in the 2D image plane 332. The ellipses 335 can overlap one another and can have various sizes and orientations depending on parameters of the gaussians 331 that are projected and the pose 325 of the virtual camera 330. For example, the gaussians 331 A that are visible to the virtual camera 330A in the pose 325A are projected on to a 2D image plane 332A as ellipses 335A, and the gaussians 331K that are visible to the virtual camera 330K in the pose 325K are projected on to a 2D image plane 332K as ellipses 335K.

[0130] Data 334 describing the ellipses 335 on the 2D image plane 332 is provided to a differentiable tile rasterizer 340. For example, data 334A describes the ellipses 335A associated with the pose 325A, and data 334K describes the ellipses 335K associated with the pose 325K. The differentiable tile rasterizer 340 is configured to perform operations to blend the ellipses 335 to generate pixel data representing a 2D image that would be captured by the virtual camera 330 for the particular selected pose 325. To generate the pixel data, the differentiable tile rasterizer 340 divides the 2D image plane 332 into tiles (represented in FIG. 3B by bold lines and different shading in the 2D image planes 332A and 332K of the differentiable tile rasterizer 340). For purposes of this description, each tile is treated as corresponding to a single pixel: however, the exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel. Additionally, to simplify the drawings, in FIG. 3B, the 2D image plane 332 includes some regions that are notQUALCOMM Ref. No. 2501500WO- 46 / 98 -associated with any gaussian; however, it will generally be the case that each tile is associated with at least one gaussian 156 to provide color information for the tile.

[0131] The differentiable tile rasterizer 340 obtains depth information 342 for the visible gaussians 331 of the pose 325, where depth refers to the distance in the scene of a gaussian from the virtual camera 330. For example, the depth information 342 can be calculated for each of the visible gaussians 331 based on the location of the virtual camera 330 in the scene (as indicated by data describing the pose 325) and the location of the center of the gaussian in the scene (e.g., as indicated by the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 of each gaussian). For example, depth information 342A indicates distances in the scene between each of the gaussians 331 A and the virtual camera 330A, and depth information 342K indicates distances in the scene between each of the gaussians 331K and the virtual camera 330K.

[0132] The differentiable tile rasterizer 340 can perform tile-depth association 344 to, for each tile of the 2D image plane 332, sort the ellipses 335 that at least partially overlap the tile based on the depth information 342. Since the gaussians 156 are generally not opaque, a first gaussian that is more distant from the virtual camera 330 in the scene can be partially visible through a second gaussian that is closer to the virtual camera 330. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 340 accounts for the colors of each gaussian of a set of gaussians that influence the particular pixel, the opacities of the gaussians of the set, and the order of the gaussians of the set as viewed from the virtual camera 330.

[0133] To parallelize generation of the pixel data, the differentiable tile rasterizer 340 can perform thread association 346 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources, such computing resources can include one or more cores of one or more central processing unit (CPU), one or more cores of one or more graphics processing unit (GPU), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colorsQUALCOMM Ref. No. 2501500WO- 47 / 98 -of the gaussians (as determined based on the pose 325 of the virtual camera 330 and color parameters specified by spherical harmonic color coefficients, opacity of the gaussians, and the depth information 342).

[0134] Blending the contributions of the gaussians results in pixel data 341 representing an image (e.g., a view) captured by the virtual camera 330 for the particular pose 325. The pixel data 341 is provided to a loss calculator 348, which is configured to compare the pixel data 341 to the ground-truth image 349 corresponding to the pose 325. The loss calculator 348 determines a loss value for each pixel (e g., a photometric L2 loss) to quantify how the pixel data 341 differs from the ground-truth image 349. Gradient values 343, 345 based on the loss values are backpropagated through the differentiable tile rasterizer 340 and the projector 326 to estimate updated parameter values 347 for the visible gaussians 331 of the pose 325.

[0135] The full update 148 continues by iteratively repeating the processes of pose selection, projection, rasterization, loss calculation, and backpropagation of gradients until all of the poses 362 have been processed. At one or more points during the full update 148, an adaptive density controller 350 performs operations that change the number of gaussians 156 of the gaussian scene representation 154, such as pruning one or more gaussians 156 (e.g., the gaussian 156B) or splitting one or more gaussians 156. Thus, in some senses, a different gaussian scene representation 154 exists after the full update 148 than existed before the full update 148 since a different number and arrangement of gaussians 156 can be present, and any gaussian 156 that is carried over can have different parameters. For example, in FIG. 3B, the gaussian scene representation 154 is illustrated as including a first gaussian 156A, a second gaussian 156B, and one or more additional gaussians up to a Qth gaussian 156Q (where Q is an integer greater than 2). After the full update 148, the gaussian scene representation 154 can include more than Q or fewer than Q gaussians 156. Further, the full update 148 is a feedback-based process since, during each iteration, the visible gaussians 331 for a pose 325 are updated based on loss values indicating differences between an image (e.g., the pixel data 341) of a view rendered for the pose 325 and a reference image (e.g., the ground-truth image 349 associated with the pose 325).QUALCOMM Ref. No. 2501500WO- 48 / 98 -

[0136] When the update engines 146 of FIG. 3 A select the partial update 150, the update engines 146 cause a gaussian selector 354 of FIG. 3C (as indicated by off-page reference B) to select a set of gaussians (e.g., gaussians 156A and 156B in the example of FIG. 3C) that are to be updated based on the ROI data 313 and mapping data 122. As an example, for illustrative purposes, the sensor data 303 of FIG. 3 A can include an image 351 of FIG. 3C. In this illustrative example, the image subspace selector 312 can determine that the ROI includes a ball 365 depicted in the image 351, in which case the ROI data 313 can include data identifying the ball 3 5. The data identifying the ball 365 can include object identification data (e.g., a text label or a numerical designator), a set of pixels that represent the ball 365 in the image 351, identifiers of the set of pixels that represents the ball 365 in the image 351, or other data that enables the gaussian selector 354 to select the set of gaussians based on the mapping data 122 and the ROI data 313. In a particular aspect, the image subspace selector 312, the gaussian selector 354, or both, are integrated within the update data selector 224 of FIG. 2.

[0137] The gaussian selector 354 uses the ROI data 313 and the mapping data 122 to generate data 353 identifying the selected set of gaussians, and provides the data 353 to a partial updater 356. The data 353 identify ing the selected set of gaussians can include the gaussians themselves, memory addresses of the selected set of gaussians, indices of the selected set of gaussians, or other information that enables the partial updater 356 to obtain the selected set of gaussians.

[0138] The partial updater 356 is configured to determine updates 357 to the selected set of gaussians (leaving remaining gaussians 156 of the gaussian scene representation 154 unchanged) to generate updated gaussians 156’. In some implementations, the partial updater 356 can use a feedforward approach to determine the updates 357. For example, the partial updater 356 can determine the updates 357 based on the sensor data 305 of FIG. 3 A. As one example, the sensor data 303 can include motion data that can be used to determine how far an object (e.g., the ball 365) in the scene moved, and the partial updater 356 can determine the updates 357 based on the determined movement of the object. As another example, optical flow can be used to compare the image 351 to one or more reference images to identify movement of one or more objects in the scene, and the partial updater 356 can determine the updates 357 based on the identified movement. As yet another example, the sensor data 303 can include light data whichQUALCOMM Ref. No. 2501500WO- 49 / 98 -can be used to determine changes in lighting or color in a portion of the scene, and the partial updater 356 can determine the updates 357 based on the changes in lighting or color.

[0139] The partial updater 356 can apply the updates 357 to the selected set of gaussians and store updated versions of the selected set of gaussians (e.g., updated gaussians 156A’ and 156B’ in FIG. 3C) in an update buffer 358. After all gaussians of the selected set of gaussians have been updated, the updated gaussians 156' can be written to a memory where the gaussian scene representation 154 is stored.

[0140] Optionally, in some embodiments, a rendering-based error calculator 359 can be used to check the updated gaussians 156’ before the gaussian scene representation 154 is modified. The rendering-based error calculator 359 is configured to perform projection, rendering, and error calculation, as described with reference to the projector 326, the differentiable tile rasterizer 340, and the loss calculator 348, respectively, of FIG. 3B to generate a rendered view of the gaussian scene representation 154. During the partial update 150, the rendering-based error calculator 359 can determine (e.g., obtain from the sensor data 305 or from the pose estimator 306 of FIG. 3 A) a pose associated with the image 351. The rendering-based error calculator 359 can render a view of the gaussian scene representation 154 (with the updated gaussians 156’) from the pose associated with the image 351, and compare the rendered view to the image 351 to determine whether the updated gaussians 156’ are acceptable or whether additional updates 357 should be determined. For example, the rendering-based error calculator 359 can determine loss value(s) (e.g., using similar operations as described with reference to the loss calculator 348 of FIG. 3B) based on differences between the image 351 and the rendered view. In this example, if the loss value(s) satisfy a threshold, the updated gaussians 156’ are used to update the gaussian scene representation 154. However, if the loss value(s) do not satisfy a threshold, the partial updater 356 generates a second set of updated gaussians 156’ and repeats the process above to determine whether loss value(s) for the second set of updated gaussians 156’ satisfy the threshold. This process can be repeated iteratively until a termination condition is reached. The termination condition can include identification of a set of updated gaussians 156’ that satisfies the threshold, or performance of a specified number of iterations. If no set of updated gaussians 156’ that satisfies the threshold isQUALCOMM Ref. No. 2501500WO- 50 / 98 -identified after the specified number of iterations, the partial updater 356 can change to another approach to generate updates 357. or the update strategy selector 144 of FIG.3A can select a different update strategy (e.g., no update, a different type of partial update, or the full update 148). In some examples, rather than process one set of updated gaussians 156’ at a time, as described above, the partial updater 356 can generate multiple candidate sets of updated gaussians 156’ in parallel, and check which, if any, best match the image 351.

[0141] As one specific example, in FIG. 3C, the image 351 represents a portion of a soccer pitch and includes a goal and a ball 365. FIG. 3C also illustrates an example 370A of the gaussian scene representation 154 before the partial update, and an example 370B of the gaussian scene representation 154 after the partial update.

[0142] In this example, the ROI data 313 can include object data identifying an object (e.g., the ball 365) that has changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the object mapping data 270 of FIG. 2, which identifies a set of the gaussians 156 that represent the object, and the gaussian selector 354 can select the set of gaussians that represent the object as the set of gaussians that are to be updated. In the specific example shown in FIG. 3C. the gaussians 156 A and 156B represent a ball 367 in the scene; thus, the data 353 includes the gaussians 156A and 156B.

[0143] As another example, the ROI data 313 can identify a first set of pixels that have changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the pixel-to-gaussian mapping data 274 of FIG. 2, which identifies a set of the gaussians 156 that correspond to the first set of pixels, and the gaussian selector 354 can select the set of gaussians that correspond to the first set of pixels as the set of gaussians that are to be updated. The selection data 353 can include the selected set of gaussians or information indicating how the partial updater 356 can access the selected set of gaussians, such as memory addresses of the selected set of gaussians, indices of the selected set of gaussians, etc.

[0144] The partial updater 356 is configured to modify parameters of the selected set of gaussians (e.g., the gaussians 156A and 156B in this example) based on the sensor data 305, the input 307, and optionally other data. For example, when the ROI data 313QUALCOMM Ref. No. 2501500WO- 51 / 98 -indicates that an object in the scene has moved, the partial updater 356 can change location parameters and / or orientation parameters of the gaussians 156A and 156B that represent the object in the gaussian scene representation 154 in order to update the gaussian scene representation 154 to reflect movement of the object. In other cases, the partial updater 356 can change color parameters (e.g., spherical harmonics coefficients) and / or opacity parameters of the gaussians 156A and 156B to represent lighting changes in the scene.

[0145] A technical benefit of performing the partial update 150 rather than the full update 148 is that significant computer resources can be conserved. For example, when the update to be performed can be calculated or estimated based on the sensor data 305, the rendering operations associated with the projector 326 and the differentiable tile rasterizer 340, the loss calculations performed by the loss calculator 348, and backpropagation of gradient values 343, 345 based on the loss values can be omitted during the partial update 150. Further, even if the partial update 150 uses the renderingbased error calculator 359 to check one or more candidate updated gaussians 156’, only one or a few views are rendered, rather than views associated with all of the images 361 of training data 360 of FIG. 3B, saving significant rendering time and resources.Further, backpropagation of gradients based on the loss values can be omitted during the partial update 150, which also results in conservation of significant computing resources.

[0146] FIG. 4 is a diagram 400 of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure. FIG. 4 illustrates an example of the sensors 110, the image source(s) 106, the data analyzer 142, the update strategy selector 144, and the update engines 146. In FIG. 4, the data analyzer 142 is configured to obtain sensor data 406 corresponding to a scene that is modeled by a gaussian scene representation, such as the scene 132 modeled by the gaussian scene representation 154 of FIG. 1, and the update strategy selector 144 is configured to select, from among a plurality of update strategies 460, an update strategy 462 for the gaussian scene representation 154 based on whether the sensor data 406 indicates a change in the scene 132 that satisfies an update criterion 452.QUALCOMM Ref. No. 2501500WO- 52 / 98 -

[0147] The sensors 110 include at least one of the motion sensor(s) 210, the light sensor(s) 212, and one or more ranging sensors 402, such as one or more sonar, lidar, or radar sensors. The sensor data 406 includes data from one or more of the sensors 110, from the image source(s) 106, or both. To illustrate, the sensor data 406 includes motion data 410 from the motion sensor 210, ranging data 412 from the ranging sensor 402, and light data 414 from the light sensor 212. Alternatively, or in addition, the sensor data 406 includes input pixel data 416 from the image source 106. To illustrate, the image source 106 includes one or more image sensors 404 (e.g., one or more cameras) configured to capture images of the scene 132 and to provide data corresponding to the images as the input pixel data 416. According to an aspect, the image sensor 404 corresponds to a video camera that captures a sequence of frames of the scene 132 and outputs a sequence of sets of the input pixel data 416, with each set of the input pixel data 416 including pixel data for a respective captured frame. In a particular embodiment, the sensor data 406 includes, corresponds to, or is included in the image data 108. the sensor data 112, the image data 147. the sensor data 145, the sensor data 303, the sensor data 305, the input 307, or any combination thereof.

[0148] The data analyzer 142 is configured to process the sensor data 406 to generate one or more change metrics 444. illustrated as a movement metric 440 and a lighting change metric 442. The data analyzer 142 includes a motion evaluator 420 configured to generate the movement metric 440, and a lighting change evaluator 422 configured to generate the lighting change metric 442.

[0149] According to an aspect, the motion evaluator 420 is configured to obtain motion sensor data 424 based on the motion data 410. In some examples, the motion sensor data 424 includes, corresponds to, is based on, or is included in the motion data 410. In a particular example, the motion evaluator 420 processes the motion data 410 to: identify one or more objects moving in the scene, one or more regions of the scene associated with the motion, or both; determine size(s) of one or more regions of interest associated with the motion; correlate and / or combine motion data for each motion sensor 210 in embodiments in which the sensors 110 includes multiple motion sensors 210, or any combination thereof. In an illustrative example in which the sensors 110 include a first motion sensor 210 such as an inertial measurement unit (IMU) to detect movement of a camera or motion sensor, and a second motion sensor 210 (e.g., a depthQUALCOMM Ref. No. 2501500WO- 53 / 98 -camera or ultrasound sensor) to detect motion within the scene relative to the second motion sensor 210. the motion evaluator 420 may use the motion data from the first motion sensor 210 to convert the motion data from the second motion sensor 210 into an inertial frame when generating the motion sensor data 424.

[0150] Alternatively, or in addition, the motion evaluator 420 is configured to obtain ranging sensor data 426 that is based on the ranging data 412. In some examples, the ranging sensor data 426 includes, corresponds to, is based on, or is included in the ranging data 412. In a particular example, the motion evaluator 420 processes the ranging data 412 to: identify one or more objects moving in the scene, one or more regions of the scene associated with the motion, or both; determine size(s) of one or more regions of interest associated with the motion; correlate and / or combine ranging data for each ranging sensor 402 in embodiments in which the sensors 110 includes multiple ranging sensors 402, or any combination thereof. Although the motion sensor 210 and the ranging sensor 402 are depicted as distinct classes of sensors, in some embodiments the ranging sensor(s) 402 are a subset of the motion sensor(s) 210. In some examples, the motion evaluator 420 may use at least a portion of the motion data 410 to convert the ranging data 412 into an inertial frame when generating the ranging sensor data 426, in a similar manner as described for the motion sensor data 424.

[0151] Alternatively, or in addition, the motion evaluator 420 includes a movement estimator 428 that is configured to process sets of input pixel data 416 obtained from the image sensor 404, such as first input pixel data 430 and second input pixel data 432 that is obtained after the first input pixel data 430. For example, the first input pixel data 430 is designated with a first timestamp indicating a first time and the second input pixel data 432 is designated with a second timestamp indicating a second time that is subsequent to the first time. In this example, a timestamp can indicate a capture time, a receipt time, a generation time, a play out time, or a combination thereof. In an example, the image sensor 404 is a video camera and the first input pixel data 430 and the second input pixel data 432 correspond to sequential frames captured by the image sensor 404, or the first input pixel data 430 and the second input pixel data 432 may not correspond to sequential frames and may instead be separated by one or more intervening frames. In other examples, the image sensor 404 may not be a video camera, such as low-frame-rate time-lapse camera (e.g., fewer than one frame per second). The movementQUALCOMM Ref. No. 2501500WO- 54 / 98 -estimator 428 is configured to determine an estimated movement based on the first input pixel data 430 and the second input pixel data 432, such as by comparing values of pixels of the second input pixel data 432 to values of corresponding pixels of the first input pixel data 430, or by performing motion estimation to generate motion vectors and / or motion fields based on block matching, feature tracking, and / or optical flow techniques as illustrative, non-limiting examples.

[0152] In a particular embodiment, the motion evaluator 420 determines the movement metric 440 based on the motion sensor data 424, the ranging sensor data 426, the estimated movement generated by the movement estimator 428, or a combination thereof. For example, the movement metric 440 can indicate an average amount of movement detected in the scene, movement (e.g., distance and direction) of one or more objects in the scene and / or one or more regions of the scene, or other movement data. In some embodiments, the movement metric 440 can include separate movement metrics associated with two or more of the motion sensor data 424, the ranging sensor data 426, and the estimated movement generated by the movement estimator 428 instead of, or in addition to, one or movement metrics that are based on combinations of two or more the motion sensor data 424, the ranging sensor data 426, and the estimated movement generated by the movement estimator 428. In some embodiments, the motion evaluator 420 includes a machine learning (ML) model configured to perform at least a portion of the operations described above to evaluate the movement metric 440 based on the sensor data 406.

[0153] According to an aspect, the lighting change evaluator 422 is configured to obtain the light data 414 from the light sensor 212 and to estimate lighting changes in the scene, such as based on detecting changes in the light data 414 over time. In some examples, the light data 414 includes one or more time series of light data corresponding to the scene 132, and the lighting change evaluator 422 is configured to compare earlier instances of the light data 414 to later instances of the light data 414 to estimate the lighting changes over time. In a particular example, the light data 414 indicates a set of intensity measurements from a photodetector array, and the lighting change evaluator 422 is configured to process the light data 414 to detect an amount of lighting change for each of multiple regions in the scene 132. In some examples, the lighting change evaluator 422 is configured to detect a single overall (e.g., average orQUALCOMM Ref. No. 2501500WO- 55 / 98 -total) amount of lighting change for the scene 132, pixel-by-pixel intensity changes, lighting changes at one or more other resolutions, or any combination thereof.

[0154] Alternatively, or in addition, the lighting change evaluator 422 is configured to determine an estimated lighting change based on the first input pixel data 430 and the second input pixel data 432 from the image sensor 404. In an example, the lighting change evaluator 422 compares intensity values and / or color data of pixels of the second input pixel data 432 to corresponding pixels of the first input pixel data 430 to generate scene-wide, region-by-region, or pixel-by-pixel lighting change data, or any combination thereof.

[0155] In a particular embodiment, the lighting change evaluator 422 determines the lighting change metric 442 based on the light data 414, the input pixel data 416, or a combination thereof. For example, the lighting change metric 442 can indicate an amount of change in a total or average amount of lighting in a scene, or an amount of lighting change for each of one or more regions of the scene, one or more other types of lighting change, or any combination thereof. In other examples, detection can include that a light switch is turned on / off, that a person has entered inside a building or warehouse with certain lighting conditions, that a person (and / or the light sensor 212 or the image sensor 404) is in a park in the daytime and it’s a sunny day, or that it’s nighttime and the person (and / or the light sensor 212 or the image sensor 404) is indoors and the light switch is on, as illustrative, non-limiting examples. In some embodiments, the lighting change evaluator 422 includes a ML model configured to perform at least a portion of the operations described above to evaluate the lighting change metric 442 based on the sensor data 406.

[0156] The changes in the scene 132 detected by the data analyzer 142 can thus correspond to a lighting change, detected movement, or both. The data analyzer 142 provides the movement metric 440 corresponding to the detected movement, the lighting change metric 442 corresponding to the lighting change, or both, to the update strategy selector 144.

[0157] The update strategy selector 144 is configured to select, from among the plurality of update strategies 460, an update strategy 462 for the gaussian scene representation 154 based on whether change(s) in the scene 132 indicated by theQUALCOMM Ref. No. 2501500WO- 56 / 98 -movement metric 440 and / or the lighting change metric 442 satisfy one or more update criterion 452. For example, the update strategy selector 144 can obtain (e.g., store, retrieve, update, etc.) update criteria data 450 that includes one or more update criteria 452 to determine which of the update strategies 460 to select. As illustrated, the update criteria data 450 includes a first update criterion 452A that corresponds to comparison of one or more of the change metric(s) 444 to one or more first thresholds 456, one or more second thresholds 458, or a combination thereof. The update criteria data 450 optionally includes one or more additional update criteria 452 including a Tth update criterion 452T (T is an integer greater than 1). According to an aspect, the update strategy’ selector 144 is configured to perform comparisons of the change metric(s) 444 to one or more of the first threshold(s) 456, one or more of the second threshold(s) 458, or both, at a comparator 446, to evaluate one or more of the update criteria 452.

[0158] As an illustrative example, the first update criterion 452A can correspond to determining if an amount of movement indicated in the movement metric 440 is sufficient to perform an update of the gaussian scene representation 154 and, if so, whether the amount of movement is sufficient to perform a full update or a partial update. In this example, a first threshold 456 can correspond to a first amount of movement, and a second threshold 458 that is greater than the first threshold 456 can correspond to a second amount of movement that is greater than the first amount of movement. The update strategies 460 can include a '‘partial update” strategy corresponding to the partial update 150 at the update engines 146, a “full update" strategy’ corresponding to the full update 148, and a “no update" strategy which may correspond to the no update 152. To evaluate the first update criterion 452A, the update strategy’ selector 144 can compare the movement metric 440 to the first threshold 456 and / or to the second threshold 458 at the comparator 446. If the movement metric 440 exceeds the second threshold 458, the update strategy selector 144 selects the “full update” strategy. Otherwise, if the movement metric 440 exceeds the first threshold 456 but does not exceed the second threshold 458, the update strategy selector 144 selects the “partial update” strategy. If the movement metric 440 does not exceed the first threshold 456, the “no update” strategy’ is selected.

[0159] One or more of the update criteria 452 may correspond to one or more other movement criteria, such as a number of regions or objects in the scene detected to haveQUALCOMM Ref. No. 2501500WO- 57 / 98 -non-insignificant movement, a number of regions or objects detected to have movement above one or more thresholds, a combined size of the regions of the scene determined to be in motion, etc., which may be individually evaluated at the update strategy selector 144, such as via comparisons to one or more thresholds. According to some aspects, the movement metric 440 can include separate metrics for movement detected based on the motion sensor 210. based on the ranging sensor 402, or based on the image sensor 404, and each of the separate metrics can be evaluated against one or more distinct sets of thresholds, such as to take into account differences between the various sources of motion data, such as differences in accuracy, resolution, frame rate (e.g., time delay between successive measurements), etc., of the different sensors. In a particular embodiment, conditions such as lighting may be used during evaluation of one or more movement-based update criteria 452. For example, motion detected via the motion sensor 210 and / or the ranging sensor 402 may be more accurate than motion detected via the image sensor 404 in low-light conditions; as a result, one or more of the thresholds 456. 458 may be adjusted to reduce the likelihood of performing updates due to false positives from the movement estimator 428 when ambient light is low.

[0160] As another illustrative example, the Tth update criterion 452T can correspond to determining if an amount of lighting change indicated in the lighting change metric 442 is sufficient to perform an update of the gaussian scene representation 154 and, if so, whether the amount of lighting change is sufficient to perform a full update or a partial update. In this example, the Tth update criterion 452T can include a first threshold 456 that corresponds to a first amount of lighting change, and a second threshold 458 that is greater than the first threshold 456 and that corresponds to a second amount of lighting change that is greater than the first amount of lighting change. To evaluate the Tth update criterion 452T, the update strategy selector 144 can compare the lighting change metric 442 to the first threshold 456 and / or to the second threshold 458 at the comparator 446. If the lighting change metric 442 exceeds the second threshold 458, the update strategy selector 144 selects the ‘Tull update” strategy. Otherwise, if the lighting change metric 442 exceeds the first threshold 456 but does not exceed the second threshold 458, the update strategy selector 144 selects the “partial update” strategy. If the movement metric 440 does not exceed the first threshold 456. the “no update” strategy is selected.QUALCOMM Ref. No. 2501500WO- 58 / 98 -

[0161] One or more of the update criteria 452 may correspond to one or more additional lighting change criteria, such as a number of regions or objects in the scene detected to have non-insignificant lighting change, a number of regions or objects detected to have a lighting change that exceeds one or more thresholds, a combined size of the regions of the scene determined to have changes in lighting, etc., which may be individually- evaluated at the update strategy selector 144, such as via comparisons to one or more thresholds. According to some aspects, the lighting change metric 442 can include separate metrics for lighting change detected based on the light sensor 212 or based on the image sensor 404, and each of the separate metrics can be evaluated against one or more distinct sets of thresholds, such as to take into account differences between the various sources of light data, such as differences in accuracy, resolution, frame rate (e.g., time delay between successive measurements), etc., of the different sensors.

[0162] The update strategy selector 144 can be configured to process each of the update criteria 452 and to select the most intensive update strategy that is determined by any of the update criteria 452. To illustrate, if one or more of the update criteria 452 evaluate to selection of a full update strategy', the full update strategy' is selected. If none of the update criteria 452 evaluates to selection of the full update strategy but one or more of the update criteria 452 evaluates to a partial update strategy, the partial update strategy is selected. According to an aspect, upon determining that any of the update criteria 452 results in selection of a full update strategy, the update strategy selector 144 bypasses evaluating further update criteria 452, and the selected update strategy' 462 is provided to the update engines 146.

[0163] In each of the examples provided above for the first update criterion 452A and the Tth update criterion 452T, the update criterion 452 includes a change metric 444 (e.g., the movement metric 440 and the lighting change metric 442, respectively) associated with the change in the scene being greater than a first threshold 456 and not greater than (e.g., less than) a second threshold 458, and evaluating to a partial update strategy'. However, in other examples, one or more of the update criteria 452 corresponds to a comparison involving a single threshold (e.g., a single comparison to determine whether or not a detected change is significant enough to select a full update strategy'), or comparisons involving more than two thresholds. For example, a multimodal update criterion 452 can include comparison of the movement metric 440 toQUALCOMM Ref. No. 2501500WO- 59 / 98 -one or more movement thresholds, comparison of the lighting change metric 442 to one or more lighting threshold, comparison to combinations of movement and lighting changes (e.g., a weighted sum of the movement metric 440 and the lighting change metric 442) to one or more thresholds, or any combination thereof. In some embodiments, one or more of the update criteria 452 can be based on results of evaluating other update criteria 452. For example, a full update strategy may be selected based on determining that a threshold number of individual update criteria 452 evaluate to a partial update strategy, even though none of the individual update criteria 452 evaluate to selection of the full update strategy.

[0164] Thus, the sensor data 406 can be used to select an update strategy 462 based on motion, lighting changes, or other characteristics associated with updating the gaussian scene representation. A technical advantage of using the sensor data 406 from the sensors 110 to select the update strategy 462 is that processing sensor data from the sensors 110 uses fewer computing resources than processing pixel data. Similarly, performing comparisons between two sets of the input pixel data 416 from the image sensor 404 uses fewer computing resources as compared to performing direct comparisons of input pixel data and reference pixel data, which may involve performing rendering to generate the reference pixel data, to determine regions of interest.Processing the sensor data 406 also uses fewer computing resources than performing a full update of the gaussian scene representation; accordingly, using the sensor data 406 to detect changes in the scene in order to select an update strategy can save significant computing resources.

[0165] Although the data analyzer 142 is depicted as receiving the sensor data 406 from the motion sensor 210, the ranging sensor 402, the light sensor 212, and the image source 106, in other embodiments one or more of the motion sensor 210, the ranging sensor 402, the light sensor 212, or the image source 106 may be omitted, or the data analyzer 142 may not generate the change metric(s) 444 based on the sensor data 406 from one or more of the motion sensor 210, the ranging sensor 402, the light sensor 212, or the image source 106.

[0166] In a first example, the data analyzer 142 determines scene changes based on changes in lighting and without evaluating motion in the scene. In this example, the dataQUALCOMM Ref. No. 2501500WO- 60 / 98 -analyzer 142 may omit, or bypass operation of, the motion evaluator 420 and generates the lighting change metric 442 based on processing of the light data 414, the input pixel data 416, or both, at the lighting change evaluator 422. The update strategy selector 144 can restrict or bypass evaluation of any movement-based update criteria 452 during selection of the selected update strategy 462. A technical advantage of selecting an update strategy based on changes in lighting is that the gaussian scene representation 154 can be updated in response to changing lighting conditions that may not reliably be detected via motion sensing techniques. A technical advantage of selecting an update strategy based on changes in lighting and omitting motion detection is that scene changes can be estimated using less power and fewer computational resources as compared to performing detection of both lighting changes and motion. In addition, power and computational resources associated with operation of the motion sensor 210, the ranging sensor 402, or both, may be reduced or eliminated.

[0167] In a second example, the data analyzer 142 determines scene changes based on detected motion and without evaluating lighting changes in the scene. In this example, the data analyzer 142 may omit, or bypass operation of, the lighting change evaluator 422 and generates the movement metric 440 based on processing of the motion data 410, the ranging data 412, the input pixel data 416, or any combination thereof, at the motion evaluator 420. The update strategy selector 144 can restrict or bypass evaluation of any lighting change-based update criteria 452 during selection of the selected update strategy 462. A technical advantage of selecting an update strategy based on detected motion is that the gaussian scene representation 154 can be updated in response to movement of one or more objects in the scene that may not reliably be detected via lighting changes, such as when a moving object has a similar color and / or brightness as compared to a background of the scene. A technical advantage of selecting an update strategy based on motion detection and omitting detecting lighting changes is that scene changes can be estimated using less power and fewer computational resources as compared to performing detection of both lighting changes and motion. In addition, power and computational resources associated with operation of the light sensor 212 may be reduced or eliminated.

[0168] In some examples, the data analyzer 142 omits processing of the input pixel data 416 when determining the movement metric 440, the lighting change metric 442, orQUALCOMM Ref. No. 2501500WO- 61 / 98 -both. In such examples, the motion evaluator 420 performs motion processing based on the motion data 410, the ranging data 412. or both, and omits or bypasses operation of the movement estimator 428. Similarly, the lighting change evaluator 422 detects lighting changes based on the light data 414 and without processing input pixel data from the image sensor 404. A technical advantage of selecting an update strategy based on sensor data from the one or more sensors 110 and without processing the input pixel data 416 is that scene changes can be estimated using less power and fewer computational resources as compared to performing change detection using the sensor data from the one or more sensors 110 in addition to processing the input pixel data 416. In addition, power and computational resources associated with operation of the image sensor 404 may be reduced or eliminated.

[0169] In some other examples, the data analyzer 142 omits processing of sensor data from any of the sensors 110 when determining the movement metric 440. the lighting change metric 442. or both. In such examples, the motion evaluator 420 performs motion processing based on processing the input pixel data 416 at the movement estimator 428, and omits or bypasses processing corresponding to the motion sensor data 424 or the ranging sensor data 426. Similarly, the lighting change evaluator 422 detects lighting changes based on the input pixel data 416 from the image sensor 404 (e.g., the first input pixel data 430 and the second input pixel data 432) and without processing light data from the light sensor 212. A technical advantage of selecting an update strategy based on motion and / or lighting changes using the input pixel data 416 and without processing sensor data from the one or more sensors 110 is that scene changes can be estimated using less power and fewer computational resources as compared to performing change detection using the sensor data from the one or more sensors 110 in addition to processing the input pixel data 416. In addition, power and computational resources associated with operation of the sensors 110 may be reduced or eliminated.

[0170] FIG. 5A is a flow chart illustrating particular aspects of a method 500 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 500 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the data analyzer 142, the updateQUALCOMM Ref. No. 2501500WO- 62 / 98 -strategy selector 144, the lighting change evaluator 422 of FIG. 4, or a combination thereof.

[0171] In particular, the method 500 illustrates aspects of operations associated with selecting an update strategy based on a change in the scene detected based on sensor data. The method 500 includes, at block 504, detecting a lighting change based on the light data 414. For example, the data analyzer 142 receives the light data 414 from the light sensor 212 and generates the lighting change metric 442 based on processing the light data 414 at the lighting change evaluator 422, as described with reference to FIG.4.

[0172] The method 500 includes, at block 506, determining whether the lighting change matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154. Based on the lighting change matching one or more of the update criterion(s) 452, the method 500 includes updating the gaussian scene representation 154, at block 510. Otherwise, the method 500 continues to block 508, where a “no update” strategy' is selected for the gaussian scene representation 154 based on the lighting change. For example, the update criterion(s) 452 can include the lighting change metric 442 associated with the change in the scene being greater than a first threshold 456, and the update strategy selector 144 compares the lighting change metric 442 to the first threshold 456 at the comparator 446. If the lighting change metric 442 is greater than the first threshold 456, the update strategy selector 144 selects an update strategy’ associated with performing a partial update 150 or a full update 148. Otherwise, if the lighting change metric 442 is not greater than the first threshold 456, the update strategy' selector 144 selects the “no update” strategy.

[0173] According to some aspects, determining whether the lighting change matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154, at block 506, also includes evaluating whether a partial update 150 or a full update 148 is to be performed. For example, the update criterion(s) 452 can include evaluating whether the lighting change metric 442 is greater than a second threshold 458, and the update strategy selector 144 compares the lighting change metric 442 to the second threshold 458 at the comparator 446. If the lighting change metric 442 is greater than the second threshold 458, the update strategy' selector 144 selects anQUALCOMM Ref. No. 2501500WO- 63 / 98 -update strategy associated with performing the full update 148; otherwise, if the lighting change metric 442 is greater than the first threshold 456 and is not greater than (e.g., is less than) the second threshold 458, the update strategy selector 144 selects an update strategy' associated with performing the partial update 150.

[0174] In an illustrative, non-limiting example, the first threshold 456 can correspond to a 2% change in the lighting of the scene, and the second threshold 458 can correspond to a 20% change in the lighting of the scene. For example, the light data 414 can be normalized to have a value in the range from 0 (e.g., no light, or the lowest-light value output by the light sensor 212) to 1 (e.g., full illumination, or the brightest-light value output by the light sensor 212), and the lighting change metric 442 can correspond to a difference between sequentially received values of the light data 414. In this example, the full update strategy' is selected when the lighting change metric 442 is greater than 0.2. the partial update strategy is selected when the lighting change metric 442 is greater than 0.02 but is not greater than 0.2. and the "no update7’ strategy is selected when the lighting change metric 442 is not greater than 0.02.

[0175] FIG. 5B is a flow chart illustrating particular aspects of a method 520 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 520 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the data analyzer 142, the update strategy’ selector 144, the motion evaluator 420, or a combination thereof.

[0176] In particular, the method 520 illustrates aspects of operations associated with selecting an update strategy based on a change in the scene detected based on sensor data. The method 520 includes, at block 524, detecting movement in the scene based on the motion data 410. For example, the data analyzer 142 receives the motion data 410 from the motion sensor 210 and generates the movement metric 440 at least partially' based on processing the motion data 410 at the motion evaluator 420, as described with reference to FIG. 4.

[0177] The method 520 includes, at block 526, determining whether the movement matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154. Based on the movement matching one or more of theQUALCOMM Ref. No. 2501500WO- 64 / 98 -update criterion(s) 452, the method 520 includes updating the gaussian scene representation 154. at block 510. Otherwise, the method 500 continues to block 508, where a “no update” strategy is selected for the gaussian scene representation 154 based on the movement. For example, the update criterion(s) 452 can include the movement metric 440 associated with the change in the scene being greater than a first threshold 456, and the update strategy selector 144 compares the movement metric 440 to the first threshold 456 at the comparator 446. If the movement metric 440 is greater than the first threshold 456, the update strategy selector 144 selects an update strategy associated with performing a partial update 150 or a full update 148. Otherwise, if the movement metric 440 is not greater than the first threshold 456, the update strategy selector 144 selects the “no update” strategy.

[0178] According to some aspects, determining whether the movement matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154. at block 526. also includes evaluating whether a partial update 150 or a full update 148 is to be performed. For example, the update criterion(s) 452 can include evaluating whether the movement metric 440 is greater than a second threshold 458, and the update strategy selector 144 compares the movement metric 440 to the second threshold 458 at the comparator 446. If the movement metric 440 is greater than the second threshold 458, the update strategy selector 144 selects an update strategy associated with performing the full update 148; otherwise, if the movement metric 440 is greater than the first threshold 456 and is not greater than (e.g., is less than or equal to) the second threshold 458, the update strategy selector 144 selects an update strategy associated with performing the partial update 150.

[0179] In an illustrative, non-limiting example, the first threshold 456 can correspond to a 1 centimeter (cm) change in the position of an object in the scene, and the second threshold 458 can correspond to a 10 cm change in the position of an object in the scene. In this example, the full update strategy is selected when the movement metric 440 indicates movement greater than 10 cm, the partial update strategy is selected when the movement metric 440 indicates movement greater than 1 cm but not greater than 10 cm. and the “no update” strategy is selected when the movement metric 440 does not indicate movement greater than 1 cm.QUALCOMM Ref. No. 2501500WO- 65 / 98 -

[0180] FIG. 5C is a flow chart illustrating particular aspects of a method 530 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 530 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the data analyzer 142, the update strategy’ selector 144, the motion evaluator 420, the lighting change evaluator 422, or a combination thereof.

[0181] In particular, the method 530 illustrates aspects of operations associated with selecting an update strategy based on a change in the scene detected by sensor data. The method 530 includes, at block 534, detecting one or more changes in the scene based on the first input pixel data 430 and the second input pixel data 432. For example, the data analyzer 142 obtains the input pixel data 416 from the image sensor 404, including the first input pixel data 430 and the second input pixel data 432 obtained after the first input pixel data 430. The data analyzer 142 generates the movement metric 440 based on processing the first input pixel data 430 and the second input pixel data 432 at the motion evaluator 420, generates the lighting change metric 442 based on processing the first input pixel data 430 and the second input pixel data 432 at the lighting change evaluator 422, or both, such as described with reference to FIG. 4.

[0182] The method 530 includes, at block 536, determining whether the detected change(s) (e.g.. movement, lighting change, or both) matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154. Based on the movement, the lighting change, or both, matching one or more of the update criterion(s) 452, the method 530 includes updating the gaussian scene representation 154, at block 510. Otherw ise, the method 500 continues to block 508, where a “no update” strategy is selected for the gaussian scene representation 154. For example, the update criterion(s) 452 can include the lighting change metric 442 associated with the change in the scene being greater than a particular first threshold 456, and the update strategy' selector 144 compares the lighting change metric 442 to the particular first threshold 456. If the lighting change metric 442 is greater than the particular first threshold 456, the update strategy selector 144 selects an update strategy associated with performing a partial update 150 or a full update 148. As another example, the update criterion(s) 452 can include the movement metric 440 associated with the change in theQUALCOMM Ref. No. 2501500WO- 66 / 98 -scene being greater than another first threshold 456, and the update strategy selector 144 compares the movement metric 440 to the other first threshold 456. If the movement metric 440 is greater than the other first threshold 456, the update strategy selector 144 selects an update strategy associated with performing a partial update 150 or a full update 148. Otherwise, if the lighting change metric 442 (if generated) is not greater than (e.g., is less than) the particular first threshold 456. and the movement metric 440 (if generated) is not greater than (e.g., is less than) the other first threshold 456, the update strategy selector 144 selects the “no update” strategy.

[0183] According to some aspects, determining whether the detected scene change(s) match one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154, at block 536, also includes evaluating whether a partial update 150 or a full update 148 is to be performed. For example, the update criterion(s) 452 can include evaluating whether the lighting change metric 442 is greater than a particular second threshold 458, evaluating whether the movement metric 440 is greater than another second threshold 458, or both. Tn this case, if the lighting change metric 442 is greater than the particular second threshold 458, the movement metric 440 is greater than the other second threshold 458, or both, the update strategy selector 144 selects an update strategy associated with performing the full update 148; otherwise, the update strategy selector 144 selects an update strategy associated with performing a partial update 150.

[0184] In an illustrative, non-limiting example, the movement metric 440 can correspond to a number of pixels that an object’s position has changed between the first input pixel data 430 and the second input pixel data 432, a movement-based first threshold 456 can correspond to a 3 pixel change, and a movement-based second threshold 458 can correspond to a 15 pixel change. Similarly, in an illustrative, nonlimiting example, the lighting change metric 442 can correspond to how much a normalized pixel intensity for an object or region has changed between the first input pixel data 430 and the second input pixel data 432, a lighting-based first threshold 456 can correspond to 0.02, and a lighting-based second threshold 458 can correspond to 0.2.QUALCOMM Ref. No. 2501500WO- 67 / 98 -

[0185] FIG. 5D is a flow chart illustrating particular aspects of a method 540 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 540 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the data analyzer 142, the update strategy’ selector 144, or both.

[0186] In particular, the method 540 illustrates aspects of operations associated with selecting an update strategy based on a change in the scene detected by sensor data. The method 540 includes, at block 544, performing a multimodal detection of one or more changes in the scene based on the sensor data 406. For example, the data analyzer 142 obtains one or more of the motion data 410 from the motion sensor 210, the ranging data 412 from the ranging sensor 402, the light data 414 from the light sensor 212, the input pixel data 416 from the image sensor 404, or any combination thereof. The data analyzer 142 generates the movement metric 440 based on processing the motion data 410, the ranging data 412, the first input pixel data 430 and the second input pixel data 432, or any combination thereof, at the motion evaluator 420. The data analyzer 142 also generates the lighting change metric 442 based on processing the light data 414, the first input pixel data 430 and the second input pixel data 432, or both, at the lighting change evaluator 422.

[0187] The method 540 includes, at block 546, determining whether the detected change(s) (e.g., movement, lighting change, or both) matches one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154. Based on the movement, the lighting change, or both, matching one or more of the update criterion(s) 452, the method 540 includes updating the gaussian scene representation 154, at block 510. Otherwise, the method 500 continues to block 508, where a “‘no update’7strategy is selected for the gaussian scene representation 154. For example, the update criterion(s) 452 can include the lighting change metric 442 associated with the change in the scene being greater than a particular first threshold 456, and the update strategy selector 144 compares the lighting change metric 442 to the particular first threshold 456. If the lighting change metric 442 is greater than the particular first threshold 456, the update strategy selector 144 selects an update strategy associated with performing a partial update 150 or a full update 148. As another example, the updateQUALCOMM Ref. No. 2501500WO- 68 / 98 -criterion(s) 452 can include the movement metric 440 associated with the change in the scene being greater than another first threshold 456. and the update strategy- selector 144 compares the movement metric 440 to the other first threshold 456. If the movement metric 440 is greater than the other first threshold 456, the update strategy selector 144 selects an update strategy- associated with performing a partial update 150 or a full update 148. Otherwise, if the lighting change metric 442 is not greater than (e.g., is less than) the particular first threshold 456, and the movement metric 440 is not greater than (e.g., is less than) the other first threshold 456, the update strategy- selector 144 selects the “no update'’ strategy-.

[0188] According to some aspects, determining whether the detected scene change(s) match one or more of the update criterion(s) 452 associated with updating the gaussian scene representation 154, at block 546, also includes evaluating whether a partial update 150 or a full update 148 is to be performed. For example, the update criterion(s) 452 can include evaluating whether the lighting change metric 442 is greater than a particular second threshold 458, evaluating whether the movement metric 440 is greater than another second threshold 458, or both. In this case, if the lighting change metric 442 is greater than the particular second threshold 458, the movement metric 440 is greater than the other second threshold 458, or both, the update strategy- selector 144 selects an update strategy associated with performing the full update 148; otherwise, the update strategy- selector 144 selects an update strategy associated with performing a partial update 150.

[0189] In an illustrative, non-limiting example, the movement metric 440, the lighting change metric 442, the one or more first threshold 456, and the one or more second thresholds 458 can have similar values as described for FIGS. 5A, 5B, and / or 5C. To illustrate, the movement metric 440 can include a sensor-based movement metric, a pixel-based movement metric, or both, and the movement- based first threshold 456 can include a sensor-based movement threshold of 2 cm, a pixel-based movement threshold of 3 pixels, or both. Similarly, the movement-based second threshold 458 can include a sensor-based movement threshold of 10 cm, a pixel -based movement threshold of 15 pixels, or both.QUALCOMM Ref. No. 2501500WO- 69 / 98 -

[0190] In an illustrative aspect, the lighting change metric 442 can include a sensorbased lighting change metric, a pixel-based lighting change metric, or both, and the lighting change-based first threshold 456 can be 0.02 for both sensor-based and pixelbased lighting changes. Alternatively, different lighting change-based first thresholds 456 may be used for sensor-based lighting changes and pixel-based lighting changes. Similarly, the lighting change-based second threshold 458 can be 0.2 for both sensorbased and pixel-based lighting changes, or different lighting change-based second thresholds 458 may be used for sensor-based lighting changes and pixel-based lighting changes.

[0191] Although specific threshold values are provided in the descriptions of FIGS. 5A-5D, it should be understood that such values are provided for illustrative purposes only, and different values can be selected for use in various implementations.

[0192] FIG. 6 is a flow chart illustrating particular aspects of a method 600 associated with updating gaussian scene representations that may be performed by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 600 can be initiated, performed, or controlled by the processor(s) 130 of FIG. 1 executing instructions to perform operations of the data analyzer 142. the update strategy selector 144, or both.

[0193] In particular, the method 600 illustrates aspects of operations associated with selecting an update strategy based on a change in the scene detected by sensor data. The method 600 includes, at block 602, obtaining sensor data corresponding to a scene modeled by a gaussian scene representation. For example, the data analyzer 142 of FIG.4 receives the sensor data 406, such as one or more of the motion data 410, the ranging data 412, the light data 414, or the input pixel data 416.

[0194] The method 600 includes, at block 604, selecting, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion. For example, the update strategy selector 144 selects the update strategy 462 for the gaussian scene representation 154 from among the update strategies 460 based on whether the change metric(s) 444 satisfy one or more of the update criterion 452.According to an aspect, the method 600 includes comparing a movement metric, aQUALCOMM Ref. No. 2501500WO- 70 / 98 -lighting change metric, or both, to one or more first thresholds to select the update strategy. For example, the update strategy selector 144 compares the movement metric 440, the lighting change metric 442, or both, to one or more of the first thresholds 456 to select the update strategy462.

[0195] According to some aspects, the sensor data includes light data from a light sensor, and the change in the scene corresponds to a lighting change. For example, the data analyzer 142 processes the light data 414 from the light sensor 212 and generates the lighting change metric 442 corresponding to a lighting change in the scene.

[0196] According to some aspects, the sensor data includes motion data from a motion sensor, and the change in the scene corresponds to detected movement. For example, the data analyzer 142 processes the motion data 410 from the motion sensor 210 and generates the movement metric 440 corresponding to motion detected in the scene. The sensor data can also include data from one or more ranging sensors, such as the ranging data 412 from the ranging sensor 402.

[0197] According to some aspects, the method 600 includes obtaining the sensor data from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data, and determining an estimated movement based on the first input pixel data and the second input pixel data. For example, the data analyzer 142 receives the input pixel data 416, including the first input pixel data 430 and the second input pixel data 432, from the image sensor 404. The data analyzer 142 determines the change in the scene corresponding to an estimated movement based on processing the first input pixel data 430 and the second input pixel data 432 at the motion evaluator 420.

[0198] According to some aspects, the method 600 includes obtaining the sensor data from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data, and determining an estimated lighting change based on the first input pixel data and the second input pixel data. For example, the data analyzer 142 receives the input pixel data 416, including the first input pixel data 430 and the second input pixel data 432, from the image sensor 404. The data analyzer 142 determines the change in the scene corresponding to an estimatedQUALCOMM Ref. No. 2501500WO- 71 / 98 -lighting change based on processing the first input pixel data 430 and the second input pixel data 432 at the lighting change evaluator 422.

[0199] According to some aspects, the update criterion includes a metric associated with the change in the scene being greater than a first threshold, such as the first threshold 456 of the first update criterion 452A. To illustrate, each of FIGS. 5 A, 5B, 5C. and 5D describes an example in which the movement metric 440 and / or the lighting change metric 442 is compared to a first threshold 456. According to some aspects, the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold, such as the first threshold 456 and the second threshold 458 of the first update criterion 452A. To illustrate, FIG. 5A includes an example in which the update strategy selector 144 selects an update strategy associated with performing the partial update 150 based on the lighting change metric 442 being greater than the first threshold 456 and less than the second threshold 458.

[0200] The method 600 thus uses the sensor data to detect a change in the scene and selects an update strategy for the gaussian scene representation based on whether the change satisfies an update criterion. A technical advantage of using the sensor data to select the update strategy is that processing the sensor data uses fewer computing resources than processing pixel data. Using the sensor data also consumes fewer computing resources than performing a full update of the gaussian scene representation; accordingly, using the sensor data to detect changes in the scene in order to select an update strategy can save significant computing resources.

[0201] FIG. 7 is a flow chart illustrating particular aspects of a method 700 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 700 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 700 can be initiated, performed, or controlled by the controller 160.

[0202] The method 700 includes, at block 702, obtaining input pixel data. For example, the input pixel data can correspond to, include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1. AsQUALCOMM Ref. No. 2501500WO- 72 / 98 -another example, the input pixel data can correspond to. include, or be included within image data 147 from the image source(s) 106 of FIG. 2. As another example, the input pixel data can correspond to, include, or be included within the sensor data 305 or the ROI data 313 of FIG. 3 A.

[0203] The method 700 includes, at block 704, based at least in part on differences between the input pixel data and reference pixel data, selecting, from among a lurality of update strategies, an update strategy' for the gaussian scene representation. The reference pixel data represents at least a portion of one or more views associated with the gaussian scene representation. For example, the update strategy selector 144 can receive information indicating a level of differences between the input pixel data and reference pixel data from the data analyzer 142 and can select to perform no update 152, the partial update 150, or the full update 148 based on the level of differences.

[0204] FIG. 8 is a flow chart illustrating particular aspects of a method 800 associated with updating gaussian scene representations that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, the method 800 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the selective model update pipeline 140. As another example, the method 800 can be initiated, performed, or controlled by the controller 160.

[0205] The method 800 includes, at block 806, determining difference(s) between input pixel data 802 and reference pixel data 804. The input pixel data 802 can correspond to, include, or be included within the image data 108 from the image source(s) 106, the modem 114, or the buffer(s) 118 of FIG. 1; image data 147 from the image source(s) 106 of FIG. 2; or the sensor data 305 or the ROI data 313 of FIG. 3 A. The reference pixel data 804 can correspond to, include, or be included within the reference image(s) 124 (e.g., the training image(s) 126 and / or the synthetic image(s) 128) of FIG. 1.

[0206] In a particular aspect, determining the difference(s) between the input pixel data 802 and the reference pixel data 804 includes determining value(s) of one or more difference metrics. For example, the input pixel data 802 can be compared (e g., pixel-by-pixel or region-by-region) to the reference pixel data 804 to identify pixel values that differ significantly (e.g., by more than a threshold amount) between correspondingQUALCOMM Ref. No. 2501500WO- 73 / 98 -pixels or regions of the input pixel data 802 and the reference pixel data 804. Locations associated with significant differences between the input pixel data 802 and the reference pixel data 804 can be identified as difference regions (or regions of interest), and a value of a difference metric can be determined for each difference region. In various examples, the difference metric can indicate a size of or dimensions of the difference region, a count of pixels within the difference region, a magnitude of difference between pixels of the difference region (e.g., based on distance between points in a color space of the pixels), or a combination thereof.

[0207] The method 800 also includes, at block 808, selecting an update strategy based on the difference(s) between the input pixel data 802 and the reference pixel data 804. For example, the update strategy can be selected by the update strategy selector 144 of any of FIGS. 1 to 3 A. The update strategy' can be selected, for example, from among no update 852, a partial update 850. or a full update 848. As an example, the update strategy’ can be selected based on a number (e.g., a count) of identified difference regions, a size of one or more difference regions, a magnitude of difference associated with one or more difference regions, a location of a difference region within an image (e.g., in the input pixel data 802 or in the reference pixel data 804), a location within a scene (e.g.. a three-dimensional location) associated with a difference region, or a combination thereof.

[0208] In some cases, the update strategy can be selected based on other factors in addition to the difference(s) between the input pixel data 802 and the reference pixel data 804. For example, the update strategy can be selected based, in part, on device condition information. To illustrate, for a first set of differences betw een the input pixel data 802 and the reference pixel data 804, the partial update 850 may be selected (based on the first set of differences) if a device updating the gaussian scene representation has a first device condition (e.g., is plugged into an external power source) and no update may be selected if the device has a second device condition (e.g., is operating on battery power).

[0209] Technical advantages of the methods 500, 520, 530. 540, 600. 700, and 800 include enabling updating of gaussian scene representations in a manner that is efficient in terms of computing resources utilized. For example, performing the full update usesQUALCOMM Ref. No. 2501500WO- 74 / 98 -significant computing resources and generally uses all of the input pixel data to perform calculations that account for all of the gaussians of the gaussian scene representation. In contrast, the methods 500, 520, 530, 540, 600, 700, and 800 enable selection of an update strategy (e.g., a partial update) that uses fewer calculations under particular circumstances, such as when the differences detected affect only a subset of the gaussians of the gaussian scene representation. Thus, fewer calculations can be used to keep the gaussian scene representation updated based on the input pixel data, resulting in savings of computing resources (e.g., processor time, power, and memory). Further, when the differences associated with the input pixel data and / or scene changes associated with input sensor data are small, the methods 500, 520, 530, 540, 600, 700, and 800 can decide to perform no update, in which case even the computing resources associated with performing the partial update are conserved.

[0210] FIG. 9 depicts an implementation 900 of the device 102 of FIG. 1 as an integrated circuit 902 that includes the one or more processors 130. The integrated circuit 902 also includes one or more input connectors 906 (e.g., pins, contacts, leads, solder balls, etc.) to enable input data 904 to be received for processing. For example, the input data 904 can correspond to, include, or be included within, the image data 108, the sensor data 112, the data 116, the reference images 124. the mapping data 122, the gaussian scene representation 154, or other data described with reference to FIG. 1; the reference pixel data or other data described with reference to FIG. 2; the sensor data 303, the sensor data 305, the input 307, the ROI data 313, the strategy' selection data 320, the mapping data 122, the gaussians 156, the ground-truth image 349 of FIGS. 3A-3C; the sensor data 406, the change metric(s) 444, the selected update strategy 462 of FIG. 4, or a combination thereof

[0211] The integrated circuit 902 also includes one or more output connectors 908 (e.g., pins, contacts, leads, solder balls, etc.) to enable output data 910 to be sent to another device or component. For example, the output data 910 can correspond to, include, or be included within, the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the mapping data 122, the view 136, another output of the selective model update pipeline 140. or other data described with reference to FIG. 1; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154 or other data described with reference to FIG. 2; the gaussian sceneQUALCOMM Ref. No. 2501500WO- 75 / 98 -representation 154 or gaussians 156 of the gaussian scene representation 154, the selection indication 321, the pixel data 341, the gradient values 343, the updated parameter values 347, or other data described with reference to FIGS. 3A-3C; the change metric(s) 444, the selected update strategy 462 of FIG. 4; or a combination thereof.

[0212] The processor(s) 130 of the integrated circuit 902 are configured to initiate, perform, or control various operatons described with reference to FIGS. 1-8. For example, in FIG. 9, the processor(s) 130 of the integrated circuit 902 include the selective model update pipeline 140 (or components thereof) and the gaussian scene representation 154 (or components thereof, e.g., a subset of the gaussians 156 of FIG.1). Optionally, the processor(s) 130 can also include the renderer(s) 158, the view selector 162, or both.

[0213] The integrated circuit 902 enables implementation of selective partial gaussian scene representation updating as a component in a system that includes or is coupled to one or more image sources, such as a mobile phone or tablet as described with reference to FIG. 10; a mixed reality or augmented reality glasses device as described with reference to FIG. 11; a wearable electronic device as described with reference to FIG.12; a voice-controlled speaker system described with reference to FIG. 13; a camera described with reference to FIG. 14; a virtual reality, mixed reality, or augmented reality7headset described with reference to FIG. 15; or a vehicle described with reference to FIG. 16 or FIG. 17.

[0214] FIG. 10 depicts an implementation 1000 in which the device 102 includes a mobile device 1002, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 1002 includes one or more microphones 1010. one or more speakers 1006, one or more cameras 1008, and a display screen 1004. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the mobile device 1002 and are illustrated using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1002.QUALCOMM Ref. No. 2501500WO- 76 / 98 -

[0215] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1008 of the mobile device 1002 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1008 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, whether the mobile device 1002 is connected to an external pow er source, a battery state of the mobile device 1002, availability of computing resources onboard the mobile device 1002, etc.

[0216] The mobile device 1002 can use the gaussian scene representation to generate one or more vie s, which can be displayed to a user via the display screen. For example, the view(s) can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the mobile device 1002 and subsequently sent to another device for rendering to generate one or more views. In some cases, the mobile device 1002 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a remote computing device, such as a server). In some cases, the mobile device 1002 can be configured to perform partial updates or full updates of the gaussian scene representation based on input pixel data received from another device, such as augmented reality or mixed reality glasses, a headset device, a body cam, or another wearable electronic device.

[0217] FIG. 11 depicts an implementation 1100 in which the device 102 includes a portable electronic device that corresponds to augmented reality or mixed reality glasses 1102. The glasses 1102 include a holographic projection unit 1112 configured to project visual data onto a surface of a lens 1114 or to reflect the visual data off of a surface ofQUALCOMM Ref. No. 2501500WO- 77 / 98 -the lens 1114 and onto the wearer's retina. The glasses 1102 also include one or more cameras 1108, and optionally one or more microphones 1110, one or more speakers 1106, or both. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the glasses 1102.

[0218] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1108 of the glasses 1102 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1108 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality7of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the glasses 1102.

[0219] The glasses 1102 can use the gaussian scene representation to generate one or more views, which can be projected by the holographic projection unit 1112 onto the lens 1114. For example, the view can be used to assist the user with navigation through the scene, can assist the user with identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the glasses 1102 and subsequently sent to another device for rendering to generate one or more views. In some cases, the glasses 1102 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the glasses 1102 or a remote computing device, such as a server).

[0220] FIG. 12 depicts an implementation 1200 in which the device 102 includes a wearable electronic device 1202, illustrated as awearable camera device (sometimesQUALCOMM Ref. No. 2501500WO- 78 / 98 -referred to as a “body cam”). A front 1222 and a back 1220 of the wearable electronic device 1202 are shown in FIG. 12.

[0221] In the example illustrated, the wearable electronic device 1202 includes one or more microphones 1210, one or more speakers 1206, and one or more cameras 1208. The wearable electronic device 1202 may also include a body 1224 with an integrated attachment mechanism, such as clip 1226. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the wearable electronic device 1202.

[0222] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1208 of the wearable electronic device 1202 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1208 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the wearable electronic device 1202.

[0223] The wearable electronic device 1202 can use the gaussian scene representation to facilitate user interaction with the scene. For example, the wearable electronic device 1202 can generate verbal output (e.g., instructions or information about the scene) based on the gaussian scene representation and can provide the verbal output to the user via the speaker(s) 1206. In some cases, the gaussian scene representation can be updated at the wearable electronic device 1202 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wearable electronic device 1202 can be configured to perform partial updates of the gaussian scene representationQUALCOMM Ref. No. 2501500WO- 79 / 98 -and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wearable electronic device 1202 or a remote computing device, such as a server).

[0224] FIG. 13 is an implementation 1300 in which the device 102 includes a wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 can have wireless network connectivity and is configured to execute an assistant operation. In the example illustrated, the wireless speaker and voice activated device 1302 includes one or more microphones 1310, one or more speakers 1306, one or more cameras 1308. and a display 1304. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the wireless speaker and voice activated device 1302.

[0225] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1308 of the wireless speaker and voice activated device 1302 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1308 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the wireless speaker and voice activated device 1302.

[0226] In a particular aspect, the wireless speaker and voice activated device 1302 can provide assistant operations based at least in part on the gaussian scene representation. For example, the wireless speaker and voice activated device 1302 can respond to verbal queries from the user based on the gaussian scene representation, such as by providing directions to an object or person in the scene. In some cases, the gaussianQUALCOMM Ref. No. 2501500WO- 80 / 98 -scene representation can be updated at the wireless speaker and voice activated device 1302 and subsequently sent to another device for rendering to generate one or more views. In some cases, the wireless speaker and voice activated device 1302 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the wireless speaker and voice activated device 1302 or a remote computing device, such as a server).

[0227] FIG. 14 depicts an implementation 1400 in which the device 102 includes a portable electronic device that corresponds to a camera device 1402. In the example illustrated, the camera device 1402 includes one or more microphones 1410, one or more speakers 1406, and one or more cameras 1408. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the camera device 1402.

[0228] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1408 of the camera device 1402 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1408 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the camera device 1402.

[0229] The camera device 1402 can use the gaussian scene representation to facilitate user interaction with the scene, documentation of the scene (e.g., by object or location tagging), etc. In some cases, the gaussian scene representation can be updated at the camera device 1402 and subsequently sent to another device for rendering to generateQUALCOMM Ref. No. 2501500WO- 81 / 98 -one or more views. In some cases, the camera device 1402 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the camera device 1402 or a remote computing device, such as a server).

[0230] FIG. 15 depicts an implementation 1500 in which the device 102 includes a portable electronic device that corresponds to a virtual reality, mixed reality, or augmented reality headset 1502. In the example illustrated, the headset 1502 includes one or more microphones 1510, one or more speakers 1506, and one or more cameras 1508. The headset 1502 also includes a display 1504 positioned to be visible to a user when the headset 1502 is worn by the user. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the headset 1502.

[0231] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1508 of the headset 1502 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1508 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the headset 1502.

[0232] In some cases, the headset 1502 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to generate a virtual reality or augmented reality view for a user. In some cases, the gaussian scene representation can be updated at the headset 1502 and subsequently sent to another device for rendering to generate one or more views. In some cases, the headset 1502 can be configured to perform partialQUALCOMM Ref. No. 2501500WO- 82 / 98 -updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g.. a mobile computing device associated with the headset 1502 or a remote computing device, such as a server).

[0233] FIG. 16 depicts an implementation 1600 in which the device 102 corresponds to, or is integrated within, a vehicle 1602. illustrated as a manned or unmanned aerial device (e g., a package delivery drone). In the example illustrated, the vehicle 1602 includes one or more microphones 1610, one or more speakers 1606, and one or more cameras 1608. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140. the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1602.

[0234] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1608 of the vehicle 1602 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1608 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the vehicle 1602.

[0235] In some cases, the vehicle 1602 can use the gaussian scene representation (e.g.. the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1602 and subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1602 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1602 or a remote computing device, such as a server).QUALCOMM Ref. No. 2501500WO- 83 / 98 -

[0236] FIG. 17 depicts another implementation 1700 in which the device 102 corresponds to. or is integrated within, a vehicle 1702, illustrated as a car. In the example illustrated, the vehicle 1702 includes one or more displays 1704, one or more microphones 1708, one or more speakers 1706, and one or more cameras 1710. For example, the cameras 1710 can include one or more interior cameras 1710A (which can be positioned to capture interior or exterior images), one or more front cameras 1710B, one or more rear cameras 1710C, one or more side cameras 1710D and 1710E, or a combination thereof. Components of the integrated circuit 902, including the processor(s) 130, the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162. are integrated in the vehicle 1702.

[0237] As a particular example, the integrated circuit 902 can include the gaussian scene representation 154 and the selective model update pipeline 140. In this example, the camera(s) 1710 of the vehicle 1702 can be configured to capture (automatically or in response to an input) an image of a scene near the camera(s) 1710 and to provide the selective model update pipeline 140 with input pixel data based on the image. In this example, the selective model update pipeline 140 can be configured to select an update strategy’ (from among a plurality of update strategies) for the gaussian scene representation based on the input pixel data. To illustrate, the selective model update pipeline 140 can select between performing no update, performing a partial update, or performing a full update of the gaussian scene representation. In some cases, a determination whether to perform a partial update of the gaussian scene representation is further based on additional data, such as sensor data, movement data, lighting data, availability of computing resources and / or power onboard the vehicle 1702.

[0238] In some cases, the vehicle 1702 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1702 and subsequently sent to another device for rendering to generate one or more views. In some cases, the vehicle 1702 can be configured to perform partial updates of the gaussian scene representation and to offload full updates of the gaussian scene representation to another device (e.g., a mobile computing device associated with the vehicle 1702 or a remote computing device, such as a server).QUALCOMM Ref. No. 2501500WO- 84 / 98 -

[0239] Referring to FIG. 18, a block diagram of a particular illustrative implementation of a device is depicted and generally designated 1800. In various implementations, the device 1800 may have more or fewer components than illustrated in FIG. 18. In an illustrative implementation, the device 1800 may correspond to the device 102. In an illustrative implementation, the device 1800 may perform one or more operations described with reference to FIGS. 1-17.

[0240] In a particular implementation, the device 1800 includes a processor 1806 (e.g., a CPU). The device 1800 may include one or more additional processors 1810 (e.g., one or more DSPs). In a particular aspect, the processor(s) 130 of FIG. 1 corresponds to the processor 1806, the processors 1810, or a combination thereof. The processors 1810 may include a speech and music coder-decoder (CODEC) 1808 that includes the selective model update pipeline 140, the gaussian scene representation 154, a voice coder ("‘vocoder’) encoder 1836, a vocoder decoder 1838, or a combination thereof.

[0241] The device 1800 may include a memory 1886 and a CODEC 1834. The memory 1886 may include instructions 1856, that are executable by the one or more additional processors 1810 (or the processor 1806) to implement the functionality described with reference to the selective model update pipeline 140, the gaussian scene representation 154, or both. The device 1800 may include a modem 1870 coupled, via a transceiver 1850, to an antenna 1852. In a particular aspect, the modem 1870 corresponds to the modem 114 of FIG. 1.

[0242] The device 1800 may include a display 1828 coupled to a display controller 1826. One or more speakers 1892 and one or more microphones 1894 may be coupled to the CODEC 1834. The CODEC 1834 may include a digital-to-analog converter (DAC) 1802. an analog-to-digital converter (ADC) 1804, or both. In a particular implementation, the CODEC 1834 may receive analog signals from the microphone(s) 1894, convert the analog signals to digital signals using the analog-to-digital converter 1804, and provide the digital signals to the speech and music codec 1808. In a particular implementation, the speech and music codec 1808 may provide digital signals to the CODEC 1834. The CODEC 1834 may convert the digital signals to analog signals using the digital-to-analog converter 1802 and may provide the analog signals to the one or more speakers 1892.QUALCOMM Ref. No. 2501500WO- 85 / 98 -

[0243] In a particular implementation, the device 1800 may be included in a system-in-package or system-on-chip device 1822. In a particular implementation, the memory 1886, the processor 1806, the processors 1810, the display controller 1826, the CODEC 1834, and the modem 1870 are included in the system-in-package or system-on-chip device 1822. In a particular implementation, an input device 1830 (e.g., one or more cameras) and a power supply 1844 are coupled to the system-in-package or the system-on-chip device 1822. Moreover, in a particular implementation, as illustrated in FIG. 18, the display 1828, the input device 1830, the one or more speakers 1892, the microphone(s) 1894, the antenna 1852, and the power supply 1844 are external to the system-in-package or the system-on-chip device 1822. In a particular implementation, each of the display 1828. the input device 1830, the one or more speakers 1892, the microphone(s) 1894, the antenna 1852, and the power supply 1844 may be coupled to a component of the system-in-package or the system-on-chip device 1822, such as an interface, or a controller.

[0244] The device 1800 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a computing device, a communication device, an intemet-of-things (loT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0245] In conjunction with the described implementations, an apparatus includes means for obtaining sensor data corresponding to a scene modeled by a gaussian scene representation. For example, the means for obtaining sensor data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106, the sensor(s) 110, the modem 114. the processor(s) 130, the selective model update pipeline 140, the data analyzer 142, the update strategy selector 144, the motion sensor(s) 210. the light sensor(s) 212, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the ranging sensor(s) 402, the image sensor(s) 404, the motion evaluator 420, theQUALCOMM Ref. No. 2501500WO- 86 / 98 -lighting change evaluator 422, the integrated circuit 902, the input connector(s) 906, the camera(s) 1008, the camera(s) 1108. the camera(s) 1208, the camera(s) 1308. the camera(s) 1408, the camera(s) 1508, the camera(s) 1608, the camera(s) 1710, the input device 1830, the processor 1806, the processor(s) 1810, the modem 1870, one or more other circuits or components configured to obtain sensor data, or any combination thereof.

[0246] The apparatus also includes means for selecting, from among a plurality of update strategies, an update strategy for a gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion. For example, the means for selecting an update strategy can correspond to the device 102, the processor(s) 130, the selective model update pipeline 140, the update strategy selector 144, the comparator 446, the integrated circuit 902, the processor 1806, the processor(s) 1810, one or more other circuits or components configured to select an update strategy for a gaussian scene representation, or any combination thereof.

[0247] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 1886) includes instructions (e.g., the instructions 1856) that, when executed by one or more processors (e.g.. the one or more processors 1810 or the processor 1806), cause the one or more processors to obtain sensor data corresponding to a scene modeled by a gaussian scene representation, and select, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0248] Particular aspects of the disclosure are described below in sets of interrelated Examples:

[0249] According to Example 1, a device includes a memory configured to store a gaussian scene representation; and one or more processors coupled to the memory and configured to obtain sensor data corresponding to a scene modeled by the gaussian scene representation; and select, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.QUALCOMM Ref. No. 2501500WO- 87 / 98 -

[0250] Example 2 includes the device of Example 1, wherein the sensor data includes light data from a light sensor, and wherein the change in the scene corresponds to a lighting change indicated by the light data.

[0251] Example 3 includes the device of Example 1 or Example 2, wherein the sensor data includes motion data from a motion sensor, and wherein the change in the scene corresponds to detected movement indicated by the motion data.

[0252] Example 4 includes the device of any of Examples 1 to 3, wherein the sensor data includes ranging data from one or more ranging sensors, and wherein the change in the scene corresponds to detected movement indicated by the ranging data.

[0253] Example 5 includes the device of any of Examples 1 to 4, wherein the sensor data is obtained from an image sensor and includes first input pixel data and second input pixel data that is obtained after the first input pixel data; the one or more processors are configured to determine an estimated movement based on the first input pixel data and the second input pixel data; and the change in the scene corresponds to the estimated movement.

[0254] Example 6 includes the device of any of Examples 1 to 5, wherein the sensor data is obtained from an image sensor and includes first input pixel data and second input pixel data that is obtained after the first input pixel data: the one or more processors are configured to determine an estimated lighting change based on the first input pixel data and the second input pixel data; and the change in the scene corresponds to the estimated lighting change.

[0255] Example 7 includes the device of any of Examples 1 to 6, wherein the one or more processors are configured to compare a movement metric, a lighting change metric, or both, to one or more first thresholds to select the update strategy.

[0256] Example 8 includes the device of any of Examples 1 to 7, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold.QUALCOMM Ref. No. 2501500WO- 88 / 98 -

[0257] Example 9 includes the device of any of Examples 1 to 8, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold.

[0258] Example 10 includes the device of any of Examples 1 to 9, wherein the plurality of update strategies includes a full update, a partial update, and no update.

[0259] Example 11 includes the device of any of Examples 1 to 10, and further includes one or more sensors coupled to the one or more processors and configured to generate the sensor data.

[0260] Example 12 includes the device of Example 11, wherein the one or more sensors include a motion sensor configured to generate motion data that is included in the sensor data.

[0261] Example 13 includes the device of Example 11 or Example 12, wherein the one or more sensors include a light sensor configured to generate light data that is included in the sensor data.

[0262] Example 14 includes the device of any of Examples 1 to 13, and further includes one or more cameras coupled to the one or more processors and configured to capture multiple images to generate the sensor data.

[0263] Example 15 includes the device of any of Examples 1 to 14, and further includes a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the gaussian scene representation, the sensor data, or both, via the modem.

[0264] Example 16 includes the device of any of Examples 1 to 15, wherein the one or more processors are integrated in at least one of a laptop computer, a mobile phone, a tablet computer device, or a wearable electronic device.

[0265] Example 17 includes the device of any of Examples 1 to 15, wherein the one or more processors are integrated in an extended reality headset, and wherein the extended reality headset includes at least one display device configured to display one or more views of the scene based on the gaussian scene representation.QUALCOMM Ref. No. 2501500WO- 89 / 98 -

[0266] Example 18 includes the device of any of Examples 1 to 15. wherein the one or more processors are integrated in a vehicle, and wherein the gaussian scene representation is used to generate control signals to the vehicle.

[0267] Example 19 includes the device of any of Examples 1 to 18, wherein the one or more processors are included in an integrated circuit.

[0268] According to Example 20, a method includes obtaining, at one or more processors, sensor data corresponding to a scene modeled by a gaussian scene representation; and selecting, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0269] Example 21 includes the method of Example 20, wherein the sensor data includes light data from a light sensor, and wherein the change in the scene corresponds to a lighting change indicated by the light data.

[0270] Example 22 includes the method of Example 20 or Example 21, wherein the sensor data includes motion data from a motion sensor, and wherein the change in the scene corresponds to detected movement indicated by the motion data.

[0271] Example 23 includes the method of any of Examples 20 to 22, wherein the sensor data includes ranging data from one or more ranging sensors, and wherein the change in the scene corresponds to detected movement indicated by the ranging data.

[0272] Example 24 includes the method of any of Examples 20 to 23, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data; and further including determining an estimated movement based on the first input pixel data and the second input pixel data, wherein the change in the scene corresponds to the estimated movement.

[0273] Example 25 includes the method of any of Examples 20 to 24, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data; and further including determining an estimated lighting change based on the first input pixelQUALCOMM Ref. No. 2501500WO- 90 / 98 -data and the second input pixel data, wherein the change in the scene corresponds to the estimated lighting change.

[0274] Example 26 includes the method of any of Examples 20 to 25, and further includes comparing a movement metric, a lighting change metric, or both, to one or more first thresholds to select the update strategy.

[0275] Example 27 includes the method of any of Examples 20 to 26, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold.

[0276] Example 28 includes the method of any of Examples 20 to 27, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold.

[0277] Example 29 includes the method of any of Examples 20 to 28, wherein the plurality of update strategies includes a full update, a partial update, and no update.

[0278] According to Example 30, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain sensor data corresponding to a scene modeled by a gaussian scene representation; and select, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0279] Example 31 includes the non-transitory computer-readable medium of Example 30, wherein the sensor data includes light data from a light sensor, and wherein the change in the scene corresponds to a lighting change indicated by the light data.

[0280] Example 32 includes the non-transitory’ computer-readable medium of Example 30 or Example 31, wherein the sensor data includes motion data from a motion sensor, and wherein the change in the scene corresponds to detected movement indicated by the motion data.

[0281] Example 33 includes the non-transitory computer-readable medium of any of Examples 30 to 32, wherein the sensor data includes ranging data from one or moreQUALCOMM Ref. No. 2501500WO- 91 / 98 -ranging sensors, and wherein the change in the scene corresponds to detected movement indicated by the ranging data.

[0282] Example 34 includes the non-transitory computer-readable medium of any of Examples 30 to 33, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data, wherein the instructions are further executable to cause the one or more processors to determine an estimated movement based on the first input pixel data and the second input pixel data, and wherein the change in the scene corresponds to the estimated movement.

[0283] Example 35 includes the non-transitory computer-readable medium of any of Examples 30 to 34, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data, wherein the instructions are further executable to cause the one or more processors to determine an estimated lighting change based on the first input pixel data and the second input pixel data, and wherein the change in the scene corresponds to the estimated lighting change.

[0284] Example 36 includes the non-transitory computer-readable medium of any of Examples 30 to 35, wherein the instructions are further executable to cause the one or more processors to compare a movement metric, a lighting change metric, or both, to one or more first thresholds to select the update strategy.

[0285] Example 37 includes the non-transitory computer-readable medium of any of Examples 30 to 36, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold.

[0286] Example 38 includes the non-transitory computer-readable medium of any of Examples 30 to 37, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold.

[0287] Example 39 includes the non-transitory computer-readable medium of any of Examples 30 to 38, wherein the plurality of update strategies includes a full update, a partial update, and no update.QUALCOMM Ref. No. 2501500WO- 92 / 98 -

[0288] According to Example 40, an apparatus includes means for obtaining sensor data corresponding to a scene modeled by a gaussian scene representation; and means for selecting, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

[0289] Example 41 includes the apparatus of Example 40, wherein the sensor data includes light data from a light sensor, and wherein the change in the scene corresponds to a lighting change indicated by the light data.

[0290] Example 42 includes the apparatus of Example 40 or Example 41, wherein the sensor data includes motion data from a motion sensor, and wherein the change in the scene corresponds to detected movement indicated by the motion data.

[0291] Example 43 includes the apparatus of any of Examples 40 to 42, wherein the sensor data includes ranging data from one or more ranging sensors, and wherein the change in the scene corresponds to detected movement indicated by the ranging data.

[0292] Example 44 includes the apparatus of any of Examples 40 to 43, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data; and further comprising means for determining an estimated movement based on the first input pixel data and the second input pixel data, wherein the change in the scene corresponds to the estimated movement.

[0293] Example 45 includes the apparatus of any of Examples 40 to 44, wherein the sensor data is obtained from an image sensor, the sensor data including first input pixel data and second input pixel data that is obtained after the first input pixel data; and further comprising means for determining an estimated lighting change based on the first input pixel data and the second input pixel data, wherein the change in the scene corresponds to the estimated lighting change.

[0294] Example 46 includes the apparatus of any of Examples 40 to 45, and further includes means for comparing a movement metric, a lighting change metric, or both, to one or more first thresholds to select the update strategy.QUALCOMM Ref. No. 2501500WO- 93 / 98 -

[0295] Example 47 includes the apparatus of any of Examples 40 to 46, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold.

[0296] Example 48 includes the apparatus of any of Examples 40 to 47, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold.

[0297] Example 49 includes the apparatus of any of Examples 40 to 48, wherein the plurality of update strategies includes a full update, a partial update, and no update.

[0298] Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

[0299] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computingQUALCOMM Ref. No. 2501500WO- 94 / 98 -device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

[0300] The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims

QUALCOMM Ref. No. 2501500WO- 95 / 98 - WHAT IS CLAIMED IS:

1. A device comprising:a memory configured to store a gaussian scene representation; andone or more processors coupled to the memory and configured to:obtain sensor data corresponding to a scene modeled by the gaussian scene representation; andselect, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.

2. The device of claim 1, wherein the sensor data includes light data from a light sensor, and wherein the change in the scene corresponds to a lighting change indicated by the light data.

3. The device of claim 1, wherein the sensor data includes motion data from a motion sensor, and wherein the change in the scene corresponds to detected movement indicated by the motion data.

4. The device of claim 3, wherein the sensor data includes ranging data from one or more ranging sensors, and wherein the change in the scene corresponds to detected movement indicated by the ranging data.

5. The device of claim 1. wherein:the sensor data is obtained from an image sensor and includes first input pixel data and second input pixel data that is obtained after the first input pixel data;the one or more processors are configured to determine an estimated movement based on the first input pixel data and the second input pixel data; and the change in the scene corresponds to the estimated movement.

6. The device of claim 1, wherein:QUALCOMM Ref. No. 2501500WO- 96 / 98 - the sensor data is obtained from an image sensor and includes first input pixel data and second input pixel data that is obtained after the first input pixel data;the one or more processors are configured to determine an estimated lighting change based on the first input pixel data and the second input pixel data; andthe change in the scene corresponds to the estimated lighting change.

7. The device of claim 1, wherein the one or more processors are configured to compare a movement metric, a lighting change metric, or both, to one or more first thresholds to select the update strategy.

8. The device of claim 1, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold.

9. The device of claim 1, wherein the update criterion includes a metric associated with the change in the scene being greater than a first threshold and less than a second threshold.

10. The device of claim 1, wherein the plurality’ of update strategies includes a full update, a partial update, and no update.

11. The device of claim 1, further comprising one or more sensors coupled to the one or more processors and configured to generate the sensor data.

12. The device of claim 11, wherein the one or more sensors include a motion sensor configured to generate motion data that is included in the sensor data.

13. The device of claim 11, wherein the one or more sensors include a light sensor configured to generate light data that is included in the sensor data.

14. The device of claim 11, further comprising one or more cameras coupled to the one or more processors and configured to capture multiple images to generate the sensor data.QUALCOMM Ref. No. 2501500WO- 97 / 98 - 15. The device of claim 1, further comprising a modem coupled to the one or more processors, wherein the one or more processors are configured to obtain the gaussian scene representation, the sensor data, or both, via the modem.

16. The device of claim 1, wherein the one or more processors are integrated in at least one of a laptop computer, a mobile phone, a tablet computer device, or a wearable electronic device.

17. The device of claim 1. wherein the one or more processors are integrated in an extended reality headset, and wherein the extended reality headset includes at least one display device configured to display one or more views of the scene based on the gaussian scene representation.

18. The device of claim 1, wherein the one or more processors are integrated in a vehicle, and wherein the gaussian scene representation is used to generate control signals to the vehicle.

19. The device of claim 1, wherein the one or more processors are included in an integrated circuit.

20. A method comprising:obtaining, at one or more processors, sensor data corresponding to a scene modeled by a gaussian scene representation; andselecting, from among a plurality of update strategies, an update strategy for the gaussian scene representation based on whether the sensor data indicates a change in the scene that satisfies an update criterion.