Object insertion within a gaussian scene representation
The method of using object templates for gaussian scene representations addresses the computational challenges by reducing input data and preventing object collisions, improving efficiency and resource utilization.
Patent Information
- Application Number
- PCT/US2026/024904
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-09-19
- Filing Date
- 2026-04-23
- Publication Date
- 2026-11-19
AI Technical Summary
Gaussian scene representations require significant computing resources for rendering views due to the large number of gaussians involved, posing a challenge for their adoption in various applications.
A method and apparatus for inserting objects into a gaussian scene representation by using object templates, reducing the complexity and computing resources required by identifying and adding gaussians based on object templates, and incorporating collision detection to prevent overlapping objects.
Reduces the input data needed for object insertion and minimizes collisions between objects in gaussian scene representations, enhancing efficiency and resource utilization.
Smart Images

Figure US2026024904_19112026_PF_FP_ABST
Abstract
Description
QUALCOMM Ref. No. 2507839WO- 1 / 107 -OBJECT INSERTION WITHIN A GAUSSIAN SCENE REPRESENTATIONI. Cross-Reference to Related Applications
[0001] The present application claims priority from the commonly owned U. S.Provisional Patent Application No. 63 / 804,910, filed May 13, 2025, U. S. Provisional Patent Application No. 63 / 831,815, filed June 27, 2025. U. S. Provisional Patent Application No. 63 / 885,035, filed September 19, 2025, U. S. Provisional Patent Application No. 63 / 885,042, filed September 19, 2025, U. S. Provisional Patent Application No. 63 / 885,033, filed September 19, 2025, U. S. Provisional Patent Application No. 63 / 885,048, filed September 19, 2025, U. S. Provisional Patent Application No. 63 / 825,138. filed June 17, 2025, and U. S. Provisional Patent Application No. 63 / 820,238, filed June 9, 2025, the contents of which are expressly incorporated herein by reference in their entirety.IL Field
[0002] The present disclosure is generally related to gaussian scene representation.III. Description of Related Art
[0003] Computer science uses various techniques to generate three dimensional (3D) models or four dimensional (4D) models (where time is a fourth dimension) of a scene based on images of the scene. One example of a scene model used in computer science is a gaussian scene representation, which uses primitive elements commonly referred to as “gaussians”.
[0004] Various rendering operations (sometimes referred to as ‘‘gaussian splatting"’) are used to generate views based on a gaussian scene representation. The views can include views of the scene from various perspectives, and in particular, the gaussian scene representation can be used to generate novel views of the scene (e.g., views of the scene that do not correspond directly to any particular image used to generate the gaussian scene representation). For example, a gaussian scene representation can be used to generate images or sequences of images (e.g., videos) that simulate how the scene would appear from a different vantage point, perspective, or lighting condition.QUALCOMM Ref. No. 2507839WO- 2 / 107 - Gaussian scene representations can be used to generate imagery for computer vision, graphics, video games, extended reality visualization. 3D reconstruction, security and surveillance, medical imaging, computer assisted navigation, as well as other uses.
[0005] Gaussian scene representations can include a large number (e.g., hundreds of thousands or millions) of gaussians to represent a scene. Rendering views of a scene represented by such a large number of gaussians takes significant computing resources, which represents a challenge to adoption of gaussian scene representations for some use cases.IV. Summary
[0006] According to one implementation of the present disclosure, a device includes a memory configured to store a gaussian scene representation representing a scene. The device also includes one or more processors coupled to the memory and configured to obtain input data that includes a request to modify the gaussian scene representation to add a first object in a view of the scene. The one or more processors are configured to identify, based on the request, an object template associated with the first object. The one or more processors are configured to obtain, based on the object template, first gaussians that represent the first object. The one or more processors are configured to add the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0007] According to another implementation of the present disclosure, a method includes obtaining input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene. The method also includes identifying, based on the request, an object template associated with the first object. The method also includes obtaining, based on the object template, first gaussians that represent the first object. The method also includes adding the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0008] According to another implementation of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene. TheQUALCOMM Ref. No. 2507839WO- 3 / 107 -instructions also cause the one or more processors to identify, based on the request, an object template associated with the first object. The instructions also cause the one or more processors to obtain, based on the object template, first gaussians that represent the first object. The instructions also cause the one or more processors to add the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0009] According to another implementation of the present disclosure, an apparatus includes means for obtaining input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene. The apparatus also includes means for identifying, based on the request, an object template associated with the first object. The apparatus also includes means for obtaining, based on the object template, first gaussians that represent the first object. The apparatus also includes means for adding the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0010] Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.V. Brief Description of the Drawings
[0011] FIG. 1 is a block diagram of an illustrative system operable to insert an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0012] FIG. 2 is a diagram of particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.
[0013] FIGs. 3A, 3B, and 3C, together, illustrate particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.
[0014] FIGs. 4A, 4B, and 4C, together, illustrate particular aspects of the system of FIG. 1, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2507839WO- 4 / 107 -
[0015] FIG. 5 is a diagram illustrating particular aspects of a first example of object insertion within a gaussian scene representation by the system of FIG. 1, in accordance with some examples of the present disclosure.
[0016] FIG. 6 is a diagram illustrating particular aspects of a second example of object insertion within a gaussian scene representation by the system of FIG. 1, in accordance with some examples of the present disclosure.
[0017] FIG. 7 is a flow chart illustrating particular aspects of a method associated with inserting an object within a gaussian scene representation that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure.
[0018] FIG. 8 illustrates an example of an integrated circuit operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0019] FIG. 9 is a diagram of a mobile device operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0020] FIG. 10 is a diagram of a mixed reality or augmented reality glasses device operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0021] FIG. 11 is a diagram of a wearable electronic device operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0022] FIG. 12 is a diagram of a voice-controlled speaker system operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0023] FIG. 13 is a diagram of a camera operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.QUALCOMM Ref. No. 2507839WO- 5 / 107 -
[0024] FIG. 14 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0025] FIG. 15 is a diagram of a first example of a vehicle operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0026] FIG. 16 is a diagram of a second example of a vehicle operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.
[0027] FIG. 17 is a block diagram of a particular illustrative example of a device that is operable to perform operations associated with inserting an object within a gaussian scene representation, in accordance with some examples of the present disclosure.VI. Detailed Description
[0028] Use of a gaussian scene representation in a situation in which the gaussian scene representation is periodically or occasionally updated to add new objects presents significant challenges due to the complexity associated with modifying gaussian scene representations. Aspects disclosed herein reduce the complexity associated with adding objects within an existing gaussian scene representation by using object templates. For example, based on a request (e.g., an instruction received from a user) to modify a gaussian scene representation representing a scene to add an object in a view of the scene, an object template associated with the object is identified. In some cases, the object template is retrieved from a gaussian object template library. In other cases, the object template is generated. Gaussians that represent the object are obtained from the object template and added to the gaussian scene representation, producing a modified gaussian scene representation that includes the object in the view of the scene.
[0029] In some conventional gaussian systems, updating gaussian scenes with new objects can require a significant amount of input data (e.g., corresponding to gaussians used to represent the new object, coordinate locations within the scene where the newQUALCOMM Ref. No. 2507839WO- 6 / 107 -object is to be located, scale parameter(s) of the new object, etc.) to be provided. The disclosed aspects provide a gaussian object editor that adds gaussians representing an object to a gaussian scene representation based on, in some cases, less input data, as compared to the conventional systems. The gaussian object editor can receive input data that includes a request to add an object to the gaussian scene representation and, when not provided in the request, can obtain (e.g.. retrieve or generate) information such as gaussians to use to represent the object, a scale of the object within the scene, a location of the object within the scene, or any combination thereof. As a result, a technical advantage of the disclosed aspects is a reduction of input data provided in a request to add an obj ect to a view of a scene represented by a gaussian scene representation.
[0030] Additionally, in some scenes, newly added objects in a scene can unintentionally overlap with previously existing objects in the scene. In some embodiments, when the gaussians representing the object are added to the gaussian scene representation, collision detection is performed to prevent newly added objects in the scene from overlapping with previously existing objects in the scene. When a collision is detected (e.g., based on one or more error metrics), error correction can be performed to correct (e.g., adjust, update, or modify) a position of the object, a spatial orientation of the object, a size of the object, or any combination thereof. As a result, another technical advantage of the disclosed aspects is a reduction of collisions between objects when adding an object to a view of a scene represented by a gaussian scene representation.
[0031] In the present discussion, a ‘“gaussian” refers to a graphical element of a gaussian scene representation. Each gaussian is specified by a set of parameters indicating characteristics, described further below, of the gaussian in scene space. In some examples, gaussians have a generally ellipsoidal 3D shape in the scene space, with a maximum opacify at a center position of the gaussian. In some examples, the opacity diminishes following an equation similar to a gaussian probability distribution.
[0032] In a particular aspect, the parameters specifying a set of gaussians include:• a position vector, p,, which indicates the center position of a gaussian with index• an opacify parameter, tt6 [0,1], which indicates a maximum opacify of the gaussian;QUALCOMM Ref. No. 2507839WO- 7 / 107 - • orientation parameters. qf. which can include a quaternion defining the spatial orientation (rotation) of the gaussian;• a scale vector, sf, which defines a size and shape of the ellipsoid of the gaussian;and• a set of spherical harmonics (SH) coefficients, H,. which can include a matrix with columns composed of 3D color vectors (in portions of this disclosure Hfis referred to as an SH coefficient array).
[0033] In some implementations, not all of the parameters described above are used to specify a gaussian. For example, a reduced parameter set may include approximately 14 parameters, such as the position vector (3 parameters), the opacity parameter (1 parameter), the orientation parameters (4 parameters), and the scale vector (3 parameters), along with a single base color value (3 parameters for red-green-blue (RGB)) rather than the full set of spherical harmonics coefficients. In such implementations, the color parameters associated with spherical harmonics coefficients may be omitted or simplified, such as by using only a zeroth-order spherical harmonic (which corresponds to a view-independent color) rather than higher-order spherical harmonics that capture view-dependent color variations. Using a reduced parameter set can decrease memory’ usage and computational complexity associated with storing, updating, and rendering the gaussian scene representation, which may be beneficial in resource-constrained environments or when view-dependent color effects are not required.
[0034] In some implementations, the gaussian scene representation may use two-dimensional (2D) gaussians rather than three-dimensional (3D) gaussians. A 2D gaussian collapses the 3D volume into a 2D oriented planar disk embedded in 3D space. Unlike 3D gaussians, which distribute density throughout a model the complete volumetric ellipsoid, 2D gaussians distribute densities within a planar disk, with the normal direction defined as the direction of steepest change of density. This approach may provide improved view-consistent geometry and better alignment with thin surfaces.QUALCOMM Ref. No. 2507839WO- 8 / 107 -
[0035] A 2D gaussian primitive may be characterized by a center point, an orientation of the planar disk, and two scaling factors controlling the variance along tangential directions of the disk. The normal of the 2D gaussian primitive is perpendicular to the disk plane. The orientation of the planar disk may be defined by rotation parameters (e.g., a quaternion or rotation matrix) or by two tangent vectors defining the disk plane. The normal of the 2D gaussian primitive may be derived from the rotation parameters or defined by a cross-product of the two tangent vectors. In such implementations, the 2D gaussian primitive may have fewer parameters than a corresponding 3D gaussian, as one dimension of the scale vector is effectively zero (corresponding to the normal direction of the planar disk). Each 2D gaussian primitive may also have an opacity parameter and view-dependent appearance parameters, such as spherical harmonics coefficients or a simplified base color value.
[0036] Using 2D gaussians may be beneficial for applications requiring accurate surface reconstruction, as the 2D gaussian primitives can be more tightly aligned to object surfaces than 3D gaussians. Additionally, 2D gaussians may provide more accurate geometry' representation during rendering, as they enable perspective-correct splatting through explicit ray-splat intersection calculations.
[0037] In some implementations, the gaussian scene representation may be integrated with a mesh-based representation. In such implementations, each gaussian of the gaussian scene representation can be used to represent a triangular face of a mesh, enabling conversion between gaussian representations and mesh representations. This approach provides a one-to-one mapping between gaussians and triangular mesh faces that is differentiable, enabling the gaussian scene representation to be modified in a manner similar to modifying a mesh.
[0038] To convert a 2D gaussian to a triangular mesh face, an eigen-decomposition of the covariance matrix of the 2D gaussian may be performed. The covariance matrix S can be decomposed as X = RART, where R is a matrix of eigenvectors (el, e2, e3) and A is a diagonal matrix of eigenvalues (XI, X2, X3). The two largest eigenvalues ( I, X2) correspond to tangent directions of a surface plane, while the smallest eigenvalue (X3) corresponds to the normal direction. A triangular face can then be constructed using the mean of the gaussian as a center point and defining triangle vertices along the tangentQUALCOMM Ref. No. 2507839WO- 9 / 107 -directions. For example, triangle vertices may be defined as vl = p + alel, v2 = p + a2e2. and v3 = p - alel, where al and a2 are scale factors proportional to square roots of their respective eigenvalues I and 2 and p is the center point corresponding to the mean of the 2D gaussian. This creates a triangle aligned with the gaussian's orientation and sized according to its spread. The color of each vertex of the triangle can be calculated from the color parameters of the gaussian.
[0039] In such mesh-integrated implementations, the mesh vertices parameterize the mean and covariance matrix of the gaussian. When the mesh is modified, the gaussian parameters update accordingly. This relationship enables automatic adjustments in position, scale, and rotation of the gaussians when the mesh is animated or otherwise modified. During rendering, the gaussians may be mapped to triangles and the triangles rendered. As a result, when a partial update is performed on a subset of gaussians of the gaussian scene representation, only the corresponding subset of triangles is updated due to the one-to-one mapping between gaussians and triangular faces.
[0040] Throughout this disclosure, references to gaussians, gaussian scene representations, and operations thereon (such as partial updates, full updates, rendering, and the like) apply to 3D gaussians, 2D gaussians, mesh-integrated gaussians, and combinations thereof, unless otherwise specified. The specific parameters, rendering techniques, and update mechanisms may vary depending on the type of gaussian representation used, as described herein.
[0041] The following elements may be determined based on the parameters above:• S is a 3x3 matrix similar to the covariance matrix in gaussian probability distributions.• c is a 3D view-dependent color vector for each gaussian, computed using the set of spherical harmonics coefficients of the gaussian.
[0042] The three dimensions of the color vector c may correspond to three different color component values, such as a red value, a green value, and a blue value, that specify a color for the gaussian. The color vector c is view-dependent in the sense that the gaussian may appear to have different colors depending on a position from which the gaussian is viewed, e.g., due to reflections. The view-dependent color vector c forQUALCOMM Ref. No. 2507839WO- 10 / 107 -the gaussian may be derived from a camera position f, the center position p(of the gaussian. and the SH coefficient array H, of the gaussian. The camera position f and the center position p, may each be a vector of spherical coordinates (0, <p). Thus, a viewing direction vector comprising a 9 and a <f> coordinate value that defines a viewing direction from the camera position f to the center position p(may be described as (f-p;). Therefore, the view-dependent color vector c for a gaussian i having central position p, from camera position f may be written as c / (f - p;, H;) or simply c(f - p, H).
[0043] In certain 3D Gaussian Splatting (3DGS) processes, H;includes up to 16 spherical harmonics coefficients per color vector component (e.g., 16 spherical harmonics components for red, 16 spherical harmonics components for green, and 16 spherical harmonics components for blue). A Tenderer may calculate a value of a color component in the view-dependent color vector c based on a linear combination LC of a set of terms Ti... Ymax scaled by the spherical harmonics coefficients in H, for the color component. That is, the Tenderer may calculate the linear combination LCtj for gaussian i and color component j as:Equation (1) may be rewritten as follows:where II(;1, H; 2, etc., are spherical harmonics coefficients for gaussian i in H;for the color component j. Each of the terms in vector y may be defined by the following spherical harmonics equation:In equation (3) above, the values (0, ) specify the viewing direction vector derived from f and pi. The values I and m are the degree and order of the spherical harmonics equation,QUALCOMM Ref. No. 2507839WO- 11 / 107 -where - I < m < I. Thus, in an example where there are 16 spherical harmonics coefficients, the maximum value of I is 3, and when I is 0. m is 0; when I is 1, m is 1. 0, -1; when / is 2, m is 2, 1, 0, -1, -2, and so on. Thus, Yi may correspond to / being 0 and m being 0, YT. may correspond to I being 1 and m being 1, E may correspond to I being 1 and m being 0, E may correspond to I being 1 and m being -1, Fs may correspond to I being 2 and m being 2, and so on. With respect to conventional RGB values, the Tenderer may calculate the value of a color component in the view-dependent color vector c by applying a sigmoid function to LC and multiplying by a dynamic range scaling factor, such as 255.
[0044] To generate a rendered image based on a set of gaussians, a Tenderer may determine color values of pixels in the rendered image. A rendered image is a type of digital image that is composed of a grid of individual pixels, each with its own color vector. Each pixel is associated with a camera ray extending from the pixel through a virtual aperture of a virtual camera. In conventional 3DGS rendering, for each pixel of the rendered image, the Tenderer may identify gaussians that intersect the camera ray associated with the pixel and sort the identified gaussians based on a depth from the virtual camera. For each of the identified gaussians, the Tenderer calculates an opacify value for a location within the gaussian based on the position vector of the gaussian, the maximum opacify of the gaussian, the orientation vector of the gaussian, and the scale vector of the gaussian. The Tenderer calculates the opacify' value (a) for 3D location x along the camera ray within a gaussian i (i.e., a,(x)) as follows:1,2,, N, (4)
[0045] In equation (4) and throughout this disclosure, the exp function is ex, where e is Euler’s number, (,.)Tindicates matrix transposition, [...]-1indicates a matrix inversion, and N indicates the total quantify of identified gaussians that intersect the camera ray. In general terms, equation (4) scales the maximum opacify value (p) of the gaussian based on a gaussian distribution function according to a distance of the location x from a center pi of the gaussian. The location x may be defined as a vector of coordinates, e.g., spherical coordinates, Cartesian coordinates, etc.QUALCOMM Ref. No. 2507839WO- 12 / 107 -
[0046] When processing a location x within a gaussian, the Tenderer also calculates a view-dependent color vector c for the gaussian. The view-dependent color vector c may include color component values that define a color for the gaussian, such as red, green, and blue values. Considering a virtual camera with a focal point (i.e., virtual aperture) at position f, the Tenderer may calculate an alpha-scaled color vector for gaussian i. The alpha-scaled color vector n specifies a contribution of a location x within the gaussian i to a color of a pixel. The alpha-scaled color vector may be calculated using the following formula:1,2, •• -, N. (5)
[0047] Thus, equation (5) determines an alpha-scaled color vector r for a location x viewed from position f by scaling the view-dependent color vector c of a gaussian i that contains the location x based on the previously determined opacity value for the location x (i.e., «i(x)). The view-dependent color vector c may be determined using a spherical harmonics equation based on the SH coefficient matrix Hi of the gaussian and the differences between coordinates the center position of the gaussian (f - pi) as described above. The view-dependent color vector of a gaussian may be dependent on the viewing position f because of the directions of light sources, which can cause shadows, reflections, and other optical effects.
[0048] Depending on the size and orientation of a gaussian and a discretization of the 3D space into discrete locations, there may be multiple locations along a camera ray that intersect the same gaussian. Hence, the Tenderer may calculate multiple opacity7values and color vectors for the same gaussian. The Tenderer may use these opacity values and color values in a rendering equation for determining a color vector of a pixel of a rendered image. However, for ease of explanation, this disclosure and the equations thereof assume that there is only one location per gaussian.
[0049] An advantage of using the formula of equation (4) is that mapping gaussians to the 2D rendered image can be approximated by a 2D gaussian with a 2x2 covariance matrix that can be computed asS2D= J W S3DWTJT, (6)QUALCOMM Ref. No. 2507839WO- 13 / 107 -where W is a matrix defined by the camera’s image-generation transformation, and J is the Jacobian matrix defined by an affine approximation of the projective camera transformation.
[0050] For ease of explanation, this disclosure omits some details about how rendering is discretized for image pixels, and considers that, for all gaussians, there are opacity values aL[m, n] and color vectors c([m, n] defined for each image pixel with integer image coordinates [m, nJ. The image coordinates of a pixel indicate a 2D location of the pixel within a rendered image. In other words, each pixel of a rendered image is identified by a set of coordinates m and n. Due to the transparency of gaussians, before a scene is rendered, the gaussians to be rendered are sorted according to their distance from the camera position f. Formally, a sorting sequence s[ / c] of indexes of gaussians is determined according to increasing distance, such that1,2, -,1V. (7)In equation (7), the double straight brackets (i.e., ||... ||) indicate Euclidean distance.
[0051] Thus, the Tenderer processes locations (e.g., gaussians) along the camera ray associated with a pixel in an order from closest to the virtual camera to further from the virtual camera. Hence, if the opacity values of locations within the gaussians intersected by the camera ray associated with the pixel reach a level corresponding to complete opacity, later gaussians that intersect the camera ray do not contribute to the color vector of the pixel. Hence, the Tenderer can skip processing of locations along the camera ray associated with the pixel that intersect the later gaussians.
[0052] The Tenderer may compute a final color vector for a pixel based on the opacity values and the view-dependent color values for the locations along the camera ray associated with the pixel. For instance, after sorting the gaussians based on depth, the Tenderer may compute a rendered color vector r for the pixel according to the following equation, which corresponds to the computer graphics technique of alpha-blending:QUALCOMM Ref. No. 2507839WO- 14 / 107 -
[0053] In equation (8), N indicates a total quantity of graphics elements (e.g., gaussians) along the camera ray associated with the pixel at coordinates [m, n]. The termcs[fc] [m>n] indicates a color vector for a distance-sorted graphics element having index value k in a sorting order of the graphics elements. In an example where the graphics element is a gaussian, cs[fc] [m, n] may be a view-dependent color vector for the gaussian. The term as[kj [m, n] indicates an opacity value for the distance-sorted graphics element having an index value k in the sorting order. Similarly, as[q [m, n] indicates an opacity value for the distance-sorted graphics element having an index value I in the sorting order. Note that in equation (8), the termis calculated individually for each of index values k=l to N so that the scaled color vector for the graphics element with index value k (cs[fc] [m, n] as[fe] [m, n]) is further scaled based on the opacity values of graphics elements closer to the camera position than the graphics element with index value k. Thus, a Tenderer using equation (8) sorts the graphics elements based on depth from the camera position.
[0054] In complex scenes, like those used in computer games, most of the graphics elements are assumed to be opaque (i.e., cq = 1 in the object’s surface), so equation (8) can be implemented in a massively parallel manner by storing, for each pixel, the minimum depth of the elements already rendered, and only updating the color vector of a pixel if the newly visited element has smaller depth (Z-buffer updates). This is what enables current graphics processing units (GPUs) to render complex scenes at high resolutions and frame rates. On the other hand, when there is transparency, equation (8) defines a strictly sequential order for processing elements, and the only way to parallelize rendering is to create sets of non-overlapping elements and implement sequential alpha-blending on each set.
[0055] This made introduction of transparency in games challenging, but transparency is needed to represent many features, like flames and smoke, clouds, ‘‘force fields,'’ and so on. One proposed empiric solution to avoid sorting, named Order-IndependentQUALCOMM Ref. No. 2507839WO- 15 / 107 - Transparency (OIT), is to approximate equation (8) with a weighted sum. There are several OIT variants, such as to replace equation (8) with the following equation for determining a rendered color vector f for a pixel at coordinates [m, n] of a rendered image:where ztis the distance to a virtual camera, c0[m, n] is the background color for the pixel at position [m,n] in the rendered image, and w(z,a) is a function that decreases with distance, so that objects nearer to the camera are assigned larger weights in the alphanormalized weighted sum of equation (9). The Tenderer may calculate the term a[m, n] using the following equation:Note that equation (9) is a weighted sum of a first value and a second value, where the first value is the background color vector co for the camera ray associated with pixel \m,n\ scaled by the accumulated opacity (a[m, n]) of the camera ray, and the second term is a composited color vector scaled by the accumulated transparency (1 — a[m, n]) of the camera ray. In the OIT rendering of equation (9), there are two summation operations ■■■ f°rdetermining a color vector for a pixel, and since addition is commutative, the terms in the summation operations can be computed in any order. However, the final color vector for a pixel can only be computed after the two sums are known, which can be done with a single pass over all pixels, performing the divisions in equation (9). In other w ords, the Tenderer can perform the summation operations for all pixels, and then, after the summation operations for all pixels are complete, perform the division operations for all pixels. Since OIT uses the term w(z, a) instead of the actual opacity values of closer graphics elements, OIT may eliminate the need for sorting the graphics elements based on depth during rendering. However, use of the term iv(z, a) instead of the actual opacity values of closer graphics elements means that OIT only provides an approximation of alpha blending and does not model truly opaque surfaces.QUALCOMM Ref. No. 2507839WO- 16 / 107 -
[0056] Because OIT allows rendering transparency without sorting, OIT may be considered for eliminating the computationally expensive sorting phase of gaussian splatting rendering. However, while OIT methods have been extensively used in computer games for several years, the OIT methods were developed to be used with conventional computer graphics, and were only meant to represent semi-transparent elements. Furthermore, the OIT methods are not expected to generate good approximations since the OIT methods are used with elements of shape and appearance where differences are difficult to see, like flames and smoke. Finally, human intervention may be needed to identify when the OIT methods are not working correctly, and to fix problems.
[0057] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms '‘a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate. FIG. 1 depicts a device 102 including one or more processors (“processor(s)” 130 of FIG. 1), w hich indicates that in some implementations the device 102 includes a single processor 130 and in other implementations the device 102 includes multiple processors 130. For ease of reference herein, such features are generally introduced as “one or more” features and may subsequently be referred to in the singular or optional plural (as indicated by “(s)”) unless aspects related to multiple of the features are being described.
[0058] As used herein, the terms “comprise.” “comprises.” and “comprising” may be used interchangeably with "include.” “includes,” or "including.” Additionally, the term “wherein” may be used interchangeably with "where.” As used herein, "exemplary" indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g.. "first,” “second,” “third.” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes theQUALCOMM Ref. No. 2507839WO- 17 / 107 -element from another element having a same name (but for use of the ordinal term). As used herein, the term '“set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.
[0059] In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein (e.g., when no particular one of the features is being referenced), the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. For example, referring to FIG. 3, multiple sensors are illustrated and associated with reference numbers 302A and 302B. When referring to a particular one of these sensors, such as a sensor 302A, the distinguishing letter “A” is used. However, when referring to any arbitrary one of these sensors or to these sensors as a group, the reference number 302 is used without a distinguishing letter.
[0060] The phrase “corresponds to” as used herein is a relational phrase indicating correspondence, equivalence, or matching. For example, if A corresponds to B. then A is B, there is a mapping between A and B, or A matches B. The phrase “‘is associated with” as used herein is a broad relational phrase indicating a looser or more general relationship such as, for example, a categorical relationship (e.g., A is part of or belongs to B). a causal relationship (e.g., A causes B), a logical relationship (e.g., If A then B), correlation (e.g., when A is present B is present), a structural relationship (e.g., the B that is coupled to A), and other possible relationships. Correspondence always includes association; whereas association can, but does not always, indicate correspondence. For example, if A is associated with B, there can be a mapping between A and B that could be described as correspondence between A and B.
[0061] As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled.” and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., aQUALCOMM Ref. No. 2507839WO- 18 / 107 -wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, “directly coupled” may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.
[0062] In the present disclosure, terms such as “obtaining,” “determining,” “calculating,” “estimating,” “shifting,” “adjusting,” etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “obtaining,” “generating,” “calculating.” “estimating,” “using,” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “obtaining,” “generating,” “calculating,” “estimating,” or “detemiining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, receiving, or accessing the parameter (or signal) that is already generated, such as by another component or device.
[0063] Within this disclosure, in some cases, different entities (which are variously referred to as “components,” “units,” “devices,” etc.) are described or claimed as “configured” to perform one or more tasks or operations. This formulation-] entity | configured to [perform one or more tasks]-is used herein to refer to structure (i.e., something physical, such as an electronic circuit). More specifically, this formulation is used to indicate that this structure is arranged to perform the one or more tasks during operation. A structure can be said to be “configured to” perform some task even if the structure is not currently being operated. A “memory device configured to store data” is intended to cover, for example, an integrated circuit that has circuitry that stores data during operation, even if the integrated circuit in question is not currently being used (e.g., a power supply is not coupled to it). Thus, an entity described or recited as “configured to” perform some task refers to something physical, such as a device,QUALCOMM Ref. No. 2507839WO- 19 / 107 -circuit, memory storing program instructions executable to implement the task, etc. This phrase is not used herein to refer to something intangible. Further, the term '“configured to” is not intended to mean “configurable to.” An unprogrammed field-programmable gate array, for example, would not be considered to be “configured to” perform some specific function, although it could be “configurable to” perform that function after programming. Additionally, reciting in the appended claims that a structure is “‘configured to” perform one or more tasks is expressly intended not to be interpreted as having means-plus-function elements.
[0064] As used herein, the term “machine learning” should be understood to have any of its usual and customary meanings within the fields of computers science and data science, such meanings including, for example, processes or techniques by which one or more computers can learn to perform some operation or function without being explicitly programmed to do so. As a typical example, machine learning can be used to enable one or more computers to analyze data to identify patterns in data and generate a result based on the analysis. For certain types of machine learning, the results that are generated include data that indicates an underlying structure or pattern of the data itself. Such techniques, for example, include so called “clustering” techniques, which identify clusters (e.g.. groupings of data elements of the data).
[0065] For certain types of machine learning, the results that are generated include a data model (also referred to as a “machine-learning model” or simply a “model”). Typically, a model is generated using a first data set to facilitate analysis of a second data set. For example, a first portion of a large body of data may be used to generate a model that can be used to analyze the remaining portion of the large body of data. As another example, a set of historical data can be used to generate a model that can be used to analyze future data.
[0066] Since a model can be used to evaluate a set of data that is distinct from the data used to generate the model, the model can be viewed as a type of software (e.g., instructions, parameters, or both) that is automatically (e.g., based on detection of one or more criteria and without direct instruction from a user) generated by the computer(s) during the machine learning process. As such, the model can be portable (e.g., can be generated at a first computer, and subsequently moved to a second computer for furtherQUALCOMM Ref. No. 2507839WO- 20 / 107 -training, for use, or both). Additionally, a model can be used in combination with one or more other models to perform a desired analysis. To illustrate, first data can be provided as input to a first model to generate first model output data, which can be provided (alone, with the first data, or with other data) as input to a second model to generate second model output data indicating a result of a desired analysis. Depending on the analysis and data involved, different combinations of models may be used to generate such results. In some examples, multiple models may provide model output that is input to a single model. In some examples, a single model provides model output to multiple models as input.
[0067] Examples of machine-learning models include, without limitation, perceptrons, neural networks, support vector machines, regression models, decision trees, Bayesian models, Boltzmann machines, adaptive ncuro-fuzzy inference systems, as well as combinations, ensembles and variants of these and other types of models. Variants of neural networks include, for example and without limitation, prototypical networks, autoencoders, transformers, self-attention networks, convolutional neural networks, deep neural networks, deep belief networks, etc. Variants of decision trees include, for example and without limitation, random forests, boosted decision trees, etc.
[0068] Since machine-learning models are generated by computer(s) based on input data, machine-learning models can be discussed in terms of at least two distinct time windows - a creation / training phase and a runtime phase. During the creation / training phase, a model is created, trained, adapted, validated, or otherwise configured by the computer based on the input data (which in the creation / training phase, is generally referred to as “training data”). Note that the trained model corresponds to software that has been generated and / or refined during the creation / training phase to perform particular operations, such as classification, prediction, encoding, or other data analysis or data synthesis operations. During the runtime phase (or “inference” phase), the model is used to analyze input data to generate model output. The content of the model output depends on the ty pe of model. For example, a model can be trained to perform classification tasks or regression tasks, as non-limiting examples. In some implementations, a model may be continuously, periodically, or occasionally updated, in which case training time and runtime may be interleaved or one version of the modelQUALCOMM Ref. No. 2507839WO- 21 / 107 -can be used for inference while a copy is updated, after which the updated copy may be deployed for inference.
[0069] In some implementations, a previously generated model is trained (or re-trained) using a machine-learning technique. In this context, “training'’ refers to adapting the model or parameters of the model to a particular data set. Unless otherwise clear from the specific context, the term “training” as used herein includes “re-training” or refining a model for a specific data set. For example, training may include so called “transfer learning."’ In transfer learning a base model may be trained using a generic or typical data set, and the base model may be subsequently refined (e.g., re-trained or further trained) using a more specific data set.
[0070] A data set used during training is referred to as a “training data set” or simply “training data”. The data set may be labeled or unlabeled. “Labeled data” refers to data that has been assigned a categorical label indicating a group or category with which the data is associated, and “unlabeled data” refers to data that is not labeled. Typically, “supervised machine-learning processes” use labeled data to train a machine-learning model, and “unsupervised machine-learning processes” use unlabeled data to train a machine-learning model: however, it should be understood that a label associated with data is itself merely another data element that can be used in any appropriate machinelearning process. To illustrate, many clustering operations can operate using unlabeled data; however, such a clustering operation can use labeled data by ignoring labels assigned to data or by treating the labels the same as other data elements.
[0071] Training a model based on a training data set generally involves changing parameters of the model with a goal of causing the output of the model to have particular characteristics based on data input to the model. To distinguish from model generation operations, model training may be referred to herein as optimization or optimization training. In this context, “optimization” refers to improving a metric, and does not mean finding an ideal (e.g., global maximum or global minimum) value of the metric. Examples of optimization trainers include, without limitation, back-propagation trainers, derivative free optimizers (DFOs), and extreme learning machines (ELMs). As one example of training a model, during supervised training of a neural network, an input data sample is associated with a label. When the input data sample is provided toQUALCOMM Ref. No. 2507839WO- 22 / 107 -the model, the model generates output data, which is compared to the label associated with the input data sample to generate an error value. Parameters of the model are modified in an attempt to reduce (e.g., optimize) the error value. As another example of training a model, during unsupervised training of an autoencoder, a data sample is provided as input to the autoencoder, and the autoencoder reduces the dimensionality of the data sample (which is a lossy operation) and attempts to reconstruct the data sample as output data. In this example, the output data is compared to the input data sample to generate a reconstruction loss, and parameters of the autoencoder are modified in an attempt to reduce (e.g., optimize) the reconstruction loss.
[0072] Foundation models are large, general-purpose models that are pre-trained on very large, diverse datasets and that can be adapted (fine-tuned or prompted) for a wide range of downstream tasks. Large Language Model (LLM) foundation models, such as Generative Pre-trained Transformer (GPT), Claude, Gemini, and Large Language Model Meta Artificial Intelligence (Al) (LLaMA), have general-purpose capabilities that are not designed for specific tasks or with the preferences of a specific user in mind. One way to personalize a foundation model is to apply sets of weights (e.g., low-rank adaptation (LoRA) parameters) to portions (e.g., one or more layers) of a pre-trained foundation model to adapt outputs of the foundation model to have particular characteristics. As a result of the weights, output data sets from the foundation model are specialized at performing specific tasks or output data sets personalized to preferences of a user. Thus, fine-tuning can be used to adapt a pre-trained foundation model to a specific use case without re-training the model, which can be resourceintensive.
[0073] As used herein, “image data” and “pixel data” should be understood according to any of their usual and customary meanings in the fields of image processing, computer vision, photography, medical imaging, and / or artificial intelligence. For example, image data includes a digital representation of visual information, such as color, intensities, and spatial arrangement of portions of an image, and may also include metadata and / or auxiliary information, such as a time, a date, and / or a location associated with the image data; field of view information; camera pose information; camera intrinsic parameters; etc. The color, intensities, and spatial arrangement information of an image isQUALCOMM Ref. No. 2507839WO- 23 / 107 -represented in pixel data, where each pixel has a corresponding set of values representing color, intensity, and position within a field of view of the image.
[0074] FIG. 1 is a block diagram of an illustrative system 100 that is operable to insert an object within a gaussian scene representation, in accordance with some examples of the present disclosure. In FIG. 1. the system 100 includes a device 102 that includes a memory 120 configured to store one or more gaussian scene representations, such as an illustrative gaussian scene representation 154, and one or more processors 130 configured to insert an object within a gaussian scene representation based on input data 176, one or more templates 174 from a template library 172, or both.
[0075] FIG. 1 also illustrates a schematic representation of a scene 132 that can be represented by the gaussian scene representation 154. The scene 132 can be a 3D scene or a 4D scene, where a fourth dimension of the scene 132 is time, such that the 4D scene models the 3D scene over time. FIG. 1 also illustrates a view 136 associated with the scene 132. For example, the view 136 represents what a camera 134 with a particular pose and field of view would capture of the scene 132.
[0076] The device 102 includes or is coupled to one or more sources 104 of input data. For example, the source(s) 104 illustrated in FIG. 1 include one or more image sources 106, one or more sensors 110, a modem 114, the template library 172, or a combination thereof. In other examples, the source(s) 104 can also, or alternatively, include other devices or systems, such as a memory device, a media player, a game system, etc. In some examples, the source(s) 104 include one or more input sources configured to provide the input data 176, as further described with reference to FIGs. 4A and 4B. Although the image source(s) 106, the sensor(s) 110, the modem 114, and the template library 172 are illustrated outside the device 102 in FIG. 1, in other examples, the image source(s) 106, the sensor(s) 110, the modem 114, the template library 172, or a combination thereof, are integrated within the device 102.
[0077] The image source(s) 106 are configured to provide image data 108 to the device 102. For example, the image source(s) 106 can include or correspond to one or more still cameras, one or more video cameras, or both. As another example, the image source(s) 106 can include a memory' that stores the image data 108 or a computer or similar device that generates the image data 108 by executing instructions, such as aQUALCOMM Ref. No. 2507839WO- 24 / 107 -game engine that generates images to represent a virtual reality, augmented reality, or mixed reality scene. The image source(s) 106 can be controlled manually (e.g.. based on user input) or automatically (e.g., based on sensor data 112, based on position information, based on a schedule, etc.).
[0078] The sensor(s) 110 are configured to provide sensor data 112 to the device 102. For example, the sensor(s) 110 can include a light sensor that is configured to generate light data representing lighting intensity or lighting changes within a scene. As another example, the sensor(s) 110 can include a motion sensor that is configured to generate motion data representing movement of the device 102, movement of another device, or movement within a portion of the scene 132. In still other examples, the sensor(s) 110 can include one or more ranging sensor(s), such as sonar, lidar, or radar sensor(s). In some cases, the image data 108 can be processed to generate motion data, light data, or both.
[0079] The modem 114 is configured to provide data 116 received from a remote device to the device 102. The data 116 received via the modem 114 can include sensor data 112, image data 108, user input data, mapping data 122. reference images 124, gaussians 156 of a gaussian scene representation 154, one or more templates 174, input data 176, other data, or a combination thereof.
[0080] As further described below with reference to FIGs. 4A-4C, the template library 172 is configured to store and provide one or more template(s) 174 to the device 102. The template(s) 174 can represent attributes of one or more gaussians that, when included in a view of a scene, causes the view of the scene to represent respective objects. For example, the template(s) 174 can include indications of gaussians that can be used to represent a ball in the scene 132. In some cases, the template(s) 174 additionally include a scale reference that can be used to set a size of the object in the scene, a location reference that can be used to set a location of the object in the scene, or both.
[0081] In a particular aspect, the gaussian scene representation 154 can be stored in the memory 120. The memory 120 can also include one or more buffer(s) 118 configured to store data received from the source(s) 104 for analysis and / or use in updating the gaussian scene representation 154. For example, the buffer(s) 118 can include an imageQUALCOMM Ref. No. 2507839WO- 25 / 107 -buffer configured to store the image data 108, a sensor data buffer configured to store the sensor data 112, or both. The memory 120 may also be configured to store reference images 124. The reference images 124 can include one or more training images 126 (e.g., an image of the scene 132 used to train the gaussian scene representation 154), synthetic images 128 (e.g., an image of the scene 132 generated as output by the gaussian scene representation 154), or both. The reference images 124 can be used as a reference for a current state of a gaussian scene representation 154 for purposes of determining whether and / or how to update the gaussian scene representation 154 based on newly received input pixel data (e.g., at least a portion of the image data 108).
[0082] In FIG. 1, the processor(s) 130 include a selective model update pipeline 140, a gaussian object editor 170, a Tenderer 158, a controller 160, and a view selector 162. The selective model update pipeline 140 is optional and is described herein as one example of how the gaussian scene representation 154 can be generated and / or maintained. For example, in some embodiments, the gaussian scene representation 154 can be preexisting. To illustrate, the device 102 can use the modem 114 to download the gaussian scene representation 154 from another device (e.g., a server, a data store, or another device that creates, updates, or stores gaussian scene representations). In such embodiments, the selective model update pipeline 140 can be omitted from the device 102. Further, in at least some embodiments in which the selective model update pipeline 140 is omitted, one or more of the source(s) 104 can be omitted from the device 102. Additionally, in at least some embodiments in which the selective model update pipeline 140 is omitted, one or more of the buffer(s) 118, the mapping data 122, the reference image(s) 124, or a combination thereof, can be omitted from the memory 120.
[0083] FIG. 1 also shows the gaussian scene representation 154 in the processor(s) 130, which may be the case during certain operations. However, in other situations, such as during a partial update of the gaussian scene representation 154 or during rendering, only a subset of the gaussian scene representation 154 (e g., selected gaussians 156) may be loaded to the processor(s) 130. Similarly, the view selector 162 and the Tenderer 158 may only be loaded to the processor(s) 130 for particular operations, such as rendering the view 136 or rendering views during a full update of the gaussian scene representation 154.QUALCOMM Ref. No. 2507839WO- 26 / 107 -
[0084] As explained further below, the gaussian object editor 170 is configured to manage object insertion within the gaussian scene representation 154. The selective model update pipeline 140, if present, is configured to manage updating of the gaussian scene representation 154 based on input pixel data. The Tenderer 158 is configured to generate views (e.g., the view 136) of the scene 132 based on output of the gaussian scene representation 154 and based on view selection data from the view selector 162. The view selector 162 is configured to provide the gaussian scene representation 154, the Tenderer 158, or both, with the view selection data, where the view selection data indicates a location, a pose, a field of view, a focal length, other information specifying a portion of the scene 132 to be rendered, or a combination thereof. In the example of FIG. 1, the Tenderer 158 is coupled to a display 138 that is configured to display the view 136; however, in other cases, the view 136 can be stored or sent to another device for use. To illustrate, the view 136 can be sent via the modem 114 to a remote device for further processing or display.
[0085] The controller 160 is configured to coordinate tasks among various components of the system 100. For example, the controller 160 can cause one or more of the sources(s) 104 to provide data to the processor(s) 130 and / or the memory 120 at particular times or intervals. As another example, various components of the system 100 can provide status information associated with their respective tasks to the controller 160, and the controller 160 can use the status information to initiate or control other components of system 100. For example, when the device 102 receives new image data 108 from the image source(s), the controller 160 can be notified and can initiate the selective model update pipeline 140 or a portion thereof in response to the new image data 108. As another example, when the update strategy selector 144 selects a particular update strategy, the controller 160 can cause components associated with the particular update strategy to be initiated. To illustrate, a full update 148 includes rendering of various views of the gaussian scene representation 154, whereas a partial update 150 may not entail rendering of view's. Accordingly, w hen the update strategy selector 144 selects the full update 148, the controller 160 can be notified and can cause the processor(s) 130 to load or initiate the view selector 162 and the Tenderer 158.
[0086] In the example illustrated in FIG. 1, the selective model update pipeline 140 includes one or more data analyzers (such as an illustrated data analyzer 142), an updateQUALCOMM Ref. No. 2507839WO- 27 / 107 -strategy selector 144, and update engines 146. In this example, the data analyzer 142 is configured to analyze input data, such as at least a portion of the image data 108. at least a portion of the sensor data 112, or other data, to generate one or more metrics, classification data, or other data indicative of estimated, expected, or detected changes in the scene 132. For example, changes in the scene 132 can be detected based on differences between input pixel data (e.g., at least a portion of the image data 108) and reference pixel data of the reference images 124. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and reference pixel data (e.g., based on a pixel-by-pixel comparison) and generate one or more difference metrics. A difference metric can indicate, for example, a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values. In some cases, a user can provide user input to designate a particular portion of the input pixel data as a region of interest, and the difference metric(s) can be determined based on the user designated region of interest.
[0087] The update strategy selector 144 is configured to select an update strategy to be used (if any) based on one or more update criteria. For example, the update strategy selector 144 can select from among a full update 148, one or more different types of partial updates (including a partial update 150 in FIG. 1), or no update 152 based on an update criterion (or based on more than one update criteria). If the partial update 150 is to be performed, the update strategy selector 144 may also select a portion of the input pixel data to be used to update a corresponding set of gaussians 156 of a gaussian scene representation 154.
[0088] In a particular aspect, the update strategy selector 144 selects an update strategy based on output of the data analyzer 142 (e.g., based on one or more of the difference metrics). Additionally, or alternatively, the update strategy selector 144 can select the update strategy based on a device condition associated with the device 102 or a component thereof. For example, the sensor data 112 can include device condition data indicating a location of the device 102, a battery level of the device 102, whether the device 102 is charging, a temperature of the device 102, etc. Additionally, or alternatively, the processor(s) 130, the memory 120, or another component of the deviceQUALCOMM Ref. No. 2507839WO- 28 / 107 - 102 can provide device condition data indicating a level of usage of a computing resource of the device 102. To illustrate, the device condition data can include a processor utilization metric (e.g., a processor load), a memory utilization metric (e.g., memory usage, cache load, or memory access latency), a communication metric (e.g., a bus load, an available network bandwidth, or network latency), or a similar metric. Examples of a processor utilization metric include, without limitation, user processor time (e.g., time spent executing user-space processes), system processor time (e.g., time spent on kernel-level operations), idle time (e.g., percentage of time a processor 130 is not executing any tasks), input / output wait time, load average (e.g., average number of processes waiting to execute in a given time period), a count of context switches per second, a count of processor throttling events (e.g., due to overheating or power-saving measures), or a combination thereof.
[0089] For example, the update strategy selector 144 can be configured to compare one or more difference metrics from the data analyzer 142 to corresponding thresholds to select the update strategy. To illustrate, the update strategy selector 144 can compare a value of a difference metric based on the differences between the input pixel data and the reference pixel data to one or more thresholds to determine whether to perform the partial update 150, the full update 148, or no update 152. As anon-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of the difference metric is less than a first threshold, determine to perform the partial update 150 if the value of the difference metric is greater than the first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the difference metric is greater than the second threshold. As a second non-limiting illustration, the update strategy selector 144 may determine to perform no update 152 if the value of a size metric is less than a first threshold, determine to perform the partial update 150 if the value of the size metric is greater than a first threshold and less than a second threshold, and determine to perform the full update 148 if the value of the size metric is greater than the second threshold. In this illustration, the size metric can be based on dimension of a difference region, a count of pixels of the difference region, a count of gaussians 156 of the gaussian scene representation 154 associated with the difference region (also referred to herein as "‘difference gaussians”), or a combinationQUALCOMM Ref. No. 2507839WO- 29 / 107 -thereof. For example, the size metric can be based on a count of the difference gaussians and a count of gaussians 156 of the gaussian scene representation 154.
[0090] When the update strategy7selector 144 determines to perform the full update 148, the update engines 146 perform operations that are similar to initial creation of a gaussian scene representation except that the full update 148 is initialized from the existing gaussian scene representation 154. For example, during the full update 148, all of the gaussians 156 of the gaussian scene representation 154 can be processed from various views (as described further with reference to FIGS. 3A and 3B) to update their respective parameters. In this example, processing the gaussians 156 includes rendering selected views, comparing the rendered views to ground-truth image data (e.g., images of the scene) to determine loss values, and using the loss values to update the gaussians 156 (e.g., using back-propagation and / or other model training techniques). The full update 148 can also include operations such as adaptive density control which can generate new gaussians 156 or eliminate or merge gaussians 156. Thus, the full update 148 is very computationally intensive.
[0091] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all of the gaussians 156) of the gaussian scene representation 154 are updated, as described further with reference to FIGS. 3A and 3C. For example, the update strategy7selector 144 may indicate that a first subset of gaussians 156 are selected for update and a second subset of gaussians 156 of the gaussian scene representation 154 are not selected for update and are to be held constant. The update strategy7selector 144 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and optionally a second subset of the input pixel data that is not to be used for the partial update 150.
[0092] In a particular aspect, the update strategy selector 144 selects the first subset of gaussians 156 to be updated and the first subset of the input pixel data used in the partial update 150 based on differences between the input pixel data and reference pixel data of the reference images 124, based on the sensor data 112, based on user input, or a combination thereof. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data can be used to estimate the difference region. To illustrate, changes inQUALCOMM Ref. No. 2507839WO- 30 / 107 -lighting, as detected by a light sensor, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by a motion sensor and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data. As another example, a user can provide user input designating the difference region.
[0093] The update strategy selector 144 can use the mapping data 122 (e.g., pixel-to-gaussian mapping data) to identify a subset of the gaussians 156 associated with pixels of the difference region. In this example, the pixels of the difference region can be used for the partial update 150, and the subset of the gaussians 156 associated with pixels of the difference region can be updated during the partial update 150. Other gaussians 156 of the gaussian scene representation 154 are not modified during the partial update 150.
[0094] It is likely that there will not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, a particular gaussian 156 may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g.. the view 136). Thus, in some cases, the update strategy selector 144 is configured to use the mapping data 122 (e.g., gaussian-to-pixel mapping data) to identify pixels that are influenced by the subset of the gaussians 156 associated with pixels of the difference region. In such cases, the pixels of the input pixel data that correspond to the pixels that are influenced by the subset of the gaussians 156 are included in the training data used for the partial update 150. Updating a gaussian in this context refers to changing one or more parameters of the gaussian.
[0095] In a particular aspect, the update strategy selector 144 is configured to generate and / or update the mapping data 122. For example, the update strategy selector 144 can use saliency mapping to estimate which gaussians 156 most affect particular output pixel data from the gaussian scene representation 154 to generate pixel-to-gaussian mapping data and / or can use saliency mapping to estimate which pixels of input pixel data are most important to training of particular gaussians 156 of the gaussian scene representation 154 to generate gaussian-to-pixel mapping data.QUALCOMM Ref. No. 2507839WO- 31 / 107 -
[0096] Saliency mapping generally involves at least one forward pass through rendering operations (e.g., projection and rasterization) to generate gradient values used to determine the pixel-to-gaussian mapping data and at least one backward pass through the rendering operations to determine the gaussian-to-pixel mapping data. Thus, saliency mapping can be rather computationally complex, especially if the gaussian scene representation 154 is large and / or updated frequently. To reduce the computational demand associated with running the gaussian scene representation 154 to generate mapping data 122, the selective model update pipeline 140 can use a staged approach to manage the mapping data 122. In the staged approach, after mapping data 122 (e.g.. pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both) is generated, the selective model update pipeline 140 stores the mapping data 122 in a manner that facilitates reuse of the mapping data 122. For example, after mapping data associated with a particular region of interest (e.g., a range of pixels, an object, or some other identifiable portion of the input pixel data) is generated, the mapping data 122 can be stored along with information identifying the region of interest in a searchable data store. As one example, the region of interest can be associated with an object or a portion of an object in the scene 132. In this example, the mapping data 122 can include information that maps the object (or the portion of the object) to gaussians that represent the object. The mapping data 122 can also, or alternatively, include pixel-to-gaussian mapping data, gaussian-to-pixel mapping data, or both, associated with the object. In this example, object detection (which uses fewer computing resources than saliency mapping) can be used to process input pixel data, and if an object detected in the region of interest is represented in the mapping data 122 the mapping data entry associated with the object can be retrieved from the mapping data 122. In contrast, if the object is not represented in the mapping data 122, a mapping data generation process (e.g., saliency mapping) can be performed.
[0097] In some cases, the update strategy selector 144 can use a staged approach with multiple data stores that each include mapping data 122, where a first stage includes a set of the most frequently used mapping data 122, a second stage includes a set of next most frequently used mapping data 122, and so forth. In such cases, full mapping data generation is performed if none of the searchable data stores includes the desired mapping data.QUALCOMM Ref. No. 2507839WO- 32 / 107 -
[0098] In a particular aspect, the update strategy selector 144 is configured to select an update strategy based, at least in part, on a device condition associated with the device 102. In some examples, the update strategy selection decision is based on the device condition and is independent of the magnitude or type of differences detected or expected in the input pixel data. For example, irrespective of the update strategy the update strategy selector 144 selects based on the detected or expected differences in the input pixel data, the update strategy selector 144 can select a different update strategy based on the device condition. In this example, the update strategy selected based on the device condition prevails or sets an upper bound on the type of update actually performed. To illustrate, if the device condition indicates that usage of computational resources is relatively low (e.g., below a first threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the full update 148, the partial update 150, or no update 152. However, if the device condition indicates that usage of computational resources is relatively high (e.g., above the first threshold and below a second threshold), the update strategy selector 144 may make a selection, based on detected or expected differences in the input pixel data, from among the partial update 150 or no update 152. Further, if the device condition indicates that usage of computational resources is very' high (e.g., above the second threshold), the update strategy selector 144 selects no update 152 irrespective of the detected or expected differences in the input pixel data.
[0099] In a particular aspect, results of operations performed by the device 102 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation 154 in which one or more parameters of one or more of the gaussians 156 has been modified), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views 136 generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with the scene 132. For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as anQUALCOMM Ref. No. 2507839WO- 33 / 107 -electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, an amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of the device 102 or another device, such as to turn on or turn off a display based on an aspect of the scene 132. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory' or haptic notification or a combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene 132. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene 132. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing or suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene 132.
[0100] The gaussian object editor 170 is configured to access a gaussian scene representation 154. In some examples, the gaussian object editor 170 accesses a gaussian scene representation 154 that is generated or updated using the selective model update pipeline 140. In some examples, the gaussian object editor 170 accesses a gaussian scene representation 154 that is obtained from a source 104. The gaussian object editor 170 is configured to add gaussians that represent one or more objects to theQUALCOMM Ref. No. 2507839WO- 34 / 107 -gaussian scene representation 154 based on the input data 176. For example, the input data 176 can include a request (e.g., an instruction) to add an object to a view 136 of the scene 132 represented by the gaussian scene representation 154 that includes gaussians 156. The gaussian object editor 170 can edit the gaussian scene representation 154 according to the request to produce a modified gaussian scene representation 154’ that includes modified gaussians 156' that include a representation of the object. To illustrate, in some examples, the modified gaussians 156’ include one or more of the gaussians 156 that represent object(s) included in the gaussian scene representation 154 and also include one or more gaussians that represent the added object. In some embodiments, the gaussian object editor 170 can be a distinct circuit (e.g., a dedicated circuit) within the processor(s) 130 that is configured to perform some or all of the various operations described herein, in other embodiments, the gaussian object editor 170 can correspond to one or more other portions (e.g., general-purpose circuitry) of the processor(s) 130 that can perform some or all of the various operations described herein.
[0101] It is noted that an “object” may include any element that is included in or can be added to a scene and that can be represented by one or more gaussians in a gaussian scene representation of the scene. For example, objects of the scene 132 can be represented by one or more gaussians of the gaussians 156 in the gaussian scene representation 154, one or more gaussians of the modified gaussians 156’in the modified gaussian scene representation 154’, or both. As non-limiting examples, the scene 132 includes a roadway 103 and a building 105, which are objects that can be represented using one or more gaussians (e.g.. the gaussians 156 or the modified gaussians 156’). Further, in some cases, portions of objects can be considered objects. For example, a roof of the building 105, a window of the building 105, and a door of the building 105 can be considered objects and can be represented by one or more gaussians.
[0102] The input data 176 can take several forms, including text data, eye gaze data, gesture data, data from an input / output (I / O) device, audio data (e.g., from a microphone), or any combination thereof. As discussed further with reference to FIGs.4A-4C. the gaussian object editor 170 includes an instruction parser configured to parse the input data 176 and identify the request using one or more engines. In various embodiments, the instruction parser includes various engines including a speech-to-textQUALCOMM Ref. No. 2507839WO- 35 / 107 -engine, an LLM, one or more transformers, or a computer vision engine. Further, in some cases, the input data 176 includes object data associated with an object. For example, in various cases, the input data 176 includes an object template associated with the object, a scale reference (e.g., a requested size) corresponding to the object, a location reference (e.g., a requested position and spatial orientation) corresponding to the object, or any combination thereof. In some cases, the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
[0103] The gaussian object editor 170 includes object template identification logic that identifies (e.g., locates or generates) an object template associated with the object. As described above, in some cases, the object template is received with the input data 176. In other cases, the object template is stored at an external device (e.g., the template library 172, the image source(s) 106, a device that provides the input data 176, etc.). In those cases, the gaussian object editor 170 is configured to retrieve the object template (e.g., the template(s) 174). In other cases, the gaussian object editor 170 is configured to generate the object template. Additionally, in some cases, the object template lacks particular parameters (e.g., location parameters, size parameters, or both). In those cases, the gaussian object editor 170 is configured to set particular parameter(s) corresponding to the object based on one or more parameters corresponding to another object represented by the gaussian scene representation 154, as further described below with reference to FIGs. 5 and 6. In some cases, the gaussian object editor 170 sets a particular parameter associated with the object based on a template parameter indicated in the object template, an object parameter of another object represented by the gaussian scene representation 154, or both.
[0104] Based on the object template, the gaussian object editor 170 can obtain (e.g., generate) gaussians that represent the object indicated by the request of the input data 176. The gaussian object editor 170 is configured to add the gaussians to the gaussian scene representation 154 that includes the gaussians 156 to produce a modified gaussian scene representation 154’ that includes the modified gaussians 156’. The renderer 158 may generate updated views (e.g., one or more candidate views of the scene 132) including the object based on the gaussian scene representation 154'.QUALCOMM Ref. No. 2507839WO- 36 / 107 -
[0105] Optionally, the gaussian object editor 170 may perform error detection on the scene 132. For example, subsequent to the addition of the object to the gaussian scene representation 154, the gaussian object editor 170 obtains one or more error metrics associated with the scene 132. The one or more error metrics may be based on an analysis of the scene 132. In some cases, the one or more error metrics indicate overlapped objects. The gaussian object editor 170 may perform error correction on the scene 132. For example, the error correction may include adjusting (e.g., correcting, updating, or modifying) a position of the newly inserted object, adjusting a size of the newly inserted object, or both. As a result, in some cases, a number of collisions between objects in views of scenes is reduced. In other embodiments, one or more other portions of the processor(s) 130 may perform some or all of the error detection, some or all of the error correction, or both.
[0106] As can be appreciated from the foregoing, a technical advantage of the system 100 includes using an object template to add a new object to a gaussian scene representation. Another technical advantage includes addition of gaussians representing an object to a gaussian scene representation based on, in some cases, less input data with the request, as compared to conventional systems. The gaussian object editor can receive input data that includes a request to add an object to the gaussian scene representation and, when, not provided in the request, obtain (e.g., retrieve or generate) information such as gaussians to use to represent the object, a scale of the object within the scene, a location of the object within the scene, or any combination thereof.Additionally, in some scenes, a technical advantage of the system 100 includes a reduction of collisions between objects when adding an object to a view of a scene represented by a gaussian scene representation.
[0107] FIG. 2 is a block diagram of an illustrative system 200 that is operable to insert an object within a gaussian scene representation 154, in accordance with some examples of the present disclosure. Various components, features, and / or elements of the system 200 can be stored in the memory 120 of FIG. 1 as data or instructions, can be performed by the processor(s) 130 of FIG. 1, or both. Additionally, various components of the system 200 correspond to components of the system 100 of FIG. 1, and can be configured as described with reference to FIG. 1. For example, the system 200 includes the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, theQUALCOMM Ref. No. 2507839WO- 37 / 107 -template library 172, and the memory 120, one or more of which can be configured as described with reference to FIG. 1.
[0108] In the example illustrated in FIG. 2, the system 200 includes one or more buses, such as an illustrated bus 216, coupled to the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, the memory 120, one or more other components or devices, or a combination thereof, via respective bus interfaces 131, 133, 135, 137, 139, 141, and 143. For example, the bus 216 can enable communication of data or signals (e.g., control signals, status signals, etc.), between and among the template library 172, the image source(s) 106, the modem 114, the sensors 110, the processor(s) 130, and the memory 120. The sensors 110 can include one or more motion sensors 210, one or more light sensors 212, one or more device sensors 214, other sensors, or a combination thereof.
[0109] In FIG. 2, the processor(s) 130 include the gaussian object editor 170. The processor(s) 130 optionally also include the selective model update pipeline 140, the controller 160, the renderer 158, or a combination thereof. The selective model update pipeline 140, in this example, includes the data analyzer 142, the update strategy selector 144, the update engines 146, an update data selector 224, and a mapper 222.
[0110] The image source(s) 106 are configured to generate (e.g., capture), receive, or otherwise obtain image data 147 representing a scene. As an example, the image source(s) 106 can include one or more cameras, a memory storing images, etc. In a particular aspect, the image source(s) 106 can provide image data 147 to the bus 216 via the bus interface 131 for communication to the processor(s) 130 (e.g., via the bus interface 143), to the memory' 120 (e.g., via the bus interface 141), or both. In FIG. 2, the image data 147 is illustrated as provided to the data analyzer 142; however, in some cases, the image data 147 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142. In some cases, the controller 160 can provide control signals to the image source(s) 106 (e.g., via the bus 216 and the bus interfaces 143, 131) to control when images are captured or provided to the memory 120 and / or the processor(s) 130. When the image source(s) 106 provide the image data 147 to the memory' 120, the image data 147 can be stored in an image buffer (e.g., one of the buffer(s) 118 of FIG. 1).QUALCOMM Ref. No. 2507839WO- 38 / 107 - [OHl] The template library 172 is optional, and if present, is configured to store and provide one or more template(s) 174 to the device 102. In examples in which the template library 172 is absent, the gaussian object editor 170 can obtain the one or more template(s) 174 from another source 104, the memory' 120, a network device, an input device, or a combination thereof. The template(s) 174 can represent attributes of one or more gaussians that, when included in a view of a scene, causes a view of the scene to represent respective objects. For example, a template 174 (e.g., an object template) of a ball can indicate attributes of one or more gaussians that can be used to represent a ball in a scene. In some cases, the template 174 additionally includes a scale reference that can be used to set a size of the obj ect in the scene, a location reference that can be used to set a location of the object in the scene, or both. As further described below with reference to FIG. 4C, in some cases, the template(s) 174 (e.g., object templates) include gaussian data, class data (e.g., data regarding several related types of objects), scale data, location data, or any combination thereof. In some embodiments, the memory 120 includes the template library 172.
[0112] The sensor(s) 110 are optional, and if present, are configured to generate sensor data 145. For example, the motion sensor(s) 210 can generate motion data indicating movement of a camera of the image source(s) 106 relative to the scene or movement of an object in the scene. As another example, the light sensor(s) 212 can generate light data indicative of changes in lighting of a portion of the scene. As yet another example, the device sensor(s) 214 can generate device condition data. The device condition data can indicate a condition associated with a device that includes the selective model update pipeline 140. To illustrate, the device condition data can indicate a battery level, w hether the device is connected to an external power source, a temperature of the device, a level of usage of a computing resource of the device (such as a processor utilization metric or a memory utilization metric), or other similar device condition information. In FIG. 2. the sensor data 145 is illustrated as provided to the data analyzer 142; however, in some cases, the sensor data 145 can be provided to other components of the selective model update pipeline 140 in addition to, or instead of, the data analyzer 142.
[0113] In a particular aspect, the data analyzer 142 is configured to determine differences between input pixel data of the image data 147 and reference pixel dataQUALCOMM Ref. No. 2507839WO- 39 / 107 - (e.g., pixel data of one or more of the reference images 124 of FIG. 1) and to provide information related to such differences to the update strategy selector 144 and optionally to an update data selector 224. As one example, the data analyzer 142 can be configured to determine differences between the input pixel data and the reference pixel data based on a pixel-by-pixel comparison and / or using optical flow methods. As another example, the data analyzer 142 can be configured to estimate or predict differences between the input pixel data and the reference pixel data based on motion indicated in motion data, based on lighting changes indicated in light data, or both.
[0114] The data analyzer 142 may be configured to generate one or more difference metrics associated with the differences between the input pixel data and the reference pixel data. For example, the difference metrics can indicate a count of difference pixels (e.g., pixels of the input pixel data that differ from corresponding pixels of the reference pixel data by at least a threshold amount), a size of a region or cluster of difference pixels, a magnitude of differences associated with the difference pixels, or other values.
[0115] The update strategy selector 144 is configured to select an update strategy based on output of the data analyzer 142. Thus, the update strategy selector 144 can select the update strategy based, at least in part, on the differences between input pixel data and the reference pixel data. For example, based at least in part on data provided by the data analyzer 142, the update strategy selector 144 can signal to the update engines 146 to perform no update 152, the partial update 150, or the full update 148.
[0116] When the update strategy selector 144 indicates that the partial update 150 is to be performed, the update data selector 224 can identify which pixels of the input pixel data are to be used for the partial update 150, which gaussians 156 of the gaussian scene representation 154 are to be updated, or both. For example, the update data selector 224 can use the mapping data 122 to identify a subset of the gaussians 156 that correspond to (e.g., influence rendering of) pixels of a difference region. The pixels of the difference region can correspond to or include pixels of a region of interest, in which case pixel-to-gaussian mapping data can be used to identify the subset of the gaussians 156. The pixel-to-gaussian mapping data can include, for example, a data structure that associates particular pixels of a view of the scene with gaussians of the gaussian scene representation 154 that influence rendering of the particular pixels. For example, when aQUALCOMM Ref. No. 2507839WO- 40 / 107 -particular view is rendered, the pixel-to-gaussian mapping data can be updated or generated to indicate which of the gaussians 156 influenced rendering of each pixel of the rendered view. Alternatively, the pixels of the difference region can correspond to or include pixels associated with a particular object, in which case object mapping data can be used to identify the subset of the gaussians 156. The object mapping data can include, for example, a data structure that associates particular objects detected in the scene with gaussians 156 that represent the particular objects in the gaussian scene representation 154. For example, the data analyzer 142 can include an object detector that identifies objects based on pixel data. In this example, when a particular view that includes a detected object is rendered, the object mapping data can be updated or generated to indicate which of the gaussians 156 influenced rendering of each pixel associated with the detected object.
[0117] The update data selector 224 can also, or alternatively, use gaussian-to-pixel mapping data of the mapping data 122 to identify pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156. Because each gaussian 156 can influence rendering of more than one pixel, the identified subset of gaussians can influence more pixels than the pixels used to identify the subset of gaussians. Thus, for example, the pixels of the input pixel data that are to be used to update an identified subset of the gaussians 156 can include pixels within the region of interest and one or more nearby pixels that are also influenced by the identified subset of the gaussians 156.
[0118] In some cases, the selective model update pipeline 140 includes a mapper 222 that is configured to generate or control the mapping data 122. For example, the mapper 222 can perform operations to generate the mapping data 122 (or portions thereof), to update the mapping data 122, to determine whether new mapping data 122 is to be generated, etc. As a particular example, the mapper 222 can perform operations associated with the staged approach to managing the mapping data 122 described with reference to FIG. 1.
[0119] In some embodiments, the update strategy selector 144 is configured to select the update strategy based on the differences between input pixel data and the reference pixel data and additional information. For example, the data analyzer 142 can analyze the sensor data 145 and provide analysis results to the update strategy selector 144. InQUALCOMM Ref. No. 2507839WO- 41 / 107 -this example, the update strategy selector 144 can select the update strategy based, at least in part, on the sensor data 145.
[0120] As one example, the update strategy selector 144 may select to perform no update 152 when the differences between input pixel data and the reference pixel data are insignificant (e.g., less than a first threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are insignificant, and / or when the sensor data 145 indicate that computing resources are strained (e.g., battery level is low, available processing capacity is low, etc.). Continuing this example, the update strategy selector 144 may select to perform the full update 148 when the differences between input pixel data and the reference pixel data are large (e.g., greater than a second threshold), when the sensor data 145 indicate that lighting or motion changes associated with the input image data are significant, and / or when the sensor data 145 indicate that significant computing resources are available. Further, the update strategy selector 144 may select to perform the partial update 150 when the differences between the input pixel data and the reference pixel data are constrained to one or more identified regions of interest (based on the image data 147, the sensor data 145, or both) and / or when the sensor data 145 indicate that computing resources are available but may be constrained.
[0121] Although the example above describes selecting an update strategy from among no update 152, the partial update 150, and the full update 148 based on relative states (e.g., significant, insignificant, large, constrained, etc.), in practice, the selection can be based on comparisons of various metrics to respective thresholds. For example, an “insignificant” difference or value can correspond to a value of a metric that is less than a first threshold, a “large” difference or value can correspond to a value of the metric that is greater than a second threshold, and a “significant” difference can correspond to a value of the metric between the first threshold and the second threshold. In some cases, more than two or fewer than two thresholds can be used, depending on the specific implementation.
[0122] When the update strategy selector 144 determines to perform the full update 148. all of the gaussians 156 of the gaussian scene representation 154 are eligible to be updated based on the input pixel data, and all of the input pixel data is used to performQUALCOMM Ref. No. 2507839WO- 42 / 107 -the full update 148. For example, the update strategy selector 144 can provide the input pixel data to the update engines 146. and the update engines 146 can render views of the gaussian scene representation 154, determine loss function values based on comparison of the rendered views to ground-truth views, use the loss function values to calculate gradient values, and use the gradient values to determine updated parameter values 149 for each of the gaussians 156.
[0123] When the update strategy selector 144 determines to perform the partial update 150, a subset of the gaussians 156 (i.e., fewer than all) of the gaussian scene representation 154 are updated based on the input pixel data. For example, the update data selector 224 can select a first subset of gaussians 156 that are to be modified, in which case a second subset of gaussians 156 are not to be modified and are to be held constant. The update data selector 224 may also identify a first subset of the input pixel data that is to be used for the partial update 150 and a second subset of the input pixel data that are not to be used for the partial update 150.
[0124] In a particular aspect, the update data selector 224 selects the first subset of the gaussians 156 to be updated and the first subset of the input pixel data used for the partial update 150 based on differences between the input pixel data and reference pixel data, based on the sensor data 145, or both. For example, differences between the input pixel data and the reference pixel data can be used to identify a difference region. As another example, the sensor data 145 can be used to estimate the difference region. To illustrate, changes in lighting, as detected by the light sensor(s) 212, associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data, and can be detected without direct comparison of the pixel data. As another illustrative example, motion detected by the motion sensor(s) 210 and associated with a particular area may be expected to correspond to differences between the input pixel data and the reference pixel data and can be detected without direct comparison of the pixel data.
[0125] The update data selector 224 can select the first subset of the gaussians 156 to be updated based on the pixels of the input pixel data to be used for the partial update 150 (e.g., based on the difference region). For example, the update data selector 224 can designate a subset of the gaussians 156 that correspond to or affect the pixels to be usedQUALCOMM Ref. No. 2507839WO- 43 / 107 -for the partial update 150 as eligible to be modified by the partial update 150. In this example, other gaussians 156 of the gaussian scene representation 154 are ineligible to be modified during the partial update 150.
[0126] In many cases, there may not be a one-to-one relationship between the gaussians 156 of the gaussian scene representation 154 and the pixels of the input pixel data. For example, particular gaussians may influence multiple pixels when the gaussian scene representation 154 is used to generate output to render a view (e.g., the view 136). Thus, in some cases, the update data selector 224 can use the difference region to identify the particular gaussians associated with pixels of the difference region. In such cases, the particular gaussians can be updated based on a selected set of pixels that include pixels of the difference region and additional pixels (typically near the difference region) that are influenced by the particular gaussians. In a particular aspect, the update data selector 224 (or the mapper 222) can use saliency mapping, look-up tables, or other mapping data to identify associations between pixels and gaussians of the gaussian scene representation 154.
[0127] In a particular aspect, results of operations performed by the system 200 include an updated gaussian scene representation 154 (e.g., a gaussian scene representation with updated gaussians 156), which can be used to enable a variety of activities. For example, the gaussian scene representation 154 or views generated by the gaussian scene representation 154 can be used to enable autonomous control during interaction with a scene (e.g., the scene 132 of FIG. 1). For example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to a machine, such as a vehicle, a robot, an appliance, a mobile device, a camera. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that generate control signals to an actuator, such as an electromechanical actuator, an electrohydraulic actuator, or an electroactive polymer actuator. To illustrate, output of the gaussian scene representation 154 can be used to control a brake actuator of a vehicle based on an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control an active circuit component, such as a voltage or current source, a transistor, anQUALCOMM Ref. No. 2507839WO- 44 / 107 -amplifier, an integrated circuit, a programmable device (e.g., a field-programmable gate array), or a display device. To illustrate, output of the gaussian scene representation 154 can be used to control other circuit elements of a device (e.g., the device 102 or another device), such as to turn on or turn off a display based on an aspect of the scene. As still another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that provide a notification to a user, such as a visual, auditory, or haptic notification or combination thereof. To illustrate, output of the gaussian scene representation 154 can be used to generate a notification to alert a user to an aspect of the scene. As yet another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing, or suspending an application on a computer. To illustrate, output of the gaussian scene representation 154 can be used to launch a navigation application to draw a user’s attention to an aspect of the scene. As another example, the gaussian scene representation 154 can be used to provide scene awareness input to one or more machine-learning models or control systems that control launching, closing, pausing, or suspending playback of media content (e.g., audio or visual content) on a computer. To illustrate, output of the gaussian scene representation 154 can be used to pause or stop playback of media content in order to allow a user to focus on information in the scene.
[0128] FIGs. 3A, 3B, and 3C, together, depict a system 300 that illustrates particular aspects of the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The system 300 illustrates additional details related to updating a gaussian scene representation 154, and in particular, illustrates differences between operations associated with performing a full update (e.g., the full update 148 of FIGs. 1 and 2) and a partial update (e.g., the partial update 150 of FIGs. 1 and 2). Due to the complexity of the system 300, the system 300 is split over three pages and linked by off-page references.
[0129] The system 300 includes one or more sensors 302, such as a first sensor 302A, a second sensor 302B, and a Pth sensor 302P (where P is an integer greater than two). At least one of the sensor(s) 302 is an image sensor (e.g., a camera) configured to generate image data. One or more others of the sensor(s) 302 can include or correspond to motion sensors, light sensors, device condition sensors, etc., such as described withQUALCOMM Ref. No. 2507839WO- 45 / 107 -reference to FIGs. 1 and 2. The sensor(s) 302 are configured to generate sensor data 303. The system 300 optionally includes a sensor controller 304 that is configured to control operation of the sensor(s) 302 and / or to select a subset of the sensor data 303 to generate sensor data 305, which is used for further processing. If the sensor controller304 is omitted, the sensor(s) 302 can be externally controlled (e.g., responsive to user input), can operate according to a schedule, or can generate the sensor data 303 responsive to detecting a condition (e.g., in response to detecting movement). If the sensor controller 304 is omitted, the sensor data 305 is the same as the sensor data 303.
[0130] In FIG. 3A, the system 300 includes a pose estimator 306 that is configured to determine pose data (e.g., a position and / or orientation of one of the sensor(s) 302 within a scene), which can be provided along with the sensor data 305 as input 307 to an image subspace selector 312. In some cases, the sensor data 305 can include pose information (e.g., as data or metadata). In such cases, the pose estimator 306 is optional. If the pose estimator 306 is omitted, the sensor controller 304 (or the sensor(s) 302 if the sensor controller 304 is also omitted), provide the input 307 including the sensor data305 and corresponding pose data to the image subspace selector 312.
[0131] During initial generation of the gaussian scene representation 154, the pose estimator 306, the sensor controller 304, or the sensor(s) 302 provide input 311 including sensor data 305 and corresponding pose data to a model initializer 310, which uses similar operations as described with reference to FIG. 3B for the full update 148 to generate initial parameters for the gaussians 156 of the gaussian scene representation 154. After initial generation of the gaussian scene representation 154, the model initializer 310 can be bypassed, and the input 307 can be provided to an image subspace selector 312.
[0132] In the example illustrated in FIG. 3 A, the gaussian scene representation 154 includes a plurality of gaussians 156, such as: a gaussian 156A, a gaussian 156B, and a gaussian 156Q (where Q is an integer greater than 2). Each gaussian 156 is characterized by a plurality of parameters describing various properties of the gaussian 156. For example, in FIG. 3 A. the gaussian 156A is characterized by an X-axis center position coordinate (“center pos. (X)”) 180, Y-axis center position coordinate (“center pos. (Y)”) 181, and Z-axis center position coordinate (“center pos. (Z)”) 182. Together,QUALCOMM Ref. No. 2507839WO- 46 / 107 -the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 indicate a center position of the gaussian 156A and correspond to the position vector, p,, discussed above with reference to Equations 1-10. In FIG. 3 A, the gaussian 156A is also characterized by an X-axis scale parameter 183, a Y-axis scale parameter 184, and a Z-axis scale parameter 185.Together the X-axis scale parameter 183, the Y-axis scale parameter 184. and the Z-axis scale parameter 185 indicate a size and shape of the ellipsoid of the gaussian 156A and correspond to the scale vector, si, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by multiple orientation or rotation parameters (e.g., rotation (qO) 186, rotation (ql) 187, rotation (q2) 188. rotation (q3) 189) specified as quaternions in the example illustrated in FIG. 3A. Together the rotation parameters 186-189 indicate a spatial orientation of the ellipsoid of the gaussian 156A and correspond to the orientation parameters, qf, discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by an opacity parameter 190, which indicates a maximum opacity of the gaussian 156A. The opacity parameter 190 corresponds to the opacity parameter, t discussed above with reference to Equations 1-10. The gaussian 156A is also characterized by a set of color parameters, e.g., color parameters 191, 192, and 193 in FIG. 3 A. The color parameter(s) 191-193 can be specified as a set of spherical harmonics (SH) coefficients, H as discussed above with reference to Equations 1-10. Although FIG. 3A illustrates the gaussian 156A as characterized by a set of fourteen parameters, in other examples, the gaussians 156 can be characterized by more than fourteen or fewer than fourteen parameters. For example, in different implementations, different numbers of color parameters 191-193 can be used, depending on the order of the SH coefficients used to represent the colors of the gaussians 156. To illustrate, in some implementations, a reduced set of approximately 14 parameters may be used, where the color parameters 191-193 are simplified to a single base color value (e.g., an RGB triplet) rather than a full set of spherical harmonics coefficients. In such implementations, the spherical harmonics coefficients may be omitted entirely, reducing memory and computational requirements for storing and updating the gaussians 156.
[0133] As described with reference to FIGS. 1 and 2, the gaussian object editor 170 can be used to add gaussians representing an object to a gaussian scene representation. TheQUALCOMM Ref. No. 2507839WO- 47 / 107 -gaussian object editor 170 receives the input data 176 that includes a request to modify the gaussian scene representation to add an object. In some embodiments, the gaussian object editor 170 obtains (e.g., receives from the template library 172 of FIG. 1) the template(s) 174 that indicate attributes of gaussians that can be used to represent the object. The gaussian object editor 170 generates updated gaussians 390 based on the attributes indicated by the template(s) 174 and adds the updated gaussians 390 to the gaussian scene representation 154 including gaussians 156 to generate a modified gaussian scene representation 154’ including modified gaussians 156’. As a result, the gaussian scene representation 154 is updated to the modified gaussian scene representation 154' with gaussians (e.g., the updated gaussians 390) that add the object in a view of a scene represented by the gaussian scene representation 154.
[0134] The image subspace selector 312 is configured to, based on the input 307, identify one or more patches of pixels (e.g., image patches) for a region of interest (ROI), or more than one ROI, associated with the sensor data 305 to generate ROI data 313. In a particular aspect, the image subspace selector 312 can correspond to, include, or be included within the data analyzer 142 of FIGs. 1 and 2. For example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to an object that is in motion in the scene (e.g.. based on motion data from a motion sensor, based on optical flow data from two or more images, etc.). In this example, the ROI can correspond to the image patch (or patches) associated with the object that is in motion. As another example, the image subspace selector 312 can identify one or more image patches (from one or more images) that correspond to pixels of input pixel data of the sensor data 303 or 305 that differ (by at least a threshold amount) from pixels of reference pixel data. In this example, the ROI can be defined by the location of the image patch (or patches) associated with the pixels that differ between the input pixel data and the reference pixel data. The ROI data 313 identifies one or more sets of pixels of one or more images.
[0135] The update strategy selector 144 of the system 300 is configured to select an update strategy based on strategy selection data 320. The strategy selection data 320 can include the sensor data 303 or 305, the input 307, the ROI data 313, other data, or a combination thereof. For example, the update strategy selector 144 can select the update strategy using any of the techniques described with reference to FIGs. 1 and 2. TheQUALCOMM Ref. No. 2507839WO- 48 / 107 -update strategy selector 144 is configured to provide a selection indication 321 to the update engines 146, where the selection indication 321 causes the update engines 146 to initiate operations associated with the selected update strategy. For example, if the selection indication 321 indicates that a full update strategy is selected, the update engines 146 initiate the full update 148, and if the selection indication 321 indicates that a partial update strategy is selected, the update engines 146 initiate the partial update 150. If no update is selected, the update strategy selector 144 can omit generation of the selection indication 321 or can generate a "no update’’ selection indication.
[0136] When the update engines 146 select the full update 148, the update engines 146 cause a pose selector 324 of FIG. 3B (as indicated by off-page reference A) to begin operations associated with the full update 148. The pose selector 324 is configured to select a pose 325 from training data 360 that includes a set of training images (e.g., images 361A-361K) and pose data (e.g., poses 362A-362K) associated with each image 361 A-361K (where K is an integer greater than two). An image associated with the selected pose 325 is used as a ground-truth image 349 for loss calculation after a view associated wi th the pose 325 is rendered.
[0137] The pose 325 describes the position and spatial orientation of a virtual camera 330 relative to the gaussians 156 of the gaussian scene representation 154 to be rendered as a view. Due to the pose 325 and intrinsic parameters (e.g., focal length, principal point, etc.) of the virtual camera 330, some of the gaussians (e.g., gaussians 331) are visible to the virtual camera 330, and some of the gaussians (e.g., gaussians 333) are not visible to the virtual camera 330. For example, for a pose 325 A, a virtual camera 330A can see gaussians 331A; however, gaussians 333A are not visible to the virtual camera 330A in the pose 325 A. As another example, for a pose 325K, a virtual camera 330K can see gaussians 331K; however, gaussians 333K are not visible to the virtual camera 330K in the pose 325 K.
[0138] To ensure that the full update 148 updates each gaussian 156 based on all relevant data, the full update 148 generally entails rendering a view for every available pose (e.g., poses 362A to 362K) and comparing each rendered view to the ground-truth image (e.g., each of images 361 A to 361 K) associated with the pose 362 used to render the view. Thus, performing the full update 148 is similar to operations performed by theQUALCOMM Ref. No. 2507839WO- 49 / 107 -model initializer 310 to generate an initial version of the gaussian scene representation 154 based on a set of images except that the full update 148 is initialized from a preexisting gaussian scene representation 154.
[0139] A projector 326 identifies the gaussians 331 that are visible to the virtual camera 330 for a pose 325. The projector 326 performs gaussian splatting, which entails projecting the visible gaussians 331, which are three dimensional, onto a 2D image plane 332 based on the pose 325 of the virtual camera 330 and the intrinsic parameters of the virtual camera 330. Each gaussian 331 is represented as an ellipse 335 in the 2D image plane 332. The ellipses 335 can overlap one another and can have various sizes and spatial orientations depending on parameters of the gaussians 331 that are projected and the pose 325 of the virtual camera 330. For example, the gaussians 331 A that are visible to the virtual camera 330A in the pose 325A are projected on to a 2D image plane 332A as ellipses 335A, and the gaussians 331K that are visible to the virtual camera 330K in the pose 325K are projected on to a 2D image plane 332K as ellipses 335K.
[0140] Data 334 describing the ellipses 335 on the 2D image plane 332 is provided to a differentiable tile rasterizer 340. For example, data 334A describes the ellipses 335A associated with the pose 325A, and data 334K describes the ellipses 335K associated with the pose 325K. The differentiable tile rasterizer 340 is configured to perform operations to blend the ellipses 335 to generate pixel data representing a 2D image that would be captured by the virtual camera 330 for the particular selected pose 325. To generate the pixel data, the differentiable tile rasterizer 340 divides the 2D image plane 332 into tiles (represented in FIG. 3B by bold lines and different shading in the 2D image planes 332A and 332K of the differentiable tile rasterizer 340). For purposes of this description, each tile is treated as corresponding to a single pixel; however, the exact relationship between tiles and pixels can differ between different implementations. For example, a tile can correspond to more than one pixel.Additionally, to simplify the drawings, in FIG. 3B, the 2D image plane 332 includes some regions that are not associated with any gaussian; however, it will generally be the case that each tile is associated with at least one gaussian 156 to provide color information for the tile.QUALCOMM Ref. No. 2507839WO- 50 / 107 -
[0141] The differentiable tile rasterizer 340 obtains depth information 342 for the visible gaussians 331 of the pose 325, where depth refers to the distance in the scene of a gaussian from the virtual camera 330. For example, the depth information 342 can be calculated for each of the visible gaussians 331 based on the location of the virtual camera 330 in the scene (as indicated by data describing the pose 325) and the location of the center of the gaussian in the scene (e.g., as indicated by the X-axis center position coordinate 180, the Y-axis center position coordinate 181, and the Z-axis center position coordinate 182 of each gaussian). For example, depth information 342A indicates distances in the scene between each of the gaussians 331 A and the virtual camera 330A, and depth information 342K indicates distances in the scene between each of the gaussians 331K and the virtual camera 330K.
[0142] The differentiable tile rasterizer 340 can perform tile-depth association 344 to, for each tile of the 2D image plane 332, sort the ellipses 335 that at least partially overlap the tile based on the depth information 342. Since the gaussians 156 are generally not opaque, a first gaussian that is more distant from the virtual camera 330 in the scene can be partially visible through a second gaussian that is closer to the virtual camera 330. Thus, to generate pixel data for a particular pixel of an image captured by the virtual camera, the differentiable tile rasterizer 340 accounts for the colors of each gaussian of a set of gaussians that influence the particular pixel, the opacities of the gaussians of the set, and the order of the gaussians of the set as viewed from the virtual camera 330.
[0143] To parallelize generation of the pixel data, the differentiable tile rasterizer 340 can perform thread association 346 to assign each pixel or sets of pixels to a respective computing thread to allocate the workload of pixel blending of the pixels to available computing resources, such computing resources can include one or more cores of one or more central processing units (CPUs), one or more cores of one or more graphics processing units (GPUs), or other processing units. Pixel blending includes determining a color value for each pixel based on contributions of gaussians that influence the pixel, e.g., using alpha compositing techniques. The blending uses view-direction-based colors of the gaussians (as determined based on the pose 325 of the virtual camera 330 and color parameters specified by spherical harmonic color coefficients, opacity of the gaussians, and the depth information 342).QUALCOMM Ref. No. 2507839WO- 51 / 107 -
[0144] Blending the contributions of the gaussians results in pixel data 341 representing an image (e.g., a view) captured by the virtual camera 330 for the particular pose 325. The pixel data 341 is provided to a loss calculator 348, which is configured to compare the pixel data 341 to the ground-truth image 349 corresponding to the pose 325. The loss calculator 348 determines a loss value for each pixel (e g., a photometric L2 loss) to quantify how the pixel data 341 differs from the ground-truth image 349. Gradient values 343, 345 based on the loss values are back -propagated through the differentiable tile rasterizer 340 and the projector 326 to estimate updated parameter values 347 for the visible gaussians 331 of the pose 325.
[0145] The full update 148 continues by iteratively repeating the processes of pose selection, projection, rasterization, loss calculation, and back-propagation of gradients until all of the poses 362 have been processed. At one or more points during the full update 148, an adaptive density controller 350 performs operations that change the number of gaussians 156 of the gaussian scene representation 154, such as pruning one or more gaussians 156 (e.g., the gaussian 156B) or splitting one or more gaussians 156. Thus, in some senses, a different gaussian scene representation 154 exists after the full update 148 than existed before the full update 148 since a different number and arrangement of gaussians 156 can be present, and any gaussian 156 that is carried over can have different parameters. For example, in FIG. 3B, the gaussian scene representation 154 is illustrated as including a first gaussian 156A, a second gaussian 156B, and one or more additional gaussians up to a Qth gaussian 156Q (where Q is an integer greater than 2). After the full update 148, the gaussian scene representation 154 can include more than Q or fewer than Q gaussians 156. Further, the full update 148 is a feedback-based process since, during each iteration, the visible gaussians 331 for a pose 325 are updated based on loss values indicating differences between an image (e.g., the pixel data 341) of a view rendered for the pose 325 and a reference image (e.g., the ground-truth image 349 associated with the pose 325).
[0146] When the update engines 146 of FIG. 3 A select the partial update 150, the update engines 146 cause a gaussian selector 354 of FIG. 3C (as indicated by off-page reference B) to select a set of gaussians (e.g., gaussians 156A and 156B in the example of FIG. 3C) that are to be updated based on the ROI data 313 and mapping data 122. As an example, for illustrative purposes, the sensor data 303 of FIG. 3 A can include anQUALCOMM Ref. No. 2507839WO- 52 / 107 -image 351 of FIG. 3C. In this illustrative example, the image subspace selector 312 can determine that the ROI includes a ball 365 depicted in the image 351. in which case the ROI data 313 can include data identifying the ball 365. The data identifying the ball 365 can include object identification data (e.g., a text label or a numerical designator), a set of pixels that represent the ball 365 in the image 351, identifiers of the set of pixels that represents the ball 365 in the image 351. or other data that enables the gaussian selector 354 to select the set of gaussians based on the mapping data 122 and the ROI data 313. In a particular aspect, the image subspace selector 312, the gaussian selector 354, or both, are integrated within the update data selector 224 of FIG. 2.
[0147] The gaussian selector 354 uses the ROI data 313 and the mapping data 122 to generate selection data 353 identifying the selected set of gaussians, and provides the selection data 353 to a partial updater 356. The selection data 353 identifying the selected set of gaussians can include the gaussians themselves, memory addresses of the selected set of gaussians. indices of the selected set of gaussians, or other information that enables the partial updater 356 to obtain the selected set of gaussians.
[0148] The partial updater 356 is configured to determine updates 357 to the selected set of gaussians (leaving remaining gaussians 156 of the gaussian scene representation 154 unchanged) to generate modified gaussians 156’. In some implementations, the partial updater 356 can use a feedforward approach to determine the updates 357. For example, the partial updater 356 can determine the updates 357 based on the sensor data 305 of FIG. 3 A. As one example, the sensor data 303 can include motion data that can be used to determine how far an object (e.g., the ball 365) in the scene moved, and the partial updater 356 can determine the updates 357 based on the determined movement of the object. As another example, optical flow can be used to compare the image 351 to one or more reference images to identify movement of one or more objects in the scene, and the partial updater 356 can determine the updates 357 based on the identified movement. As yet another example, the sensor data 303 can include light data which can be used to determine changes in lighting or color in a portion of the scene, and the partial updater 356 can determine the updates 357 based on the changes in lighting or color.QUALCOMM Ref. No. 2507839WO- 53 / 107 -
[0149] The partial updater 356 can apply the updates 357 to the selected set of gaussians and store updated versions of the selected set of gaussians (e.g., updated gaussians 156A’ and 156B’ in FIG. 3C) in an update buffer 358. After all gaussians of the selected set of gaussians have been updated, the modified gaussians 156’ can be written to a memory where the gaussian scene representation 154 is stored.
[0150] Optionally, in some embodiments, a rendering-based error calculator 359 can be used to check the modified gaussians 156’ before the gaussian scene representation 154 is modified. The rendering-based error calculator 359 is configured to perform projection, rendering, and error calculation, as described with reference to the projector 326, the differentiable tile rasterizer 340, and the loss calculator 348, respectively, of FIG. 3B to generate a rendered view of the gaussian scene representation 154. During the partial update 150, the rendering-based error calculator 359 can determine (e.g., obtain from the sensor data 305 or from the pose estimator 306 of FIG. 3 A) a pose associated with the image 351. The rendering-based error calculator 359 can render a view of the gaussian scene representation 154 (with the modified gaussians 156’) from the pose associated with the image 351, and compare the rendered view to the image 351 to determine whether the modified gaussians 156’ are acceptable or whether additional updates 357 should be determined. For example, the rendering-based error calculator 359 can determine loss value(s) (e.g., using similar operations as described with reference to the loss calculator 348 of FIG. 3B) based on differences between the image 351 and the rendered view. In this example, if the loss value(s) satisfy a threshold, the modified gaussians 156' are used to update the gaussian scene representation 154. However, if the loss value(s) do not satisfy a threshold, the partial updater 356 generates a second set of modified gaussians 156’ and repeats the process above to determine whether loss value(s) for the second set of modified gaussians 156’ satisfy the threshold. This process can be repeated iteratively until a termination condition is reached. The termination condition can include identification of a set of modified gaussians 156’ that satisfies the threshold, or performance of a specified number of iterations. If no set of modified gaussians 156’ that satisfies the threshold is identified after the specified number of iterations, the partial updater 356 can change to another approach to generate updates 357. or the update strategy selector 144 of FIG.3A can select a different update strategy (e.g., no update, a different type of partialQUALCOMM Ref. No. 2507839WO- 54 / 107 -update, or the full update 148). In some examples, rather than process one set of modified gaussians 156’ at a time, as described above, the partial updater 356 can generate multiple candidate sets of modified gaussians 156’ in parallel, and check which, if any, best match the image 351.
[0151] As one specific example, in FIG. 3C, the image 351 represents a portion of a soccer pitch and includes a goal and a ball 365. FIG. 3C also illustrates an example gaussian scene representation 370A of the gaussian scene representation 154 before the partial update, and an example gaussian scene representation 370B of the gaussian scene representation 154 after the partial update.
[0152] In this example, the ROI data 313 can include object data identifying an object (e.g., the ball 365) that has changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can identify a set of the gaussians 156 that represent the object, and the gaussian selector 354 can select the set of gaussians that represent the object as the set of gaussians that are to be updated. In the specific example shown in FIG. 3C, the gaussians 156A and 156B represent a ball 365 in the scene; thus, the selection data 353 includes the gaussians 156A and 156B.
[0153] As another example, the ROI data 313 can identify a first set of pixels that have changed or moved (as indicated by the sensor data 305). In this example, the mapping data 122 can include the pixel-to-gaussian mapping data of FIG. 2, which identifies a set of the gaussians 156 that correspond to the first set of pixels, and the gaussian selector 354 can select the set of gaussians that correspond to the first set of pixels as the set of gaussians that are to be updated. The selection data 353 can include the selected set of gaussians or information indicating how the partial updater 356 can access the selected set of gaussians, such as memory addresses of the selected set of gaussians. indices of the selected set of gaussians, etc.
[0154] The partial updater 356 is configured to modify7parameters of the selected set of gaussians (e.g.. the gaussians 156A and 156B in this example) based on the sensor data 305, the input 307, and optionally other data. For example, when the ROI data 313 indicates that an object in the scene has moved, the partial updater 356 can change location parameters (e.g., position and / or spatial orientation parameters) of the gaussians 156A and 156B that represent the object in the gaussian scene representation 154 inQUALCOMM Ref. No. 2507839WO- 55 / 107 -order to update the gaussian scene representation 154 to reflect movement of the object. In other cases, the partial updater 356 can change color parameters (e.g., spherical harmonics coefficients) and / or opacify parameters of the gaussians 156A and 156B to represent lighting changes in the scene.
[0155] A technical advantage of performing the partial update 150 rather than the full update 148 is that significant computer resources can be conserved. For example, when the update to be performed can be calculated or estimated based on the sensor data 305, the rendering operations associated with the projector 326 and the differentiable tile rasterizer 340, the loss calculations performed by the loss calculator 348, and back-propagation of gradient values 343, 345 based on the loss values can be omitted during the partial update 150. Further, even if the partial update 150 uses the rendering-based error calculator 359 to check one or more candidate modified gaussians 156’, only one or a few views are rendered, rather than views associated with all of the images 361 of training data 360 of FIG. 3B, saving significant rendering time and resources. Further, back-propagation of gradients based on the loss values can be omitted during the partial update 150, which also results in conservation of significant computing resources.
[0156] FIGs. 4A-4C are block diagrams illustrating aspects of the system 100 of FIG. 1, in accordance with some examples of the present disclosure. FIGs. 4A-4C illustrate additional details relating to inserting an object within a gaussian scene representation. FIGs. 4B and 4C illustrate additional details of portions of FIG. 4A. For clarify, FIGs.4A-4C are split over two pages.
[0157] In FIG. 4A, the gaussian object editor 170 is configured to access the gaussian scene representation 154 and the template library 172 and is coupled to one or more input source(s) 410 that are configured to provide the input data 176 to the gaussian object editor 170. The gaussian object editor 170 includes an instruction parser 420, a gaussian object detection circuit 430, and a gaussian object insertion circuit 450. The template library 172 includes a plurality of object templates, including an object template 471, an object template 472, one or more additional object templates, or a combination thereof. As explained with reference to FIG. 1. the gaussian object editor 170 may support editing of a gaussian scene representation, such as to insert one or more objects within a scene represented by the gaussian scene representation. In someQUALCOMM Ref. No. 2507839WO- 56 / 107 -embodiments, the gaussian object editor 170 can be a distinct circuit (e.g., a dedicated circuit) within the processor(s) 130 that is configured to perform some or all of the various operations described herein, in other embodiments, the gaussian object editor 170 can correspond to one or more other portions (e.g., general-purpose circuitry) of the processor(s) 130 that can perform some or all of the various operations described herein.
[0158] The instruction parser 420 is configured to receive input data 176 from the one or more input sources 410. In various embodiments, one or more of the input source(s) 410 are optional. The input sources 410 may include a text input source 411, an eye gaze input source 412, a gesture input source 413, one or more I / O device input source(s) 414, a microphone input source 415, or any combination thereof.
[0159] The text input source 411 is configured to provide text-based inputs related to inserting, moving, resizing, or any combination thereof, an object within a view of a scene of a gaussian scene representation. The text input source 411 can generate text data, such as text entered into a text input field of a graphical user interface. For example, the text data can include user feedback text or textual requests regarding a view of a scene (e.g.. “add a ball to the scene on a goal line'’ or “move the ball to the left”). In some cases, the text data can be input using a keyboard.
[0160] The eye gaze input source 412 is configured to provide user gaze-based inputs related to inserting, moving, resizing, or any combination thereof, an object within a view of a scene of a gaussian scene representation. The eye gaze input source 412 may be an eye gaze or eye tracking device, such as a screen-based eye tracker (e.g., a device placed near a display screen to track eye movement relative to the screen), a glasses-based eye tracker (e.g., a w earable device to track eye movements in real-world environments), a virtual reality (VR) headset eye tracker (e.g., a wearable device to track eye movement while a user is immersed in a virtual environment), or a remote eye tracking device that measures eye movement (e.g., a device that uses sensors to measure eye movement without requiring the user to wear any equipment). The eye gaze input source 412 may include sensors used to track eye movement and detect a direction and / or focus of the user’s gaze (e.g.. where the user is looking). Non-limiting examples of sensors that may be used by the eye gaze input source 412 include cameras (e.g., to detect reflections and track movement of the eyes), gy roscopes (e.g., to detect headQUALCOMM Ref. No. 2507839WO- 57 / 107 -movements), infrared sensors (e.g., infrared light sources configured to create reflections on the eye that may be used to detect reflections and track movement of the eyes), electrooculography (EOG) sensors (e.g. to measure electrical signals between the cornea and retina that may be used to measure eye movement), other types of sensors, or a combination thereof. For example, the eye gaze input source 412 may utilize a training phase that calibrates detection of the user’s gaze (e.g.. locations on a display, such as the display 138 of FIG. 1).
[0161] During operation, the eye gaze input source 412 may output gaze data as a comma separated value (.csv) file, a heatmap showing locations where the user’s gaze was focused over time, gaze plots, opacity maps, or other types of data that indicates areas of interest based on the user’s eye movements. For example, a heatmap may show regions of interest on the display in a first color or set of colors (e.g., red, orange, yellow) and show other regions of the display that are not of interest in a second color or set of colors (e.g., blue, teal. etc.). The regions of interest may relate to regions of the display in which the user focused their gaze for longer periods of time relative to the regions of the display not of interest. The gaze plot may show a progression of the user’s gaze over the display during a period of time. Regions of the display where the user’s gaze spent more time viewing (e.g.. based on eye movement or other gaze information) may indicate objects of interest, such as an object for which an edit action may be desired. Similarly, opacity maps may highlight regions of the display where the user looked more frequently and regions of the display the user paid little or no attention to may be blacked out. The.csv file may contain the raw eye gaze data captured by the one or more sensors of the eye gaze input source 412 from which the heatmap, the gaze plot(s), the opacity map(s), or other gaze information is generated.
[0162] In some embodiments, the eye gaze input source 412 may also output gaze direction information. For example, LazerGaze is an opensource video-based tool for estimating gaze vectors (e.g., a direction in which the user is looking or the user’s point of view with reference to the display on which an image rendered from a gaussian scene representation is shown). The direction information may be used to estimate or determine a direction toward which an object in a view of a scene should be inserted or moved during editing of a gaussian scene representation. In some cases, the direction information may be used in conjunction with another input source (e.g., the gestureQUALCOMM Ref. No. 2507839WO- 58 / 107 -input source 413) to identify a location within a scene where the user would like an object positioned, to identify a desired spatial orientation of the object, to identify a desired scale of the object, or any combination thereof. For example, the user can look in a direction and use one or more gestures to indicate a location along the direction to position and orient an object.
[0163] The gesture input source 413 may be configured to provide gesture-based inputs related to inserting, moving, resizing, or any combination thereof, an object within a view of a scene of a gaussian scene representation. For example, the gesture input source 413 may include an inertial measurement unit (IMU) and be intended to be held by or attached to a user. When the user moves, the gesture input source 413 can detect the movement and translate that movement into requests that are sent to the gaussian object editor 170. To illustrate, the user can gesture using the gesture input source 413 to provide a desired scale of an object by raising their hand until the object reaches the desired scale. As another example, the user can point to a desired location within a scene where they would like an object to be positioned and a desired location toward which they would like the object to be oriented. The gesture input source 413 sends data indicating movement of the user to the gaussian object editor 170. Other types of gesture input sources, such as a touchscreen device configured to select an object or objects in a view, are also contemplated.
[0164] The I / O device input source(s) 414 may be configured to receive inputs related to inserting, moving, resizing, or any combination thereof, an object within a view of a scene of a gaussian scene representation. For example, the user may click a mouse pointer to select object(s) or trace around one or more locations. The user may also use a keyboard to provide one or more alphanumeric characters as input to a text field of a graphical user interface, as explained above. As another non-limiting example, the I / O device input source(s) 414 may include a stylus that may be used to interact with a display or touchpad to select or otherwise indicate locations or sizes of objects to be added in the gaussian scene representation. A camera may also be used to capture an image of the object(s) to be inserted within the gaussian scene representation. It is noted that the particular example inputs described as provided via I / O device input source(s) 414 have been provided for purposes of illustration, rather than by way of limitation and that other types of inputs may be provided via the I / O device input source(s) 414.QUALCOMM Ref. No. 2507839WO- 59 / 107 -
[0165] The microphone input source 415 may be configured to receive audio inputs related to inserting, moving, resizing, or any combination thereof, an object within a view of a scene of a gaussian scene representation. The microphone input sources 415 may include one or more microphones to capture audio data, such as speech. For example, a user may speak into the microphone to say the name of the object(s) or describe the object(s) to be edited in the gaussian scene representation. In some cases, audio inputs related to editing a gaussian scene representation may be processed using a text-to-speech algorithm to convert the audio input into text input. In some embodiments, the speech data may be used in audio data form, rather than converting the speech data to text.
[0166] It is to be understood that the particular example input sources described above have been provided for purposes of illustration, rather than by way of limitation and that other input sources may be utilized in accordance with the concepts described herein. Further, it is noted that any of the above-described input sources may be remote from a device including the gaussian object editor 170. For example, the gaussian object editor 170 may be incorporated into a device having a modem and the input data 176 may be received from a remote device including one or more of the above-described input sources via the modem.
[0167] The instruction parser 420 receives and processes the input data 176 to detect a request to add an object in a view of a scene of the gaussian scene representation 154. For example, the instruction parser 420 detects, based on analysis of the received input data 176, object identifier(s) 424 indicating an object to be inserted within a scene. In some cases, the object identifier(s) 424 indicate a class of object (e.g., a ball or a dog). In other cases, the object identifier(s) 424 indicate an identifier of an object template in the template library 172. In some cases, the instruction parser 420 detects, based on analysis of the received input data 176, a location reference 426 that indicates a location where the object is to be positioned and a spatial orientation of the object. For example, in some cases, the location reference 426 indicates a spatial orientation of the object, a direction in which the object is to be positioned, boundaries of a region in which the object is to be positioned, or coordinates where a center of the object is to be positioned. In some cases, the location reference 426 is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.QUALCOMM Ref. No. 2507839WO- 60 / 107 -
[0168] The gaussian object detection circuit 430 receives object identifier(s) 424, and, optionally, gaussian(s) 432 of the gaussian scene representation 154 (e.g., in cases where the gaussian(s) 432 are used as location references or scale references for addition of an object), the location reference 426 (e.g., in cases where a location is indicated by a user), or both. The gaussian object detection circuit 430 generates parameters 434 including a scale parameter(s) that indicates a scale (e.g., a target size) of the object, a location parameter(s) that indicates a location (e.g., a target location) and spatial orientation of the object, or both. For example, in some cases, the gaussian object detection circuit 430 generates the scale parameter(s) based on information associated with a conventional size of an object (e.g.. baseballs have a circumference of nine inches) identified by the object identifier(s) 424. In the example, the object is sized in the scene relative to a conventional size of another object (e.g., soccer goals are twent -four feet wide and eight feet tall) in the scene. As a second example, the gaussian object detection circuit 430 generates the scale parameter(s) based on an amount of available space between other objects corresponding to the gaussian(s) 432. In the second example, the gaussian object detection circuit 430 generates the scale parameter(s) so that the object fills all of the available space or a predetermined percentage of the available space. As a third example, in some cases, the gaussian object detection circuit 430 calculates the location parameter(s) based on positions or spatial orientations of other objects in the scene corresponding to the gaussian(s) 432. In the third example, the gaussian object detection circuit 430 generates the location parameter(s) by selecting a location where the object does not overlap with other objects. In some cases, the spatial orientation of the object is based on spatial orientations of one or more other objects (e.g., adding a human to the scene facing another human). As a fourth example, in some cases, the gaussian object detection circuit generates the location parameter(s) based on the location reference 426 and positions of the gaussian(s) 432. In the fourth example, the location reference 426 describes a location and spatial orientation with reference to one or more objects in the scene (e.g., add a ball three feet from a goal post with a black pentagon facing the goal post). In a fifth example, the gaussian object detection circuit 430 extracts the location parameter(s) from the location reference 426 (e.g., by converting coordinates indicated by the location reference 426 into a form corresponding to the gaussian scene representation 154).QUALCOMM Ref. No. 2507839WO- 61 / 107 -
[0169] In some cases, the location parameter(s) of the parameters 434 indicate a spatial orientation of the object, a direction in which the object is to be positioned, boundaries of a region in which the object is to be positioned, or coordinates where a center of the object is to be positioned. In some cases, the scale parameter(s) of the parameters 434 indicate a size of the object, boundaries of a region of the scene where the object is to be added, a scale of the object relative to a scale of another object in the scene, or a combination thereof.
[0170] The gaussian object insertion circuit 450 includes scale parameter(s) 451, location parameter(s) 453, and, optionally, an error correct circuit 455. The scale parameter(s) 451 indicates a size of the object to be inserted. In some cases, the gaussian object insertion circuit 450 generates the scale parameter(s) 451 based on scale parameter(s) indicated by the parameters 434 or scale parameter(s) indicated by the template(s) 174. In some cases, the gaussian object insertion circuit 450 generates the scale parameter(s) 451 based on one or more scale parameters associated with another object in the scene, an amount of available space in the scene indicated by one or more location parameters associated with another object in the scene (e.g., corresponding to the gaussian(s) 432), or any combination thereof. The location parameter(s) 453 indicate a location where the object is to be inserted. In some cases, the gaussian object insertion circuit 450 generates the location parameter(s) 453 based on location parameter(s) indicated by the parameters 434, location parameter(s) indicated by the template(s) 174. In some cases, the gaussian object insertion circuit 450 generates the location parameter(s) 453 based on location parameter(s) associated with another object in the scene (e.g., corresponding to the gaussian(s) 432). For example, in some cases, the location parameters of the parameters 434 or the template(s) 174 indicate the location where the object is to be inserted and a spatial orientation of the object relative to the location of another object in the scene.
[0171] As discussed with reference to FIGS. 1-2, in some cases, the gaussian object editor 170 receives an object template (e.g., template(s) 174) associated with the object from the template library 172. In other cases, no object template is received (e.g.. because no template associated with the object is stored at the template library 172). If template(s) 174 associated with the object are not received, the gaussian object insertion circuit 450 is configured to generate an object template (e.g., template(s) 174) based onQUALCOMM Ref. No. 2507839WO- 62 / 107 -the scale parameter(s) 451. the location parameter(s) 453, the input data 176, or any combination thereof. For example, the gaussian object insertion circuit 450 is configured to fill out an empty object template with the scale parameter(s) 451, the location parameter(s) 453, class data associated with the object ID(s) 424 indicated by the input data 176, or any combination thereof.
[0172] The gaussian object insertion circuit 450 is configured to generate updated gaussian(s) 390 based on the template(s) 174 (e.g., received or generated), the scale parameter(s) 451, the location parameter(s) 453, or any combination thereof. For example, the object template (e.g.. the template(s) 174) indicates attributes (e.g., properties) of one or more gaussians that can be used to represent the object. To illustrate, as described with reference to FIG. 3A, the attributes of a gaussian can indicate a center position of a gaussian (e.g., the position coordinates 180-182 of FIG.3A), a size and shape of the ellipsoid of the gaussian (e.g., the scale parameters 183-185 of FIG. 3 A), a spatial orientation of the ellipsoid (e.g., the rotation parameters 186-189 of FIG. 3A), an opacity limit of the gaussian (e g., the opacity parameter 190), color information of the gaussian (e.g., the color parameters 191-193 of FIG. 3A), or a combination thereof.
[0173] The gaussian object insertion circuit 450 generates one or more updated gaussian(s) 390 having the attributes indicated by the object template. In some examples, the gaussian object insertion circuit 450 sets size parameters (e.g., the scale parameters 183-185) of the updated gaussian(s) 390 based on the scale parameter(s) 451 indicating a target size of the object. To illustrate, an aggregate size of the updated gaussian(s) 390 corresponds to the target size of the object. In some examples, the gaussian object insertion circuit 450 sets location parameters (e.g., the position coordinates 180-182 and the rotation parameters 186-189) of the updated gaussian(s) 390 based on the location parameter(s) 453 indicating a target location of the object. To illustrate, an aggregate location of the updated gaussian(s) 390 corresponds to the target location of the object.
[0174] The gaussian object insertion circuit 450 is configured to add the updated gaussian(s) 390 to the gaussian scene representation 154 including the gaussians 156 to produce a modified gaussian scene representation (e.g., the modified gaussian sceneQUALCOMM Ref. No. 2507839WO- 63 / 107 -representation 154' of FIG. 1). For example, the modified gaussian scene representation 154’ includes the updated gaussian(s) 390 in addition to the gaussians 156.
[0175] The error correct circuit 455 is configured to generate error metrics based on candidate views rendered based on the modified gaussian scene representation. In some cases, the error metrics indicate overlapped objects in the candidate views. For example, in some cases, a neural network of the error correct circuit 455 can generate an overlap metric (e.g., an error metric) that indicates a confidence level of overlapped objects in a region of a scene. When the confidence level exceeds a threshold, an error is detected. In the example, a view of the scene can be rendered and a neural network inference can be performed to identify a region of the scene corresponding to the error. The error correct circuit 455 is configured to perform error correction based on one or more error metrics. The error correct circuit 455 is configured to adjust (e.g., modify, update, or correct) a position of the object added to the gaussian scene representation 154. a spatial orientation of the object, a size of the object, or any combination thereof. For example, the error correct circuit 455 can move the object, rotate the object, reduce the size of the object, or any combination thereof, so that the confidence level no longer exceeds the threshold (e.g., because objects are no longer overlapped). In some cases, the error correct circuit 455 performs the error correction based on user input (e.g., instructions to move the object, rotate the object, reduce the size of the object, or any combination thereof). In some cases, the user input is generalized (e.g., “reduce the size of the object until no collision is detected’’). In other cases, the user input specifies how the object should be adjusted (e.g., “reduce the size of the object by three percent, rotate it five degrees, and move it three feet to the left in the scene”).
[0176] FIG. 4B is a diagram of an exemplary embodiment of the instruction parser 420 of FIG. 4A. The instruction parser 420 includes a speech-to-text engine 460, an LLM 461, transformer(s) 462, computer vision model(s) 463, a prompt generator 464, or other types of data processing logic for processing the input data 176. In some embodiments, one or more portions of the illustrated instruction parser 420 are not present. For example, in some embodiments, the instruction parser 420 does not include the computer vision model(s) 463.QUALCOMM Ref. No. 2507839WO- 64 / 107 -
[0177] The speech-to-text engine 460 is configured to generate text data based on audio content (e.g.. speech data) received from the microphone input source(s) 415. For example, suppose that the gaussian scene representation being edited is the gaussian scene representation 154 depicting a scene having a soccer goal. The user may provide speech data via the microphone input source(s) 415 such as “add a soccer ball next to the goal." The speech-to-text engine 460 converts this audio data to text data, which may be in the form of “add a soccer ball next to the goal." In some embodiments, the text data output by the speech-to-text engine 460 is provided to the LLM 461. In some embodiments, the speech-to-text engine 460 is implemented using hidden Markov models (HMMs), a neural network, deep learning models, transformers (e.g., one of the transformers ) 462), other types of models, or a combination thereof.
[0178] In some embodiments, the prompt generator 464 of the instruction parser 420 processes the text data output by the speech-to-text engine 460 to generate a prompt that is provided as input to the LLM 461. For example, the prompt output by the prompt generator 464 may be of the form “Edit Action = add; Edit object(s): object having a class related to soccer balls”. In some embodiments, the prompt generator 464 is configured to generate the prompt using the LLM 461. For example, the prompt generator 464 may receive the text data output by the speech-to-text engine 460 and pass the text data to the LLM 461 with a request such as “generate a prompt to edit [gaussian scene representation identifier] based on [text data]”, where [gaussian scene representation identifier] is an identifier of the gaussian scene representation 154 (or an identifier of an instance of the gaussian scene representation 154) and [text data] is the output of the speech-to-text engine 460. As another example, the prompt generator 464 generates a request of “add a ball next to a goal” from the input data 176 is organized by the LLM 461 to outputs of [target: “ball”, size: “1”, center: “goal”, relative position: “-180 degrees, 0.1m”].
[0179] The transformer(s) 462 may include transformer models configured to receive, as input data, outputs of one or more of the input sources 410 and to output data that may be used to generate a prompt to the LLM 461. For example, a transformer model of the transformer(s) 462 may be trained to receive eye gaze data (e.g., raw eye gaze data in.csv form, a heatmap, an opacity map, a gaze plot, etc.) from the eye gaze input source(s) 412 and output one or more regions of interest based on the eye gaze data. TheQUALCOMM Ref. No. 2507839WO- 65 / 107 -regions of interest may relate to areas within a display the user's gaze focused on. The transformer 462 may be configured to weight or rank the regions of interest based on an amount of focus the user’s gaze gave to each region. For example, a region of the display the user spent more time looking at may be indicative of an area of the display corresponding to a target location where an add action is to be performed for an object and therefore, weighted more heavily or ranked more highly, as compared to a region of the display the user spent less time looking at. In some embodiments, the transformer(s) 462 may be configured to perform natural language processing operations, such as to convert text data output by the speech-to-text engine 460 to tokens via a tokenization process. The tokenization process may convert the text data to a set of tokens that may be used to generate a set of vectors based on an embedding lookup table. The set of vectors may represent the text data in a format suitable for processing by the LLM 461 or other types of machine learning (ML) and Al models. In some embodiments, the LLM 461 may include a transformer to perform natural language processing of text data submitted to the LLM 461 for processing, rather than utilizing a standalone transformer for purposes of preparing text data for submission to the LLM 461. In additional or alternative embodiments, the prompt generator 464 may incorporate an ML or Al model to perform natural language processing of the prompt generated for submission to the LLM 461. Other implementations for performing natural language processing may also be utilized by systems in accordance with the concepts disclosed herein.
[0180] In addition to natural language or speech-to-text processing, the transformer(s) 462 may also be utilized to process other types of input data 176. For example, the transformer 462 may include a vision transformer trained to perform image recognition. The image recognition may be configured to analyze an image, such as an image of a view generated from a gaussian scene representation, to identify objects within the image, their locations, and the like. Such information may be used to generate a set of object candidates in the image, which may be used to identify spaces where object(s) can be added within the gaussian scene representation. Optionally, the vision transformer or another type of computer vision model, may be used during generation of gaussian scene representations to classify' objects in the scene. The object classifications may be stored as part of data (e.g., metadata) associated with the object. As explained in more detail below, information associated with spaces where object(s) can be added, theQUALCOMM Ref. No. 2507839WO- 66 / 107 -objects in the scene, or both, along with a class of the object, may be used to identify gaussians related to object(s) in the scene between which another object can be added in an add action to be performed.
[0181] The transformers 462 may also utilize vision transformers to perform gaze estimation (e.g., TransGaze and the like). The gaze estimation by the transformer(s) 462 may receive an input image from an input data source, such as a camera (e.g., one of the I / O device input source(s) 414), where the image show s a user’s face and facial features. The transformer(s) 462 analyzes the input image to identify a direction of the user’s gaze or other eye gaze information. As explained above, the direction may be used during editing of a gaussian scene representation to determine a point of view from w hich a view- of the edited gaussian scene should be generated. This may enable error or anomaly detection to be performed and any errors corrected.
[0182] In some embodiments, the computer vision model(s) 463 of the instruction parser 420 may perform computer-vision tasks similar to those described above with reference to transformers. For example, the computer vision model(s) 463 may detect objects in the scene, be used to capture eye gaze data, or other types of analysis that may be used to perform editing of a gaussian scene representation, such as during anomaly and error detection.
[0183] In some cases, based on feedback (e.g., user feedback), the LLM 461 is further trained, fine-tuned, or both. For example, based on insertion of an apple in response to a request to add a ball to a scene, examples of balls are provided to the LLM 461 to teach the LLM 461 to provide better results in the future. Further, in some cases, it is desirable to fine-tune outputs of the LLM 461. To illustrate, a set of LoRA parameters are applied to portions (e.g., one or more layers) of a pretrained foundation model corresponding to the LLM 461 to adapt outputs of the foundation model to have particular characteristics, resulting in changes to results produced by the LLM 461. In some cases, the LoRA parameters personalize output of the LLM 461 to preferences of a user. For example, the user may be more interested in soccer than baseball or basketball. As a result, in response to a request to add a ball to a scene, it may be desirable to increase a likelihood of the LLM 461 adding a soccer ball to a scene rather than a baseball or a basketball. In the example, a set of LoRA parameters are applied toQUALCOMM Ref. No. 2507839WO- 67 / 107 -the LLM 461 so the LLM 461 is more likely to select a soccer ball when presented with a generic request to add a ball to a scene. In addition to. or instead of. fine tuning the LLM 461, output of the LLM 461 can be personalized to preference of the user by generation of context parameters that are used with input from the user to generate the output. For example, when the user is more interested in soccer than in baseball, the context parameters may indicate that '’ball’' generally refers to a soccer ball.
[0184] In some embodiments, the gaussian object editor 170 and the LLM 461 may operate as part of an agentic Al system. As used herein, an agentic Al system refers to an Al-based system in which an LLM or other Al model acts as an autonomous agent that perceives inputs from a user or environment, reasons over those inputs in combination with a memory of prior interactions, selects and invokes one or more tools or actions to accomplish a user-specified goal, and iteratively refines its actions based on feedback received from the user or the environment. Unlike conventional Al inference systems that produce a single output in response to a single input, an agentic Al system operates across multiple turns of interaction, maintaining context and state across the full interaction sequence to accomplish tasks that may require a series of interdependent steps, corrections, or confirmations. In the context of the present disclosure, the agentic Al system leverages the LLM 461 as the central reasoning agent and incorporates the edit tools, including object insertion tools, of the gaussian object editor 170 as callable actions available to the agent, enabling the LLM 461 to interpret natural language instructions from a user and autonomously invoke the object insertion tools of the gaussian object editor 170 as a callable action available to the agent, enabling the LLM 461 to interpret natural language instructions from a user and autonomously invoke the appropriate deletion operations on the gaussian scene representation 154 to fulfill the user's intent. In some embodiments, the delete tool corresponds to or invokes the gaussian object insertion circuit 450 of the gaussian object editor 170, as described above.
[0185] In some embodiments, the agentic Al system enables multi-turn, context-aware insertion of objects within the gaussian scene representation 154. Because the agentic Al system retains session memory of prior user instructions and edit actions, it is able to resolve subsequent instructions that implicitly reference prior interactions, in a manner not achievable using conventional instruction-based editing methods that process eachQUALCOMM Ref. No. 2507839WO- 68 / 107 -input independently. For example, a user may issue an initial instruction such as ‘‘add a ball in front of the goal,” which may trigger insertion of gaussians 156 corresponding to a ball object into the gaussian scene representation 154 at a position in between a current view of the user and gaussians 156 representing a goal followed by a corrective instruction such as “sorry, that is the wrong type of ball, use the correct one,” followed by a preference revision such as “move the ball to the left,” and then a further modification such as “now make it larger.” The session memory of the agentic Al system allows the LLM 461 to maintain the context of each prior insertion action and user correction across the full interaction sequence and resolve each subsequent instruction accordingly.
[0186] In some embodiments, prior to executing insertion of an identified object into the gaussian scene representation 154, the agentic Al system may optionally render a view of the scene in which the object proposed for insertion is visually marked or distinguished from other objects in the scene. For example, the Tenderer 158 may generate a view of the scene in which the proposed object is inserted and highlighted using a bounding box drawn around the object, a modification of one or more color parameters of the gaussians associated with the object (e.g., overlaying the object with a distinct highlight color or changing the color of the object to red or some other color to make the target object for insertion more readily visible), a modification of one or more transparency parameters of the gaussians associated with the object, or another visual indicator technique. This preliminary rendering may be presented to the user prior to executing the insertion, allowing the user to confirm that the correct object has been identified. In response to a confirmation from the user that the identified object is correct (e.g., via a text input such as “yes,” a gesture input, a voice input, or another confirmation input), the agentic Al system may proceed to invoke the insertion tool, which causes the gaussian object insertion circuit 450 to perform the insertion of the identified object into the gaussian scene representation 154. as described above. The optional confirmation step is particularly advantageous in scenarios in which an object to be inserted is of a type or class where one of multiple potential objects (e.g., one of multiple types of balls) could be inserted or in which multiple potential object scales or object locations within the gaussian scene representation 154 could be interpreted from the user’s instruction.QUALCOMM Ref. No. 2507839WO- 69 / 107 -
[0187] In some embodiments, the agentic Al system supports an undo operation that enables a user to reverse a previously performed insertion of an object within the gaussian scene representation 154 during the current editing session. For example, after one or more objects have been inserted into the gaussian scene representation 154, the user may issue an undo instruction (e.g., “undo,'’ “revert that change,” “put it back,” or another undo instruction) that causes the agentic Al system to restore the gaussian scene representation 154 to its state prior to the most recently performed insertion action. The undo operation is performed by the gaussian object editor 170 by retrieving restore data stored in the memory 120 at the time the edit action was performed, and using the restore data to remove the previously inserted gaussians within the gaussian scene representation 154. In some embodiments, the agentic Al system maintains an ordered record of insertion actions performed during the current editing session as part of the session memory, enabling the user to sequentially undo multiple prior insertions.
[0188] In some embodiments, the session memory of the agentic Al system may be implemented as a structured record of prior user prompts, LLM 461 responses, invoked insertion tool operations corresponding to invocations of one or more sub-components of the gaussian object insertion circuit 450, confirmation and rejection inputs, undo operations, and the resulting edit actions performed on the gaussian scene representation 154 during the current editing session, stored in the memory' 120 of FIG. 1 and retrieved by the LLM 461 as context for each new interaction.
[0189] A technical advantage of the agentic Al system described herein is the ability to support complex, multi-step object insertion workflows through natural language interaction in a manner not achievable using conventional instruction-based editing methods. By maintaining session memory, the agentic Al system resolves ambiguous or corrective user instructions that implicitly reference prior edit actions, reducing the burden on the user to provide complete and unambiguous instructions with each input. A further technical advantage is the optional pre-insertion confirmation step, in which the agentic Al system renders a view of the scene with the proposed object visually marked prior to executing the insertion. This reduces the occurrence of erroneous insertions, particularly when inserting objects of a class in which multiple different objects could correspond to user instructions, thereby reducing unnecessary- consumption of computing resources that would otherwise be required to detect andQUALCOMM Ref. No. 2507839WO- 70 / 107 -correcl an erroneous insertion, such as re-invoking background restore logic or regenerating gaussians via the prompt generator 464. A further technical advantage is the undo operation, which enables a user to efficiently reverse one or more prior insertions by leveraging restore data already stored in the memory 120, without requiring the user to regenerate or re-specify the gaussian scene representation 154. Collectively, these features reduce user error, reduce unnecessary computational overhead associated with correcting erroneous insertions, and provide a more natural and efficient interface for performing object insertion operations on gaussian scene representations 154.
[0190] FIG. 4C is a diagram of an exemplary embodiment of the object template 471 of FIG. 4A. The object template 471 includes gaussian(s) 480 and class data 481.Optionally, the object template 471 also includes a scale reference 482, a location reference 483, or both.
[0191] If the object template 471 indicates that multiple gaussians 480 are to be used to depict the object associated with the object template 471, the object template 471 indicates relative positions of the gaussians 480 to each other when represented in a gaussian scene representation.
[0192] The object template 471 indicates attributes (e g., properties) of each of the gaussian(s) 480 and the class data 481 indicates a class associated with the object that the gaussian(s) 480 can be used to represent. For example, in some cases, the class data 481 indicates a description of the object (e.g., a ball, a house, or a dog breed). To illustrate, the class data 481 can indicate a type of the object (e.g., a ball), a use of the object (e.g., sports equipment, soccer, or both), a person associated with the object (e.g., Jim), or a combination thereof. In some embodiments, the class data 481 can be used to select the object template 471 based on input data 176 that includes a description of an object but does not specify a particular object template within the template library 172. For example, an object template 471 indicating a description that matches the description included in the input data 176 can be selected.
[0193] The scale reference 482 (e.g., a template size parameter) indicates a size of the object. For example, if the object template 471 corresponds to a baseball, the scale reference 482 could indicate that the baseball has a circumference of nine inches. When the scale reference 482 is present in a received object template (e.g., the template(s)QUALCOMM Ref. No. 2507839WO- 71 / 107 - 174), the gaussian object editor 170 is configured to set the scale parameter(s) 451 based on the scale reference 482.
[0194] The location reference 483 (e.g., a template location parameter) indicates a position and spatial orientation of the object. For example, the location reference 483 could indicate a coordinate position of a center of the object, boundaries of a region within a scene within the object is to be positioned, or a distance the object is to be positioned from another object. To illustrate, if the object is a pole of a goal, the location reference 483 could indicate that the pole is to be positioned twenty-four feet from another pole of the goal. When the location reference 483 is present in a received object template (e.g., the template(s) 174), the gaussian object editor 170 is configured to set the location parameter(s) 453 based on the scale reference 483. In some cases, the location reference 483 indicates a spatial orientation of the object (e.g., a particular gaussian should be the closest gaussian of the object to a particular coordinate position in the scene).
[0195] A technical advantage of the gaussian object editor 170 includes using an object template to add gaussians representing an object to a gaussian scene representation. The gaussians can be added based on, in some cases, less input data, as compared to conventional systems. The gaussian object editor can receive input data that includes a request to add an object to the gaussian scene representation and, when information such as gaussians to use to represent the object, a scale of the object within the scene, a location of the object within the scene, or any combination thereof, is not provided in the request, obtain (e.g., retrieve or generate) the information. Additionally, in some scenes, a technical advantage of the gaussian object editor 170 having the error correct circuit 455 includes a reduction of collisions between objects when adding an object to a view of a scene represented by a gaussian scene representation.
[0196] FIG. 5 illustrates particular aspects of an example 500 of object insertion within a gaussian scene representation by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The images 351 and 351’ are shown for clarity purposes. In some embodiments, the system 100 does not store the image 351, does not generate the image 351, or both.QUALCOMM Ref. No. 2507839WO- 72 / 107 -
[0197] In the example 500, an image 351 depicts a soccer goal. A scene corresponding to the image 351 is represented by the gaussian scene representation 370A. including a set of gaussians that depict the goal 514. In the example 500, input data is obtained that includes a request to modify the gaussian scene representation 370A to add a soccer ball into the scene. The parameters 434 identify one or more template(s) 174 corresponding to the ball 365. which includes a particular gaussian 520. Further, the parameters 434, the template(s) 174, or both, include scale parameter(s) and location parameter(s). As a result, the gaussian object insertion circuit 450 sends (e.g., adds) updated gaussians (e.g., the updated gaussians 390 of FIG. 3A) to the gaussian scene representation 370A so that, in the scene, the ball 365 is sized based on the one or more scale parameters (e.g., gaussians corresponding to the ball are sized, gaps between the gaussians corresponding to the ball are sized, or both) and positioned and oriented based on the location parameter(s) (e.g., positioned at a specified location and oriented so that the particular gaussian 520 is the closest gaussian of the ball 365 to the goal 514) in a region 512. As a result, the gaussian scene representation 370B, which includes the set of gaussians that depict the goal 514 and the set of gaussians that depict the ball 365, is produced. The image 351 ’ corresponds to a rendered view of a scene represented by the gaussian scene representation 370B and shows a soccer ball and the soccer goal.
[0198] FIG. 6 illustrates particular aspects of an example 600 of object insertion within a gaussian scene representation by the system 100 of FIG. 1, in accordance with some examples of the present disclosure. The images 351 and 351’ are shown for clarity purposes. In some embodiments, the system 100 does not store the image 351, does not generate the image 351, or both.
[0199] In the example 600, an image 351 depicts a soccer goal and goal box lines. A scene corresponding to the image 351 is represented by the gaussian scene representation 370A, including a set of gaussians that depict the goal 614, goal box lines 616, and a region 618 bounded by the goal box lines 61. In the example 600, input data is obtained that includes a request to modify the gaussian scene representation 370A to add a soccer ball into the scene. The parameters 434 identify one or more template(s) 174 corresponding to the ball 365, which includes a particular gaussian 620. However, unlike the parameters 434 of FIG. 5, the parameters 434 and the template(s) 174 of FIG.6 do not include scale parameter(s) and location parameter(s) of the ball 365. As aQUALCOMM Ref. No. 2507839WO- 73 / 107 -result, the gaussian object insertion circuit 450 generates the scale parameter(s) and the location parameter(s). In the example 600. the gaussian object insertion circuit 450 sets the scale parameter(s) associated with the ball 365 based on scale parameter(s) associated with the goal 614, scale parameter(s) associated with a length or a thickness of the goal box lines 616, an area associated with the region 618, or any combination thereof. Additionally, the gaussian object insertion circuit 450 sets the location parameter(s) based on a proximity to the goal 614, a proximity to the goal box lines 616, or both, so that the ball is positioned in a region 612 and oriented so that the particular gaussian 620 is the closest gaussian of the ball 365 to the goal 614. As a result, the gaussian scene representation 370B, which includes the set of gaussians that depict the goal 614, the set of gaussians that depict the goal box lines 616, and the set of gaussians that depict the ball 365, is produced. The image 351’ corresponds to a rendered view of a scene represented by the gaussian scene representation 370B and shows a soccer ball, the soccer goal, and the goal box lines.
[0200] FIG. 7 is a particular embodiment of a method 700 of inserting an object within a gaussian scene representation that may be performed by the system of FIG. 1, in accordance with some examples of the present disclosure. For example, one or more operations of the method 700 may be performed by the gaussian object editor 170 of FIG. 1, another portion of the processor(s) 130, or any combination thereof. In some examples, the method 700 can be initiated, performed, or controlled by the processor(s) 130 executing instructions to perform operations of the gaussian object editor 170. As another example, the method 700 can be initiated, performed, or controlled by the controller 160.
[0201] The method 700 includes, at block 702, obtaining input data including a request to modify a gaussian scene representation to add an object in a view of a scene. For example, the gaussian object editor 170 of FIG. 1 can obtain input data 176 including a request to modify the gaussian scene representation 154 to add an object in the view 136 of the scene 132.
[0202] The method 700 also includes, at block 704. identifying, based on the request, an object template associated with the object. For example, the gaussian object editor 170 of FIG. 1 can identify the object template 471 of FIGs. 4A and 4C. As another example,QUALCOMM Ref. No. 2507839WO- 74 / 107 -the gaussian object editor 170 can generate an object template associated with the object. In some cases, the object template is generated based on a determination that a gaussian object template library (e.g., the template library 172 of FIG. 1) lacks a stored object template corresponding to the object.
[0203] The method 700 also includes, at block 706. obtaining, based on the object template, gaussians that represent the object. For example, the gaussian object editor 170 of FIG. 1 can obtain gaussians based on the object template 471 of FIGs. 4A and 4C by receiving the object template 471 as the template(s) 174 and generating gaussians (e.g., the updated gaussians 390 of FIG. 3A) based on gaussian attributes indicated in the object template 471. As another example, the gaussian object editor 170 can generate gaussians (e.g., the updated gaussians 390) based on gaussian attributes indicated in an object template 471 generated by the gaussian object editor 170 of FIG.1.
[0204] The method 700 also includes, at block 708, adding the gaussians to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 of FIG. 1 can add the generated gaussians (e.g., the updated gaussians 390) to the gaussians 156 of the gaussian scene representation 154 to produce the modified gaussian scene representation 154’ including the modified gaussians 156’.
[0205] In some cases, adding the gaussians to the gaussian scene representation includes sizing the object based on scale parameter(s) that indicate boundaries of a first region of the scene, where the gaussians are added within the boundaries of the first region. For example, adding the gaussians to the gaussian scene representation 370A of FIG. 5 includes sizing the ball 365 based on the parameters 434 including the scale parameter(s) 451 of FIG. 4A that indicates boundaries of the region 512 where the ball 365 is to be depicted.
[0206] In various embodiments, different techniques are used to set the scale parameter(s) (e.g., the scale parameter(s) 451) of the object. In a first example, the scale parameter(s) are set based on second scale parameter(s) associated with a second object represented by the gaussian scene representation. To illustrate, the scale parameter(s) 451 of the ball 365 of FIG. 6 are set (e.g., derived) based on scale parameter(s)QUALCOMM Ref. No. 2507839WO- 75 / 107 -associated with the goal 614. For example, a ratio between a conventional goal and a conventional ball is obtained (e.g.. calculated or retrieved) and the scale parameter(s) 451 are set based on the ratio and the scale parameter(s) of the goal 614. In a second example, the scale parameter(s) are set based on location parameter(s) of gaussians associated with a second object represented by the gaussian scene representation. To illustrate, the scale parameter(s) 451 are set (e.g., derived) based on location parameter(s) associated with the goal 614 of FIG. 6. For example, an amount of available space in the scene is calculated and the scale parameter(s) 451 are set based on the amount of available space such that the ball 365 does not overlap with the goal 614. In a third example, the scale parameter(s) are set based on a scale reference associated with the object template. For example, the object template 471 of FIGs. 4A and 4C includes a scale reference that is used to set the scale parameter(s) 451. In a fourth example, the scale parameter(s) are set based on a scale reference included in the input data. For example, the input data 176 of FIG. 4A includes or indicates the scale parameter(s) 451.
[0207] In some cases, adding the gaussians to the gaussian scene representation includes positioning and orienting the object based on location parameters). For example, adding the gaussians to the gaussian scene representation 370A of FIG. 5 includes positioning and orienting the ball 365 based on the parameters 434 including the location parameter(s) 453 of FIG. 4A. In some cases, the location parameter(s) indicate a position of a center of the object, where the gaussians of the object are positioned around the center. For example, the location parameter(s) 453 indicate a coordinate position in the scene of FIG. 5 around which gaussians representing the ball 365 are positioned. In some cases, the location parameter(s) indicate boundaries of a region of the scene, where the gaussians are added within the boundaries. For example, the location parameter(s) 453 indicate a region of the scene of FIG. 5 within which gaussians representing the ball 365 are added. In some examples, the gaussians representing the ball 365 are located relative to (e.g., centered on) the region indicated by the location parameter(s) 453. To illustrate, the indicated region can be smaller than an aggregate size of the gaussians that represent the ball 365. In some cases, the location parameter(s) 453 indicate a spatial orientation of the ball 365 (e.g., the particularQUALCOMM Ref. No. 2507839WO- 76 / 107 -gaussian 520 should be the closest gaussian of the ball 365 to a coordinate position in the scene of FIG. 5).
[0208] In various embodiments, different techniques are used to set the location parameter(s) (e.g., the location parameter(s) 453) of the object. In a first example, the location parameter(s) are set based on second location parameter(s) associated with a second object represented by the gaussian scene representation. To illustrate, the location parameter(s) 453 of the ball 365 of FIG. 6 are set (e.g., derived) based on location parameter(s) associated with the goal 614. For example, the location parameter(s) 453 are set so the ball 365 is positioned between a user and the goal 614 and oriented so that the particular gaussian 620 is the closest gaussian of the ball 365 to the goal 614. In a second example, the location parameter(s) are set based on a location reference associated with the object template. For example, the object template 471 of FIGs. 4A and 4C includes a location reference that is used to set the location parameter(s) 453. In a third example, the location parameter(s) are set based on a location reference included in the input data. For example, the input data 176 of FIG.4 A includes or indicates the location parameter(s) 453. In various embodiments, the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof. For example, the location parameter(s) 453 of FIG. 4A are based on input received from one or more of the input source(s) 410.
[0209] In some embodiments, the method 700 also includes outputting the view of the scene based on the modified gaussian scene representation. For example, the gaussian object editor 170, the Tenderer 158 of FIG. 1 outputs an updated view of the scene 132 based on the modified gaussian scene representation 154’.
[0210] In some embodiments, the method 700 also includes rendering one or more candidate views of the scene subsequent to addition of the gaussians to the gaussian scene representation and obtaining one or more error metrics based on the one or more candidate views. For example, the image 351’ of FIG. 5 including pixels that represent the ball 365 is rendered and one or more error metrics of the image 351’ are obtained. In some cases, the one or more error metrics indicate errors. Based on the one or more error metrics, error correction can be performed (e.g., by the error correct circuit 455 of FIG. 4A). For example, one or more error metrics indicate a collision between the ballQUALCOMM Ref. No. 2507839WO- 77 / 107 - 365 of FIG. 6 and the goal 614. To illustrate, as used herein, a “collision"’ between a first object and a second object refers to at least a portion of a first region defined by boundaries corresponding to the first object (e.g., the ball 365) and at least a portion of a second region defined by boundaries corresponding to the second object (e.g., the goal 614) occupying the same space at the same time within a view of a scene represented by a gaussian scene representation. In some cases, the error correction corrects a position of the object, a spatial orientation of the object, a size of the object, or any combination thereof. In the example, error correction is performed to correct a position of the ball 365, a spatial orientation of the ball 365, a size of the ball 365, or both. In some examples, “‘adjusting” a position, a spatial orientation, a size, or any combination thereof, of a first object responsive to detecting a collision between the first object and a second object includes correcting, modifying, updating, or any combination thereof, the position of the object, the spatial orientation of the object, the size of the object, or any combination thereof, such that no portion of a first region defined by boundaries corresponding to the first object occupies the same space at the same time as a second region defined by boundaries corresponding to the second object in the scene represented by the gaussian scene representation. To illustrate, in the example of the collision between the ball 365 and the goal 614, the error correction includes correcting the position of the ball 365, a spatial orientation of the ball 365 (e.g., in cases where the ball 365 is not perfectly spherical), the size of the ball 365, or any combination thereof, such that no portion of a region corresponding to the ball 365 occupies the same space at the same time as a region corresponding to another object (e.g., the goal 614 as well as any other obj ect) in the scene represented by the gaussian scene representation.
[0211] A technical advantage of the method 700 includes adding gaussians representing an object to a gaussian scene representation based on, in some cases, less input data, as compared to conventional systems. The gaussian object editor 170 can receive input data that includes a request to add an object to the gaussian scene representation and. when information such as gaussians to use to represent the obj ect, a scale of the obj ect within the scene, a location of the object within the scene, or any combination thereof, is not provided in the request, obtain (e.g., retrieve or generate) the information.Additionally, in some scenes, a technical advantage of the method 700 includes aQUALCOMM Ref. No. 2507839WO- 78 / 107 -reduction of collisions between objects when adding an object to a view of a scene represented by a gaussian scene representation.
[0212] The method 700 of FIG. 7 may be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a controller, another hardware device, firmware device, or any combination thereof. As an example, the method 700 of FIG. 7 may be performed by a processor that executes instructions, such as described with reference to FIG. 17.
[0213] FIG. 8 depicts an implementation 800 of the device 102 of FIG. 1 as an integrated circuit 802 that includes the one or more processors 130. The integrated circuit 802 also includes one or more input connectors 806 (e.g., pins, contacts, leads, solder balls, etc.) to enable input data 804 to be received for processing. For example, the input data 804 can correspond to, include, or be included within, the image data 108, the sensor data 112, the data 116, the reference images 124, the mapping data 122, the gaussian scene representation 154, the input data 176, the template(s) 174, or other data described with reference to FIG. 1; the reference pixel data or other data described with reference to FIG. 2; the sensor data 303. the sensor data 305, the input 307. the ROI data 313, the strategy selection data 320, the mapping data 122, the gaussians 156, the ground-truth image 349, the updated gaussians 390, or other data described with reference to FIGs. 3A-3C; or a combination thereof.
[0214] The integrated circuit 802 also includes one or more output connectors 808 (e.g., pins, contacts, leads, solder balls, etc.) to enable output data 810 to be sent to another device or component. For example, the output data 810 can correspond to, include, or be included within, the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the modified gaussian scene representation 154’ or the modified gaussians 156’ of the modified gaussian scene representation 154’, the mapping data 122, the view 136, another output of the selective model update pipeline 140, the template(s) 174. or other data described with reference to FIG. 1; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154 or other data described with reference to FIG. 2; the gaussian scene representation 154 or gaussians 156 of the gaussian scene representation 154, the selection indication 321, theQUALCOMM Ref. No. 2507839WO- 79 / 107 -pixel data 341, the gradient values 343, the updated parameter values 347, or other data described with reference to FIGs. 3A-3C; the object identifier(s) 424. the location reference 426, the parameters 434, the updated gaussians 390, or other data described with reference to FIGs. 4A-4C, or a combination thereof.
[0215] The processor(s) 130 of the integrated circuit 802 are configured to initiate, perform, or control various operations described with reference to FIGs. 1-7. For example, in FIG. 8, the processor(s) 130 of the integrated circuit 802 include the gaussian object editor 170 (or components thereol) and the gaussian scene representation 154 (or components thereof, e.g., a subset of the gaussians 156 of FIG.1). Optionally, the processor(s) 130 can also include the renderer(s) 158, the view selector 162, the selective model update pipeline 140, or any combination thereof.
[0216] The integrated circuit 802 enables implementation of insertion of an object within a gaussian scene representation as a component in a system, such as a mobile phone or tablet as described with reference to FIG. 9; a mixed reality or augmented reality glasses device as described with reference to FIG. 10; a wearable electronic device as described with reference to FIG. 11; a voice-controlled speaker system described with reference to FIG. 12; a camera described with reference to FIG. 13; a virtual reality, mixed reality, or augmented reality headset described with reference to FIG. 14; or a vehicle described with reference to FIG. 15 or FIG. 16. In some embodiments, the gaussian object editor 170 of FIG. 8 can be a distinct circuit (e g., a dedicated circuit) within the processor(s) 130 that is configured to perform some or all of the various operations described with reference to FIGs. 8-16, in other embodiments, the gaussian object editor 170 can correspond to one or more other portions (e.g., general-purpose circuitry ) of the processor(s) 130 that can perform some or all of the various operations described with reference to FIGs. 8-16.
[0217] FIG. 9 depicts an implementation 900 in which the device 102 includes a mobile device 902, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 902 includes one or more microphones 910, one or more speakers 906, one or more cameras 908. and a display screen 904. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the renderer(s) 158, the view selector 162, the selectiveQUALCOMM Ref. No. 2507839WO- 80 / 107 -model update pipeline 140, or a combination thereof, are integrated in the mobile device 902 and are illustrated using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 902.
[0218] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the camera(s) 908, the one or more microphones 910, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.
[0219] The mobile device 902 can use the gaussian scene representation to generate one or more views, which can be displayed to a user via the display screen. For example, the view(s) can be used to assist the user with creation of the scene, navigation through the scene, identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the mobile device 902 and subsequently sent to another device for rendering to generate one or more views.
[0220] FIG. 10 depicts an implementation 1000 in which the device 102 includes a portable electronic device that corresponds to augmented reality or mixed reality glasses 1002. The glasses 1002 include a holographic projection unit 1012 configured to project visual data onto a surface of a lens 1014 or to reflect the visual data off of a surface of the lens 1014 and onto the wearer’s retina. The glasses 1002 also include one or moreQUALCOMM Ref. No. 2507839WO- 81 / 107 -cameras 1008, and optionally one or more microphones 1010, one or more speakers 1006. or both. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline, the Tenderer 158, the view selector 162, or a combination thereof, are integrated in the glasses 1002.
[0221] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1008, the one or more microphones 1010, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154'
[0222] The glasses 1002 can use the gaussian scene representation to generate one or more views, which can be projected by the holographic projection unit 1012 onto the lens 1014. For example, the view can be used to assist the user with creation of the scene, navigation through the scene, identifying objects or people within the scene, etc. In some cases, the gaussian scene representation can be updated at the glasses 1002 and subsequently sent to another device for rendering to generate one or more views.
[0223] FIG. 11 depicts an implementation 1100 in which the device 102 includes a wearable electronic device 1102, illustrated as a wearable camera device (sometimesQUALCOMM Ref. No. 2507839WO- 82 / 107 -referred to as a “body cam”). A front 1122 and a back 1120 of the wearable electronic device 1102 are shown in FIG. 11.
[0224] In the example illustrated, the wearable electronic device 1102 includes one or more microphones 1110, one or more speakers 1106, and one or more cameras 1108. The wearable electronic device 1102 may also include a body 1124 with an integrated attachment mechanism, such as clip 1126. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline 140, the Tenderer 158, the view selector 162. or a combination thereof, are integrated in the wearable electronic device 1102.
[0225] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1108, the one or more microphones 1110, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.
[0226] The wearable electronic device 1102 can use the gaussian scene representation to create the scene and facilitate user interaction with the scene. For example, the wearable electronic device 1102 can receive verbal input (e g., instructions or information about the scene) from the user via the microphone(s) 1110. In some cases,QUALCOMM Ref. No. 2507839WO- 83 / 107 -the gaussian scene representation can be updated at the wearable electronic device 1102 and subsequently sent to another device for rendering to generate one or more views.
[0227] FIG. 12 is an implementation 1200 in which the device 102 includes a wireless speaker and voice activated device 1202. The wireless speaker and voice activated device 1202 can have wireless network connectivity and is configured to execute an assistant operation. In the example illustrated, the wireless speaker and voice activated device 1202 includes one or more microphones 1210, one or more speakers 1206, one or more cameras 1208, and a display 1204. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline 140, the Tenderer 158, the view selector 162, or a combination thereof, are integrated in the wireless speaker and voice activated device 1202.
[0228] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1208, the one or more microphones 1210, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.
[0229] In a particular aspect, the wireless speaker and voice activated device 1202 can provide assistant operations based at least in part on the gaussian scene representation.QUALCOMM Ref. No. 2507839WO- 84 / 107 - For example, the wireless speaker and voice activated device 1202 can respond to verbal queries or requests from the user based on the gaussian scene representation, such as by adding requested objects to the scene or providing directions to an object or person in the scene. In some cases, the gaussian scene representation can be updated at the wireless speaker and voice activated device 1202 and subsequently sent to another device for rendering to generate one or more views.
[0230] FIG. 13 depicts an implementation 1300 in which the device 102 includes a portable electronic device that corresponds to a camera device 1302. In the example illustrated, the camera device 1302 includes one or more microphones 1310, one or more speakers 1306, and one or more cameras 1308. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline 140, the Tenderer 158. the view selector 162, or a combination thereof, are integrated in the camera device 1302.
[0231] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1308, the one or more microphones 1310, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.QUALCOMM Ref. No. 2507839WO- 85 / 107 -
[0232] The camera device 1302 can use the gaussian scene representation to facilitate creation of the scene, user interaction with the scene, documentation of the scene (e.g.. by object or location tagging), etc. In some cases, the gaussian scene representation can be updated at the camera device 1302 and subsequently sent to another device for rendering to generate one or more views.
[0233] FIG. 14 depicts an implementation 1400 in which the device 102 includes a portable electronic device that corresponds to a virtual reality, mixed reality, or augmented reality headset 1402. In the example illustrated, the headset 1402 includes one or more microphones 1410, one or more speakers 1406, and one or more cameras 1408. The headset 1402 also includes a display 1404 positioned to be visible to a user when the headset 1402 is worn by the user. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline 140, the Tenderer 158, the view selector 162. or a combination thereof, are integrated in the headset 1402.
[0234] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1408, the one or more microphones 1410, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.QUALCOMM Ref. No. 2507839WO- 86 / 107 -
[0235] In some cases, the headset 1402 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to generate a virtual reality or augmented reality view for a user. In some cases, the gaussian scene representation can be updated at the headset 1402 and subsequently sent to another device for rendering to generate one or more views.
[0236] FIG. 15 depicts an implementation 1500 in which the device 102 corresponds to, or is integrated within, a vehicle 1502, illustrated as a manned or unmanned aerial device (e g., a package delivery' drone). In the example illustrated, the vehicle 1502 includes one or more microphones 1510, one or more speakers 1506, and one or more cameras 1508. Components of the integrated circuit 802, including the processor(s) 130, the gaussian object editor 170, the gaussian scene representation 154 and optionally the selective model update pipeline 140, the Tenderer 158, the view selector 162, or a combination thereof, are integrated in the vehicle 1502.
[0237] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1508, the one or more microphones 1510, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.QUALCOMM Ref. No. 2507839WO- 87 / 107 -
[0238] In some cases, the vehicle 1502 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1502 and subsequently sent to another device for rendering to generate one or more views.
[0239] FIG. 16 depicts another implementation 1600 in which the device 102 corresponds to, or is integrated within, a vehicle 1602, illustrated as a car. In the example illustrated, the vehicle 1602 includes one or more displays 1604, one or more microphones 1608, one or more speakers 1606, and one or more cameras 1610. For example, the cameras 1610 can include one or more interior cameras 1610A (which can be positioned to capture interior or exterior images), one or more front cameras 1610B, one or more rear cameras 1610C, one or more side cameras 1610D and 1610E, or a combination thereof. Components of the integrated circuit 802, including the processor(s) 130. the gaussian object editor 170. the selective model update pipeline 140, the gaussian scene representation 154 and optionally the Tenderer 158 and the view selector 162, are integrated in the vehicle 1602.
[0240] As a particular example, the integrated circuit 802 can include the gaussian scene representation 154 and the gaussian object editor 170. In this example, the one or more cameras 1610, the one or more microphones 1608, or both, can be configured to obtain input data that includes an instruction to modify the gaussian scene representation 154 to add a first object in a view of the scene. In this example, the gaussian object editor 170 can be configured to identify, based on the instruction, an object template associated with the first object. To illustrate, the gaussian object editor 170 can identify the object template 471 of FIGs. 4A and 4C from the template library 172, can generate an object template, or any combination thereof. In this example, the gaussian object editor 170 can obtain, based on the object template, first gaussians that represent the object. For example, the gaussian object editor 170 can obtain attributes of the gaussian(s) 480 of FIG. 4C from the object template 471. In this example, the gaussian object editor 170 can generate the updated gaussian(s) 390 based on the gaussian attributes and add the updated gaussian(s) 390 to the gaussian scene representation to produce a modified gaussian scene representation. For example, the gaussian object editor 170 can add the updated gaussian(s) 390 to the gaussian scene representation 154 to produce the modified gaussian scene representation 154’.QUALCOMM Ref. No. 2507839WO- 88 / 107 -
[0241] In some cases, the vehicle 1602 can use the gaussian scene representation (e.g., the updated gaussian scene representation) to facilitate navigation or obstacle avoidance. In some cases, the gaussian scene representation can be updated at the vehicle 1602 and subsequently sent to another device for rendering to generate one or more views.
[0242] Referring to FIG. 17. a block diagram of a particular illustrative implementation of a device 1700 is depicted. In various implementations, the device 1700 may have more or fewer components than illustrated in FIG. 17. In an illustrative implementation, the device 1700 may correspond to the device 102. In an illustrative implementation, the device 1700 may perform one or more operations described with reference to FIGs. 1-16.
[0243] In a particular implementation, the device 1700 includes a processor 1706 (e.g., a CPU). The device 1700 may include one or more additional processors 1710 (e.g., one or more DSPs). In a particular aspect, the processor(s) 130 of FIG. 1 corresponds to the processor 1706, the processors 1710, or a combination thereof. The processors 1710 may include a speech and music coder-decoder (CODEC) 1708 that includes a voice coder (‘‘vocoder’) encoder 1736, a vocoder decoder 1738, or both. The processors 1710 include the gaussian object editor 170. the gaussian scene representation 154. or a combination thereof.
[0244] The device 1700 may include a memory 1786 and a CODEC 1734. The memory 1786 may include instructions 1756. that are executable by the one or more additional processors 1710 (or the processor 1706) to implement the functionality described with reference to the selective model update pipeline 140, the gaussian scene representation 154, or both. In a particular aspect, the memory 1786 corresponds to the memory 120 of FIG. 1. In some embodiments, the memory 1786 includes the template library’ 172 of FIG. 1. The device 1700 may include a modem 1770 coupled, via a transceiver 1750, to an antenna 1752. In a particular aspect, the modem 1770 corresponds to the modem 114 of FIG. 1.
[0245] The device 1700 may include a display 1728 coupled to a display controller 1726. One or more speakers 1792 and one or more microphones 1794 may be coupled to the CODEC 1734. The CODEC 1734 may include a digital-to-analog converter (DAC) 1702, an analog-to-digital converter (ADC) 1704, or both. In a particularQUALCOMM Ref. No. 2507839WO- 89 / 107 -implementation, the CODEC 1734 may receive analog signals from the microphone(s) 1794. convert the analog signals to digital signals using the analog-to-digital converter 1704, and provide the digital signals to the speech and music codec 1708. In a particular implementation, the speech and music codec 1708 may provide digital signals to the CODEC 1734. The CODEC 1734 may convert the digital signals to analog signals using the digital-to-analog converter 1702 and may provide the analog signals to the one or more speakers 1792.
[0246] In a particular implementation, the device 1700 may be included in a system-in-package or system-on-chip device 1722. In a particular implementation, the memory 1786, the processor 1706, the processors 1710, the display controller 1726, the CODEC 1734, and the modem 1770 are included in the system-in-package or system-on-chip device 1722. In a particular implementation, an input device 1730 (e.g., one or more cameras) and a power supply 1744 are coupled to the system-in-package or the system-on-chip device 1722. Moreover, in a particular implementation, as illustrated in FIG. 17. the display 1728, the input device 1730, the one or more speakers 1792, the microphone(s) 1794, the antenna 1752, and the power supply 1744 are external to the system-in-package or the system-on-chip device 1722. In a particular implementation, each of the display 1728. the input device 1730, the one or more speakers 1792, the microphone(s) 1794, the antenna 1752, and the power supply 1744 may be coupled to a component of the system-in-package or the system-on-chip device 1722, such as an interface, or a controller.
[0247] The device 1700 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a computing device, a communication device, an intemet-of-things (loT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.QUALCOMM Ref. No. 2507839WO- 90 / 107 -
[0248] In conjunction with the described implementations, an apparatus includes means for obtaining input data that includes an instruction to modify a gaussian scene representation to add a first object in a view of a scene. For example, the means for obtaining input data can correspond to the device 102, the memory 120, the buffer(s) 118, the image source(s) 106, the modem 114. the processor(s) 130, the data analyzer 142, the selective model update pipeline 140, the controller 160, the gaussian object editor 170, the sensor(s) 110, the sensor(s) 302, the sensor controller 304, the image subspace selector 312, the instruction parser 420, the gaussian object insertion circuit 450, the integrated circuit 802, the input connector(s) 806, the camera(s) 908, the microphone(s) 910, the camera(s) 1008, the microphone(s) 1010, the camera(s) 1108, the microphone(s) 1110, the camera(s) 1208, the microphone(s) 1210, the camera(s) 1308, the microphone(s) 1310, the camera(s) 1408, the microphone(s) 1410, the camera(s) 1508, the microphone(s) 1510, the camera(s) 1610, the microphone(s) 1608, the input device 1730, the processor 1706, the processor(s) 1710, the modem 1770. one or more other circuits or components configured to obtain input data, or any combination thereof.
[0249] The apparatus also includes means for identifying, based on the instruction, an object template associated with the first object. For example, the means for identifying an object template can correspond to the device 102, the processor(s) 130, the gaussian object editor 170, the gaussian object insertion circuit 450, the integrated circuit 802, the processor 1706, the processor(s) 1710, one or more other circuits or components configured to identify an object template, or any combination thereof.
[0250] The apparatus also includes means for obtaining, based on the object template, first gaussians that represent the first object. For example, the means for obtaining the first gaussians can correspond to the device 102, the processor(s) 130, the gaussian object editor 170, the gaussian object insertion circuit 450, the integrated circuit 802, the processor 1706, the processor(s) 1710, one or more other circuits or components configured to obtain gaussians based on an object template, or any combination thereof.
[0251] The apparatus also includes means for adding the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation. For example, the means for adding the first gaussians can correspond to the device 102, theQUALCOMM Ref. No. 2507839WO- 91 / 107 -processor(s) 130. the gaussian object editor 170. the gaussian object insertion circuit 450, the integrated circuit 802. the processor 1706, the processor(s) 1710, one or more other circuits or components configured to add gaussians to a gaussian scene representation, or any combination thereof.
[0252] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 1786) stores instructions (e g., the instructions 1756) that, when executed by one or more processors (e.g., the one or more processors 1710 or the processor 1706), cause the one or more processors to obtain input data that includes an instruction to modify a gaussian scene representation to add a first object in a view of a scene; identify, based on the instruction, an object template associated with the first object; obtain, based on the object template, first gaussians that represent the first object; and add the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0253] Particular aspects of the disclosure are described below in sets of interrelated Examples:
[0254] According to Example 1, a device includes a memory configured to store a gaussian scene representation representing a scene; and one or more processors coupled to the memory and configured to obtain input data that includes a request to modify the gaussian scene representation to add a first object in a view of the scene; identify, based on the request, an object template associated with the first object; obtain, based on the object template, first gaussians that represent the first object; and add the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0255] Example 2 includes the device of Example 1, wherein the one or more processors are configured to output the view of the scene based on the modified gaussian scene representation.
[0256] Example 3 includes the device of Example 1 or Example 2, wherein, to obtain the first gaussians, the one or more processors are configured to retrieve the object template from a gaussian object template library.QUALCOMM Ref. No. 2507839WO- 92 / 107 -
[0257] Example 4 includes the device of any of Examples 1 to 3, wherein, to identify the object template, the one or more processors are configured to generate the object template.
[0258] Example 5 includes the device of Example 4, wherein generation of the object template is based on a determination that a gaussian object template library lacks a stored object template corresponding to the first object.
[0259] Example 6 includes the device of any of Examples 1 to 5, wherein, to add the first gaussians, the one or more processors are configured to size the first object based on first scale parameters that indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0260] Example 7 includes the device of Example 6, wherein the one or more processors are configured to set the first scale parameters based on second scale parameters associated with a second object represented by the gaussian scene representation.
[0261] Example 8 includes the device of any of Examples 5 to 7, wherein the one or more processors are configured to set the first scale parameters based on location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0262] Example 9 includes the device of Example 5, wherein the one or more processors are configured to set the first scale parameters based on a scale reference associated with the object template.
[0263] Example 10 includes the device of Example 5, wherein the one or more processors are configured to set the first scale parameters based on a scale reference included in the input data.
[0264] Example 11 includes the device of any of Examples 1 to 10, wherein, to add the first gaussians, the one or more processors are configured to position and orient the first object based on first location parameters.QUALCOMM Ref. No. 2507839WO- 93 / 107 -
[0265] Example 12 includes the device of Example 11, wherein the first location parameters indicate a position of a center of the first object, and wherein the first gaussians are positioned around the center of the first object.
[0266] Example 13 includes the device of Example 11 or Example 12, wherein the first location parameters indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0267] Example 14 includes the device of any of Examples 11 to 13, wherein the one or more processors are configured to set the first location parameters based on second location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0268] Example 15 includes the device of any of Examples 11 to 14, wherein the one or more processors are configured to set the first location parameters based on a location reference associated with the object template.
[0269] Example 16 includes the device of any of Examples 11 to 14, wherein the one or more processors are configured to set the first location parameters based on a location reference included in the input data.
[0270] Example 17 includes the device of Example 16, wherein the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
[0271] Example 18 includes the device of any of Examples 1 to 17. wherein the one or more processors are configured to render one or more candidate views of the scene subsequent to addition of the first gaussians to the gaussian scene representation; and obtain one or more error metrics based on the one or more candidate views.
[0272] Example 19 includes the device of Example 18, wherein the one or more error metrics indicate overlapped objects in the one or more candidate views.
[0273] Example 20 includes the device of Example 18 or Example 19, wherein the one or more processors are configured to perform error correction based on the one or more error metrics.QUALCOMM Ref. No. 2507839WO- 94 / 107 -
[0274] Example 21 includes the device of Example 20, wherein the error correction is configured to adjust a position of the first object, a spatial orientation of the first object, a size of the first object, or both.
[0275] Example 22 includes the device of any of Examples 1 to 21 and further includes a modem coupled to the one or more processors, wherein the modem is configured to receive the input data, the gaussian scene representation, or both.
[0276] Example 23 includes the device of any of Examples 1 to 22, wherein the one or more processors are integrated into a circuit configured to modify gaussian scene representations.
[0277] Example 24 includes the device of any of Examples 1 to 23, wherein the one or more processors are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.
[0278] According to Example 25, a method includes obtaining input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene; identifying, based on the request, an object template associated with the first object; obtaining, based on the object template, first gaussians that represent the first object; and adding the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0279] Example 26 includes the method of Example 25, further comprising outputting the view of the scene based on the modified gaussian scene representation.
[0280] Example 27 includes the method of Example 25 or Example 26, wherein obtaining the first gaussians comprises retrieving the object template from a gaussian object template library.
[0281] Example 28 includes the method of any of Examples 25 to 27, wherein identifying the object template comprises generating the object template.QUALCOMM Ref. No. 2507839WO- 95 / 107 -
[0282] Example 29 includes the method of Example 28, wherein generating the object template is based on a determination that a gaussian object template library lacks a stored object template corresponding to the first object.
[0283] Example 30 includes the method of any of Examples 25 to 29, wherein adding the first gaussians comprises sizing the first object based on first scale parameters that indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0284] Example 31 includes the method of Example 30 and further includes setting the first scale parameters based on second scale parameters associated with a second object represented by the gaussian scene representation.
[0285] Example 32 includes the method of any of Examples 29 to 31 and further includes setting the first scale parameters based on location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0286] Example 33 includes the method of Example 29 and further includes setting the first scale parameters based on a scale reference associated with the object template.
[0287] Example 34 includes the method of Example 29 and further includes setting the first scale parameters based on a scale reference included in the input data.
[0288] Example 35 includes the method of any of Examples 25 to 34, wherein adding the first gaussians comprises positioning and orienting the first object based on first location parameters.
[0289] Example 36 includes the method of Example 35, wherein the first location parameters indicate a position of a center of the first object, and wherein the first gaussians are positioned around the center of the first object.
[0290] Example 37 includes the method of any of Example 35 or Example 36, wherein the first location parameters indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.QUALCOMM Ref. No. 2507839WO- 96 / 107 -
[0291] Example 38 includes the method of any of Examples 35 to 37 and further includes setting the first location parameters based on second location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0292] Example 39 includes the method of any of Examples 35 to 38 and further includes setting the first location parameters based on a location reference associated with the object template.
[0293] Example 40 includes the method of any of Examples 35 to 38 and further includes setting the first location parameters based on a location reference included in the input data.
[0294] Example 41 includes the method of Example 40, wherein the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
[0295] Example 42 includes the method of any of Examples 25 to 41, further includes rendering one or more candidate views of the scene subsequent to addition of the first gaussians to the gaussian scene representation; and obtaining one or more error metrics based on the one or more candidate views.
[0296] Example 43 includes the method of Example 42, wherein the one or more error metrics indicate overlapped objects in the one or more candidate views.
[0297] Example 44 includes the method of Examples 42 or Example 43 and further includes performing error correction based on the one or more error metrics.
[0298] Example 45 includes the method of Example 44, wherein the error correction adjusts a position of the first object, a spatial orientation of the first object, a size of the first object, or any combination thereof.
[0299] According to Example 46, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene; identify, based on the request,QUALCOMM Ref. No. 2507839WO- 97 / 107 -an object template associated with the first object; obtain, based on the object template, first gaussians that represent the first object; and add the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0300] Example 47 includes the non-transitory computer-readable medium of Example 46, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output the view of the scene based on the modified gaussian scene representation.
[0301] Example 48 includes the non-transitory computer-readable medium of Example 46 or Example 47, wherein, to obtain the first gaussians, the one or more processors are configured to retrieve the object template from a gaussian object template library.
[0302] Example 49 includes the non-transitory computer-readable medium of any of Examples 46 to 48, wherein the instructions, when executed by the one or more processors, cause the one or more processors to, to identify the object template, generate the object template.
[0303] Example 50 includes the non-transitory computer-readable medium of Example 49, wherein generation of the object template is based on a determination that a gaussian object template library lacks a stored object template corresponding to the first object.
[0304] Example 51 includes the non-transitory computer-readable medium of any of Examples 46 to 50, wherein, to add the first gaussians, the one or more processors are configured to size the first object based on first scale parameters that indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0305] Example 52 includes the non-transitory' computer-readable medium of Example 51, wherein the instructions, when executed by the one or more processors, cause the one or more processors to set the first scale parameters based on second scale parameters associated with a second object represented by the gaussian scene representation.
[0306] Example 53 includes the non-transitory computer-readable medium of any of Examples 50 to 52, wherein the instructions, when executed by the one or moreQUALCOMM Ref. No. 2507839WO- 98 / 107 -processors, cause the one or more processors to set the first scale parameters based on location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0307] Example 54 includes the non-transitory computer-readable medium of Example 50, wherein the instructions, when executed by the one or more processors, cause the one or more processors to set the first scale parameters based on a scale reference associated with the object template.
[0308] Example 55 includes the non-transitory computer-readable medium of Example 50, wherein the instructions, when executed by the one or more processors, cause the one or more processors to set the first scale parameters based on a scale reference included in the input data.
[0309] Example 56 includes the non-transitory computer-readable medium of any of Examples 46 to 55, wherein, to add the first gaussians, the one or more processors are configured to position and orient the first object based on first location parameters.
[0310] Example 57 includes the non-transitory computer-readable medium of Example 56, wherein the first location parameters indicate a position of a center of the first object, and wherein the first gaussians are positioned around the center of the first obj ect.
[0311] Example 58 includes the non-transitory computer-readable medium of Example 56 or Example 57, wherein the first location parameters indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0312] Example 59 includes the non-transitory' computer-readable medium of any of Examples 56 to 58, wherein the instructions, when executed by the one or more processors, cause the one or more processors to set the first location parameters based on second location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0313] Example 60 includes the non-transitory computer-readable medium of any of Examples 56 to 59, wherein the instructions, when executed by the one or moreQUALCOMM Ref. No. 2507839WO- 99 / 107 -processors, cause the one or more processors to set the first location parameters based on a location reference associated with the object template.
[0314] Example 61 includes the non-transitory computer-readable medium of any of Examples 56 to 59, wherein the instructions, when executed by the one or more processors, cause the one or more processors to set the first location parameters based on a location reference included in the input data.
[0315] Example 62 includes the non-transitory' computer-readable medium of Example 61, wherein the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
[0316] Example 63 includes the non-transitory' computer-readable medium of any of Examples 46 to 62, wherein the instructions, when executed by the one or more processors, cause the one or more processors to render one or more candidate views of the scene subsequent to addition of the first gaussians to the gaussian scene representation; and obtain one or more error metrics based on the one or more candidate views.
[0317] Example 64 includes the non-transitory computer-readable medium of Example 63, wherein the one or more error metrics indicate overlapped objects in the one or more candidate views.
[0318] Example 65 includes the non-transitory computer-readable medium of Example 63 or Example 64, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform error correction based on the one or more error metrics.
[0319] Example 66 includes the non-transitory' computer-readable medium of Example 65, wherein the error correction is configured to adjust a position of the first object, a spatial orientation of the first object, a size of the first object, or any combination thereof.
[0320] According to Example 67, an apparatus includes means for obtaining input data that includes a request to modify a gaussian scene representation to add a first object in a view of a scene; means for identifying, based on the request, an object templateQUALCOMM Ref. No. 2507839WO- 100 / 107 -associated with the first object; means for obtaining, based on the object template, first gaussians that represent the first object; and means for adding the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
[0321] Example 68 includes the apparatus of Example 67, further comprising means for outputting the view of the scene based on the modified gaussian scene representation.
[0322] Example 69 includes the apparatus of Example 67 or Example 68, wherein the means for obtaining the first gaussians includes means for retrieving the object template from a gaussian object template library.
[0323] Example 70 includes the apparatus of any of Examples 67 to 70, wherein the means for identifying the object template includes means for generating the object template.
[0324] Example 71 includes the apparatus of Example 70, wherein generation of the object template is based on a determination that a gaussian object template library- lacks a stored object template corresponding to the first object.
[0325] Example 72 includes the apparatus of any of Examples 67 to 71, wherein, the means for adding the first gaussians is configured to size the first object based on first scale parameters that indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0326] Example 73 includes the apparatus of Example 72 and further includes means for setting the first scale parameters based on second scale parameters associated with a second object represented by the gaussian scene representation.
[0327] Example 74 includes the apparatus of any of Examples 71 to 73 and further includes means for setting the first scale parameters based on location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0328] Example 75 includes the apparatus of Example 71 and further includes means for setting the first scale parameters based on a scale reference associated with the object template.QUALCOMM Ref. No. 2507839WO- 101 / 107 -
[0329] Example 76 includes the apparatus of Example 71 and further includes means for setting the first scale parameters based on a scale reference included in the input data.
[0330] Example 77 includes the apparatus of any of Examples 67 to 76, wherein the means for adding the first gaussians is configured to position and orient the first object based on first location parameters.
[0331] Example 78 includes the apparatus of Example 77, wherein the first location parameters indicate a position of a center of the first object, and wherein the first gaussians are positioned around the center of the first object.
[0332] Example 79 includes the apparatus of Example 77 or Example 78, wherein the first location parameters indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
[0333] Example 80 includes the apparatus of any of Examples 77 to 79 and further includes means for setting the first location parameters based on second location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
[0334] Example 81 includes the apparatus of any of Examples 77 to 80 and further includes means for setting the first location parameters based on a location reference associated with the object template.
[0335] Example 82 includes the apparatus of any of Examples 77 to 80 and further includes means for setting the first location parameters based on a location reference included in the input data.
[0336] Example 83 includes the apparatus of Example 82, wherein the location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
[0337] Example 84 includes the apparatus of any of Examples 67 to 83, further includes means for rendering one or more candidate views of the scene subsequent to addition ofQUALCOMM Ref. No. 2507839WO- 102 / 107 -the first gaussians to the gaussian scene representation; and means for obtaining one or more error metrics based on the one or more candidate views.
[0338] Example 85 includes the apparatus of Example 84, wherein the one or more error metrics indicate overlapped objects in the one or more candidate views.
[0339] Example 86 includes the apparatus of Example 84 or Example 85 and further includes means for performing error correction based on the one or more error metrics.
[0340] Example 87 includes the apparatus of Example 86, wherein the means for performing error correction includes means for adjusting a position of the first object, a spatial orientation of the first object, a size of the first object, or any combination thereof.
[0341] Example 88 includes the apparatus of any of Examples 67 to 87 and further includes a modem coupled to the means for obtaining input data, wherein the modem is configured to receive the input data, the gaussian scene representation, or both.
[0342] Example 89 includes the apparatus of any of Examples 67 to 88, wherein the means for obtaining input data, the means for identifying the obj ect template, the means for obtaining the first gaussians, the means for adding the first gaussians, or any combination thereof, are integrated into a circuit configured to modify gaussian scene representations.
[0343] Example 90 includes the apparatus of any of Examples 67 to 89, wherein the means for obtaining input data, the means for identifying the object template, the means for obtaining the first gaussians, the means for adding the first gaussians, or any combination thereof, are integrated in at least one of a mobile phone, a computer device, a wearable electronic device, a camera device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a vehicle.
[0344] Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have beenQUALCOMM Ref. No. 2507839WO- 103 / 107 -described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.
[0345] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.
[0346] The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.
Claims
1. QUALCOMM Ref. No. 2507839WO- 104 / 107 - WHAT IS CLAIMED IS:
1. A device comprising:a memory' configured to store a gaussian scene representation representing a scene; andone or more processors coupled to the memory' and configured to:obtain input data that includes a request to modify the gaussian scene representation to add a first object in a view of the scene; identify, based on the request, an object template associated with the first object;obtain, based on the object template, first gaussians that represent the first object; andadd the first gaussians to the gaussian scene representation to produce a modified gaussian scene representation.
2. The device of claim 1, wherein the one or more processors are configured to output the view of the scene based on the modified gaussian scene representation.
3. The device of claim 2, further comprising a display device coupled to the one or more processors and configured to display the view of the scene based on the modified gaussian scene representation.
4. The device of claim 1, wherein, to obtain the first gaussians, the one or more processors are configured to retrieve the object template from a gaussian object template library.
5. The device of claim 1, wherein, to identify the object template, the one or more processors are configured to generate the obj ect template.
6. The device of claim 5, wherein generation of the object template is based on a determination that a gaussian object template library' lacks a stored object template corresponding to the first object.
7. The device of claim 1. wherein, to add the first gaussians, the one or more processors are configured to size the first object based on first scale parameters thatQUALCOMM Ref. No. 2507839WO- 105 / 107 -indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
8. The device of claim 7, wherein the one or more processors are configured to set the first scale parameters based on second scale parameters associated with a second object represented by the gaussian scene representation.
9. The device of claim 7, wherein the one or more processors are configured to set the first scale parameters based on location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
10. The device of claim 7, wherein the one or more processors are configured to set the first scale parameters based on a first scale reference associated with the object template, based on a second scale reference included in the input data, or both.
11. The device of claim 1, wherein, to add the first gaussians, the one or more processors are configured to position and orient the first object based on first location parameters.
12. The device of claim 11, wherein the first location parameters indicate a position of a center of the first object, and wherein the first gaussians are positioned around the center of the first object.
13. The device of claim 11, wherein the first location parameters indicate boundaries of a first region of the scene, and wherein the first gaussians are added within the boundaries of the first region.
14. The device of claim 11, wherein the one or more processors are configured to set the first location parameters based on second location parameters of second gaussians associated with a second object represented by the gaussian scene representation.
15. The device of claim 11, wherein the one or more processors are configured to set the first location parameters based on a first location reference associated with the object template, based on a second location reference included in the input data, or both.QUALCOMM Ref. No. 2507839WO- 106 / 107 - 16. The device of claim 15, wherein the second location reference is based on a cursor input, a text input, an eye gaze input, an audio input, or a combination thereof.
17. The device of claim 1, wherein the one or more processors are configured to: render one or more candidate views of the scene subsequent to addition of the first gaussians to the gaussian scene representation; andobtain one or more error metrics based on the one or more candidate views.
18. The device of claim 17, wherein the one or more error metrics indicate overlapped objects in the one or more candidate views.
19. The device of claim 17, wherein the one or more processors are configured to perform error correction based on the one or more error metrics, and wherein the error correction is configured to adjust a position of the first object, a spatial orientation of the first object, a size of the first object, or any combination thereof.
20. The device of claim 1, further comprising one or more cameras coupled to the one or more processors and configured to capture one or more images of the scene, wherein the gaussian scene representation are based on the one or more images.
