Gaussian media codec

Abstract

Systems and methods for reconstructing visual content. One example provides a method for reconstructing visual content comprising receiving a sequence of visual content frames; encoding the sequence of visual content frames to obtain a sequence of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-time volume; performing time slicing on the sequence of 3D Gaussians to generate factorized two-dimensional (2D) Gaussian probabilities; and generating reconstructed frames based on the factorized 2D Gaussian probabilities.

Description

D24074WO01GAUSSIAN MEDIA CODEC CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U. S. Provisional Application No. 63 / 719,997 filed November 13, 2024, U. S. Provisional Application No. 63 / 765,320, filed February 28, 2025, and European Patent Application No. 25162456.5, filed on 7 March 2025, each of which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] This application relates generally to video processing and compression, and more specifically, to systems, apparatus, and methods for representing and reconstructing time-varying visual content using three-dimensional (3D) Gaussian distributions in a space-time volume with a time slicing mechanism for frame reconstruction.BACKGROUND

[0003] Neural networks are computational models that process information through interconnected nodes, providing complex data representation and processing. Recent applications of neural networks include implicit representation of visual data. For image representation, neural networks encode pixel information in the network weights. This approach is extended to 3D scenes by representing scene geometry and appearance through a neural network that predicts color and density at any 3D point. Neural Representations for Video (NeRV) further applies this approach to video content by modeling temporal variations through neural network parameters. While these neural methods achieve high information preservation and reconstruction quality,D24074WO01 they require substantial computational resources, specifically high-end Graphics Processing Units (GPUs), and exhibit slow decoding performance that limits their practical applications.

[0004] Explicit representations, where visual content is directly encoded in mathematical primitives, have emerged as alternatives to neural network-based approaches. Three-dimensional (3D) Gaussian splatting represents 3D scenes through a set of Gaussian functions with defined spatial parameters, providing an alternative to neural radiance fields. Similar to how 3D Gaussian splatting handles 3D scenes, two-dimensional (2D) Gaussian representations have been applied to image compression. In some examples, applying the 2D Gaussian representations lead to decoding speeds of about 1000 frames per second (FPS) through direct parameter-to-pixel mapping. These approaches demonstrate different applications of explicit representations in visual content processing.SUMMARY

[0005] Embodiments of the present disclosure provide a system for efficient representation and reconstruction of time-varying visual content using 3D Gaussian distributions in a space-time volume with a time slicing mechanism for frame reconstruction.

[0006] According to embodiments of the present disclosure, a method for reconstructing visual content comprises receiving a sequence of visual content frames; encoding the sequence of visual content frames to obtain a sequence of 3D Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a spacetime volume; performing time slicing on the sequence of 3D Gaussians to generate factorized 2D Gaussian probabilities; and generating reconstructed frames based on the factorized 2D Gaussian probabilities.D24074WO01

[0007] According to embodiments of the present disclosure, a method for training a video representation system comprises receiving a sequence of video frames; generating an initial set of 3D Gaussian parameters based on the sequence of video frames; and iteratively refining the 3D Gaussian parameters by generating a reconstructed frame using the 3D Gaussian parameters; computing a loss value between the reconstructed frame and an original frame among the sequence of video frames; and updating the 3D Gaussian parameters based on the computed loss value.

[0008] According to embodiments of the present disclosure, an apparatus for video processing, comprises an electronic processor; and a memory storing computer executable instructions, wherein the computer executable instructions, when executed, cause the electronic processor to encode a sequence of visual content frames to obtain a sequence of 3D Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-time volume; perform time slicing on the sequence of 3D Gaussians to generate factorized 2D Gaussian probabilities; and generate reconstructed frames based on the factorized 2D Gaussian probabilities.

[0009] According to embodiments of the present disclosure, a method for decoding visual content comprises receiving an encoded bitstream representing the visual content, the encoded bitstream comprising a sequence of three-dimensional (3D) Gaussians in a space-time volume, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters; performing entropy decoding and dequantization of the sequence of 3D Gaussians; selecting a temporal position in the space-time volume; generating a factorized two-dimensional (2D) Gaussian probability at the selected temporal position by performing timeD24074WO01 slicing on the sequence of 3D Gaussians; and generating a reconstructed frame by applying the factorized 2D Gaussian probability to the sequence of 3D Gaussians.DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 illustrates an example of a video processing process according to aspects of the present disclosure.

[0011] FIG. 2 illustrates an example of a video processing training process according to aspects of the present disclosure.

[0012] FIG. 3 illustrates an example of a visual content reconstruction method according to aspects of the present disclosure including the video processing process illustrated in FIG. 1.

[0013] FIG. 4 illustrates an example of a decoding process according to aspects of the present disclosure.

[0014] FIG. 5 illustrates an example of a training method according to aspects of the present disclosure including the video processing training process illustrated in FIG. 2.

[0015] FIG. 6 illustrates an example of a video processing apparatus according to aspects of the present disclosure.

[0016] FIG. 7 illustrates examples of experimental results according to aspects of the present disclosure.

[0017] FIG. 8 illustrates examples of experimental results using compression and spatially varying processing methods according to aspects of the present disclosure.D24074WO01

[0018] FIG. 9 illustrates examples of experimental results using event stream compression methods with probabilistic decoding according to aspects of the present disclosure.DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure provide a system, apparatus, and method for representing and reconstructing time-varying visual content using 3D Gaussian distributions in a space-time volume with a time slicing mechanism for frame reconstruction.

[0020] Some implementations of representing and reconstructing visual content include employing explicit Gaussian representations. However, these methods have limited capacity to capture the temporal dimension effectively. While 3D Gaussian Splatting (3DGS) theoretically provides sufficient dimensionality, 3DGS’s practical implementation requires splatting Gaussians onto a camera plane by rendering the 2D projections. For static camera configurations, the additional camera dimension does not carry useful information for the representation and thus is redundant. Four-dimensional Gaussian Splatting (4DGS) extends 3DGS to dynamic 3D scenes. 4DGS addresses temporal variation but introduces substantial storage overhead.Furthermore, these approaches require depth information for rasterization, which becomes ambiguous and arbitrary in the video context. Aspects of the present disclosure describe a time slicing method that efficiently produces 2D Gaussians and leverages a rasterizer for recovering temporal visual content.

[0021] RGB (Red, Green, and Blue) video represents a form of time-varying 2D data. Some video compression approaches use international standards like Moving Picture Experts Group (MPEG) and open-source video coding formats. These approaches operate by segmenting frames into sub-blocks and tracking their motion trajectories. The resulting motion vectors and residualD24074WO01 errors undergo quantization and compression for subsequent frames. Recent machine learning approaches have enhanced various components of these compression pipelines, particularly in motion decoding. However, these learned compression methods, while improving frame quality, often face challenges in generalization.

[0022] Alternative approaches to video representation conceptualize video as a continuous function over time, utilizing neural network parameters to approximate this function. Neural Representations for Video (NeRV) is an example of such approaches, where the number of network parameters correlates with the compression bitrate, thus providing a per- video optimization process. However, while the per-video optimization process yields compact representations, it also results in implicit neural representations (INRs) because the video content becomes encoded in the neural network's parameters during the optimization, rather than being stored in an explicit format. For example, the INRs encode visual content as continuous functions implemented by neural networks, where the content is represented implicitly through the network's parameters rather than explicitly stored values. The resulting INRs demand intensive computational resources for decoding, limiting their deployment on resource-constrained devices.

[0023] Recent developments in explicit representations have transformed both 3D scene reconstruction and image processing domains. These explicit methods achieve faster decoding compared to implicit approaches. In 3D scene reconstruction, 3D Gaussian splatting matches the render quality of neural network-based 3D scene representation methods while substantially reducing rendering time. Similarly, in image processing, some Gaussian-based image representation techniques (for example, “Gaussianlmage”) demonstrate how 3DGS principles can be adapted for 2D plane representation. In some examples, Gaussianlmage optimizes a set ofD24074WO01 2D Gaussian functions to represent image content, where each Gaussian's parameters define the position, scale, and color attributes to reconstruct pixel values. The Gaussian-based image representation technique is used to achieve efficient image encoding while enabling rapid decoding. For example, when integrated with standard compression techniques, Gaussianlmage achieves extremely fast decoding speeds of 1000 FPS while maintaining competitive performance metrics compared to traditional image compression standards.

[0024] Accordingly, the present disclosure describes example systems, apparatuses, and methods for processing time- varying 2D data through representing visual content as a collection of 3D Gaussians in a space-time volume and applying a time slicing mechanism. The system processes the 3D Gaussians by factorizing the 3D Gaussians’ probabilities into spatial and temporal components, providing efficient frame reconstruction through reduced per-pixel operations. By factorizing the 3D Gaussians’ probabilities, the system reconstructs a pixel with a reduced number of operations.

[0025] The reduced per-pixel operations decrease computational complexity when the system processes visual data in real-time applications. Applications of the system may include real-time video processing where the system provides rapid decoding for streaming tasks. The system may also process event camera data, maintaining temporal precision in the captured information. Furthermore, the system may perform video processing operations such as denoising that require accurate frame reconstruction.

[0026] FIG. 1 illustrates an example of video processing process 100 according to aspects of the present disclosure. The video processing process 100 transforms video frames into a space-time Gaussian representation where static and dynamic content are efficiently encoded using differentD24074WO01 temporal scales. The video processing process 100 may be used for encoding and reconstructing time-varying visual content. The video processing process 100 may be performed by a video processing apparatus 600 illustrated in FIG. 6. Additionally, the steps provided within in FIG. 1 are merely examples, and may instead be conducted in a different order. While illustrated as being performed serially, some steps of the process 100 may instead be performed simultaneously.

[0027] The video processing process 100 involves determining a mixture of 3D Gaussians and corresponding feature vectors to represent a function (%, y, t). f is decomposed into the sum of Gaussian probabilities weighted by their feature cp. f(x, y, t) ~ i Pi(x> y, t)q. In a case of traditional video, the output of f represents color at pixel (%, y) at frame t. Each video frame is a slice of the 3D volume. Videos can be considered as a blend of static regions and dynamic regions. Static patches can be represented as Gaussians with a large scale over t, a dynamic regions are broken into piece- wise linear movement such that each component is represented as a Gaussian rotated around the %y-planc. According to the process of determining the mixture of 3D Gaussians and corresponding feature vectors to represent the function f(x,y, t), the parameter encoding process 140 in FIG. 1 is provided and will be described in detail.

[0028] The video processing process 100 further involves 3D Gaussian formulation. The basic 3D Gaussian includes position u G R2, 3D covariance matrix S G R3*3, and color c G R3.Considering that the covariance matrix is positive semi-definite, S can be decomposed into a rotation matrix R G R3and scaling matrix S E R3= RSSTRT, where R can be any rotation matrix (such as RxRyRt). A quaternion-based representation qwqxqyqtmay be chosen for fittingD24074WO01 due to its smoothness, and the 4-quaternion parameters may be converted to 3-Euler angles for storage:2(QW + Qx)2- 1 ^(dxdy QwQt) + QwQy)7? + qwqt) + Qx) 1 QwQt)2(jJxQy QwQt) ^(flxQt + QwQy) ^ " (. Qw + Qx) 1_S is a diagonal matrix with the 3 parameters corresponding to the scale along each axis:Sx0 00 Sy 00 0 stAccording to the process of the 3D Gaussian formulation, the 3D Gaussian generation process 145 in FIG. 1 is provided and will be described in detail.

[0029] The video processing process 100 further involves 3D Gaussian rasterization. The 3D Gaussian rasterization is related to 2D Gaussian rasterization. For 2D Gaussian rasterization, in the case of having 2D Gaussians to fit (%, y), a simplified rasterizer may be used, where a coordinate value can be computed as a weighted average of the probability densities of each Gaussian at coordinate (u, v) multiplied by the Gaussian’s corresponding feature: f(u, v) = Xi Pt(u, v)C. In some examples, the rasterizer is tile-based where only Gaussians with a 99% confidence interval intersecting the tile is kept speeding up rasterization.

[0030] For 3D Gaussian rasterization, compared with the 2D case, f further depends on time t: f(u, v, t) = Xi Pi(u> D t)ci- Note that Pi(u, v, t) can be factorized into a conditional probability Pi(u, v|t) and a marginal probability p,(t) at time t, resulting in= Xi Pi(u,v\t)pi(f)Ci,D24074WO01which is referred to as “Time Slicing”. By using the Time Slicing, the following benefits may be obtained. First, per-pixel computation is faster. Because covariance inversion can be reused across pixels, the original f(u, v, t) takes 13 FLOPs (12 for pt, t) and 1 for ). In the factorization, Pi(t) can also be reused (and pre-multiplied with ), resulting in that 7 FLOPs are required (6 for Pt(u, v|t) and 1 for multiplying with p,(t)c. Second, %y-bascd culling is beneficial. Because Pi(u, v|t) is a 2D Gaussian, some culling techniques, for example, those from Gaussianlmage and 3DGS, can be used to reduce the number of Gaussians summed over per-tile (and thus per-coordinate). Third, t -based is beneficial. The Gaussian can be culled before going to the tile-based rasterizer when Pi(t) is too small. Accordingly, the conditional 2D Gaussian can be derived from the multivariate Gaussian with~ Pc)XT’"* jl V"-*— ^*1:2, 1:2 ~:2

[0031] For traditional video, c is the 3-channel color vector. For event streams, c is the polarity indicator p. For binned event streams, c is the 2-vector number of positive and negative events. Accordingly, methods provided herein extends to arbitrary feature dimensions, such as hyper-spectral imagery.

[0032] Referring to FIG. 1, the video processing process 100 may start with a parameter encoding process 140. The parameter encoding process 140 receives original frame 105 as input and generates 3D Gaussian parameters 110 as output. The parameter encoding process 140 transforms the frame data into a set of parameters that define 3D Gaussian distributions, where different types of regions may be handled differently. For example, static regions are representedD24074WO01 by parameters indicating larger scales in the temporal dimension, while dynamic regions are represented by parameters indicating smaller temporal scales but more frequent spatial variations. The parameter encoding process 140 may be performed by a Gaussian parameter generator 625 included in the encoder 615 illustrated in FIG. 6.

[0033] Next, the 3D Gaussian generation process 145 receives the 3D Gaussian parameters 110 as input and generates 3D Gaussians 115 as output. During the 3D Gaussian generation process 145, the parameters are used to construct actual 3D Gaussian distributions in a space-time volume, where these Gaussians effectively capture both static content (for example, through temporally elongated Gaussians) and dynamic content (for example, through sequences of shorter-duration Gaussians) in a unified representation. The 3D Gaussian generation process 145 may be performed by a 3D Gaussian generation component 630 included in the encoder 615 illustrated in FIG. 6.

[0034] Next, the time slicing process 150 takes 3D Gaussians 115 as input and generates factorized 2D Gaussian probabilities 120 as output. The time slicing process 150 introduces the factorization approach where each 3D Gaussian's probability density is split into spatial and temporal components. In some examples, the time slicing process 150 process reduces computation from 13 to 7 FLOPs and causes efficient frame reconstruction without losing the temporal coherence benefits of the 3D representation. The time slicing process 150 may be performed by a time slicing component 635 included in the encoder 615 illustrated in FIG. 6.

[0035] Next, the 2D Gaussian extraction process 155 receives the factorized 2D Gaussian probabilities 120 as input and generates 2D Gaussians 125 as output. The 2D Gaussian extraction process 155 process extracts 2D Gaussians at specific time steps using both xy -based and t-D24074WO01 based culling techniques, where Gaussians with negligible contribution (for example, below 99% confidence interval) are excluded to optimize computation while maintaining quality. The 2D Gaussian extraction process 155 may be performed by a 2D Gaussian extraction component 640 included in the encoder 615 illustrated in FIG. 6.

[0036] Next, the frame reconstruction process 160 receives the 2D Gaussians 125 as input and generates reconstructed frame 130 as output. The frame reconstruction process 160 process may employ a rasterization technique where pixel values are computed by combining contributions from relevant 2D Gaussians. The factorized probabilities maintain the quality benefits of Gaussian blending. The frame reconstruction process 160 may be performed by a rasterization component 645 included in the encoder 615 illustrated in FIG. 6.

[0037] FIG. 2 illustrates an example of a video processing training process 200. The video processing training process 200 includes an encoder training process 210. The encoder training process 210 may be used to train the encoder 615 to implement the video processing process 100. In this example, the training process 210 operates by iteratively refining the Gaussian representation to minimize the difference between the original frame 105 sequence and the reconstructed frame 130 sequence. For example, given an original frame collection V ={Ii, I2,..., In] and reconstructed frame collection V = {, I2,..., In], the process aims to minimize LI Loss 205 between corresponding frames. For each frame's time step, the training process 210 involves generating 3D Gaussian parameters 110 for the current frame and constructing 3D Gaussians 115 based on these parameters. The training process 210 then continues with performing time slicing process 150 to obtain the factorized 2D Gaussian probabilities 120, followed by extracting 2D Gaussians 125 for the current time step.D24074WO01

[0038] After obtaining the 2D Gaussians 125, the training process 210 proceeds with reconstructing the frame through frame reconstruction process 160. The reconstructed frame is then compared with the original frame 105 to compute the LI loss 205. The LI Loss 205 is an optimization criterion measuring the differences between pixel values in the original and reconstructed frames. Based on the computed LI Loss 205, the Gaussian parameters are updated to minimize the difference between original and reconstructed frames. The training process 210 continues iteratively until the LI Loss 205 converges or reaches a satisfactory threshold.

[0039] FIG. 3 illustrates an example of a visual content reconstruction method 300 according to aspects of the present disclosure including the video processing process 100. The visual content reconstruction method 300 may be performed by the video processing apparatus 600.Additionally, the steps provided within in FIG. 3 are merely examples, and may instead be conducted in a different order. While illustrated as being performed serially, some steps of the visual content reconstruction method 300 may instead be performed simultaneously.

[0040] At operation 305, video processing apparatus 600 receives a sequence of visual content frames. The visual content frames may represent any type of time-varying visual content, such as video streams, event streams, or other sequential visual data that changes over time.

[0041] At operation 310, video processing apparatus 600 generates a space-time volume representation by encoding the sequence of visual content frames to obtain a group of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-time volume. The space-time volume provides a unified representation that captures both spatial and temporal characteristics of the visual content. In some examples, the space-time volume comprises two spatialD24074WO01 dimensions and one temporal dimension, enabling efficient representation of both static and dynamic content. For example, regions with consistent appearance across multiple frames are represented by 3D Gaussians having larger scale parameters in the temporal dimension, while dynamic regions are represented by sequences of 3D Gaussians having smaller temporal scale parameters and varying spatial parameters.

[0042] At operation 315, video processing apparatus 600 performs time slicing on the group of 3D Gaussians to generate factorized 2D Gaussian probabilities. Operation 315 may be used for efficient frame reconstruction while maintaining temporal coherence. In some examples, the time slicing operation involves factorizing a probability density of each 3D Gaussian into a conditional probability in spatial dimensions and a marginal probability in the temporal dimension. For example, the Gaussians may be culled based on temporal probability thresholds to optimize computational efficiency while maintaining visual quality.

[0043] At operation 320, video processing apparatus 600 generates reconstructed frames based on the factorized 2D Gaussian probabilities. Operation 320 transforms the mathematical representation back into viewable frames. In some examples, this involves extracting, for each time step, a set of 2D Gaussians from the factorized 2D Gaussian probabilities and computing pixel values by combining contributions from the extracted 2D Gaussians based on their respective probability densities and feature parameters.

[0044] For example, the pixel values may be computed by calculating a weighted average of probability densities of each Gaussian at a pixel coordinate multiplied by corresponding feature parameters. In some implementations, the feature parameters may comprise RGB color values for traditional video, event polarity indicators, or numbers of positive and negative events forD24074WO01 event streams. To optimize reconstruction efficiency, Gaussians may be culled based on spatial distance thresholds during the reconstruction process.

[0045] FIG. 4 illustrates an example of a decoding process 400according to aspects of the present disclosure. The decoding process 400may be performed by the video processing apparatus 600 that includes the decoder 650. The decoding process 400 comprises receiving encoded bitstream 405, entropy decoding and dequantization step 410, temporal position selection step 415, slicing step 420, frame reconstruction step 425, and progressive decoding step 430. Additionally, the steps provided within in FIG. 4 are merely examples, and may instead be conducted in a different order. While illustrated as being performed serially, some steps of the decoding process 400 may instead be performed simultaneously.

[0046] The decoding process 400 may start with receiving encoded bitstream 405. The video processing apparatus 600 receives an encoded bitstream representing the visual content, the encoded bitstream comprising a sequence of 3D Gaussians in a space-time volume, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters. The properties of the 3D Gaussian may be determined by the parameters. For example, the position parameters determine the 3D Gaussian’s location, the scale parameters determine the 3D Gaussian’s speed, the rotation parameters determine the 3D Gaussian’s orientation, and the feature parameters determine the 3D Gaussian’s visual characteristics. For example, the feature parameters comprise spectral band parameters representing multiple frequency bands of the visual content. In some examples, the sequence of 3D Gaussians in the encoded bitstream is sequenced based on at least one of spatial significance in a frame, temporal significance across frames, and contribution to reconstruction quality.D24074WO01

[0047] In the entropy decoding and dequantization step 410, the video processing apparatus 600 performs entropy decoding and dequantization of the sequence of 3D Gaussians. By performing entropy decoding and dequantization, the video processing apparatus 600 recovers values of the 3D Gaussians parameters. According to some aspects, during the encoding, to reduce bitstream size, compression processing including quantization of parameter values is applied to the 3D Gaussian parameters to reduce precision while maintaining visual quality. The compression processing is followed by entropy coding to exploit statistical redundancies in the parameter distributions. In this example, the video processing apparatus 600 reverses these operations to restore the parameters to their working precision. Performing entropy decoding and dequantization comprises applying differential decoding relative to neighboring 3D Gaussians in the space-time volume.

[0048] In the temporal position selection step 415, the video processing apparatus 600 selects a temporal position in the space-time volume. For example, after receiving a target playback time, the video processing apparatus 600 selects a temporal position within the space-time volume that corresponds to the frame to be reconstructed. The selected temporal position may determine where to slice through the space-time volume to extract frame information.

[0049] In the slicing step 420, the video processing apparatus 600 generates a factorized 2D Gaussian probability at the selected temporal position by performing time slicing on the sequence of 3D Gaussians. This operation generates a factorized 2D Gaussian probability that describes how each 3D Gaussian contributes to the frame at that temporal position. By generating a 2D Gaussian probability at the selected temporal position, the efficiency for storage is increased. Performing time slicing may comprise at least one of the following operations:D24074WO01 factorizing a probability in a spatial plane, factorizing a probability in a horizontal-temporal plane, and factorizing a probability in a vertical-temporal plane.

[0050] In this example, the space-time volume represents visual content using three dimensions: two spatial dimensions (X and Y coordinates) and one temporal dimension (T, representing time). While examples described thus far use time slicing by taking a slice perpendicular to the temporal axis (creating an X-Y spatial plane at a specific time), the video processing apparatus 600 may also perform alternative slicing operations. These further examples include taking slices perpendicular to the spatial X axis (creating a Y-T plane showing how vertical positions change over time) or perpendicular to the spatial Y axis (creating an X-T plane showing how horizontal positions change over time).

[0051] For each slicing approach, the video processing apparatus 600 factorizes the three-dimensional Gaussian into a two-dimensional (2D) component and a one-dimensional (ID) component in different ways. In time slicing, the factorization combines the two spatial dimensions (X-Y plane) while keeping the temporal dimension (T) separate. In X-direction slicing, the video processing apparatus 600 combines vertical position and time (Y-T plane) while keeping the horizontal dimension (X) separate. In Y-direction slicing, the video processing apparatus 600 combines horizontal position and time (X-T plane) while keeping the vertical dimension (Y) separate. By using this flexible factorization approach, the video processing apparatus 600 analyzes and reconstructs the visual content from different perspectives while maintaining computational efficiency.D24074WO01

[0052] In the frame reconstruction step 425, the video processing apparatus 600 generates a reconstructed frame by applying the factorized 2D Gaussian probability to the sequence of 3D Gaussians. The factorized 2D Gaussian probabilities are used to render the frame for display.

[0053] In some examples, the decoding process 400 further involves a progressive decoding step 430. In the progressive decoding step 430, the video processing apparatus 600 progressive decodes the encoded bitstream based on prioritized Gaussian ordering. The sequence of 3D Gaussians in the encoded bitstream may be sequenced based on spatial significance in a frame, temporal significance across frames, or contribution to reconstruction quality. In this example, Gaussians contributing to visually significant regions or those crucial for temporal coherence are placed at the beginning of the bitstream. By using the prioritized Gaussian ordering, the video processing apparatus 600 generates an initial coarse reconstruction from the first subset of the bitstream, with quality gradually improving as more Gaussians are received and processed.

[0054] In some examples, the prioritized Gaussian ordering is used in the bitstream based on each Gaussian's contribution to visual quality. For example, the encoder 615 orders Gaussians in the bitstream according to each Gaussian's contribution to visual quality. During reconstruction, processing an initial portion of the bitstream produces a coarse representation of the visual content. The prioritized Gaussian ordering considers both spatial and temporal significance of the Gaussians across frames.

[0055] For foveated streaming applications, the prioritized Gaussian ordering assigns higher importance to Gaussians representing regions of high visual attention. This approach may be similar to human visual acuity patterns, where central or focal regions of the visual field receive higher priority in processing and representation.D24074WO01

[0056] For scalable video coding applications, Peak Signal-to-Noise Ratio (PSNR) may be used as a metric for reconstruction at different levels from the same bitstream. The Gaussians are ordered such that each subsequent portion of the bitstream incrementally improves the PSNR of the reconstructed content. As additional Gaussians from subsequent portions of the bitstream are processed, the reconstruction quality progressively improves.

[0057] In some examples, generating the reconstructed frame comprises generating an initial reconstruction using a first subset of the sequence of 3D Gaussians, and progressively refining the initial reconstruction using one or more subsequent subsets of the sequence of 3D Gaussians.

[0058] In some examples, while the method has been described for RGB video data, the video processing apparatus 600 may process various types of visual content. For example, in event camera applications, the RGB parameters are replaced with polarity parameters. For hyper-spectral video processing, the RGB parameters are extended to represent different spectral bands. The video processing apparatus 600 can thus handle diverse types of time- varying visual content with the flexibility in parameter representation.

[0059] FIG. 5 illustrates an example of training method 500 for training a visual content reconstruction system according to aspects of the present disclosure. The training method 500 may be implemented on the video processing apparatus 600. Additionally, the steps provided within in FIG. 5 are merely examples, and may instead be conducted in a different order. While illustrated as being performed serially, some steps of the training method 500 may instead be performed simultaneously.

[0060] During the training, the video processing apparatus 600 may randomly initialize Gaussian parameters aligned on discrete grid using discrete grid positions with random scales andD24074WO01 rotations. For example, the position parameters defining the Gaussian location in the space-time volume are mapped to a normalized range between -1 and 1. In this example, the 3-position parameters are mapped to [-1, 1] units each. The position parameters defining the Gaussian location in the space-time volume are normalized to maintain consistent scales. The scale parameters controlling the spread of each Gaussian distribution are bounded by a maximum value to optimize computational efficiency during fitting.

[0061] At operation 505, video processing apparatus 600 receives a sequence of video frames. These frames video frames are used as the ground truth data against which the reconstructed frames will be compared during the training process.

[0062] At operation 510, the video processing apparatus 600 generates an initial set of 3D Gaussian parameters based on the sequence of video frames. In some examples, this initialization process involves initializing position parameters aligned on a discrete grid to ensure proper spatial coverage. The scale parameters are initialized randomly but capped by a predetermined threshold to maintain reasonable Gaussian spreads, while the rotation parameters are initialized randomly and normalized to unit length to ensure valid rotational representations.

[0063] At operation 515, the video processing apparatus 600 iteratively refines the 3D Gaussian parameters through an optimization process. This operation involves multiple steps that are repeated until convergence or until a predetermined criterion is met. For generating reconstructed frames, the process performs time slicing on 3D Gaussians defined by the current 3D Gaussian parameters to generate factorized two-dimensional (2D) Gaussian probabilities, followed by reconstructing frames from these factorized probabilities. The quality of reconstruction isD24074WO01 evaluated by computing a loss value between the reconstructed frames and corresponding frames from the original frame collection.

[0064] In some examples, this loss value is computed using L1 loss between pixel values of the reconstructed frames and corresponding pixel values of the original frames, which provides a robust measure of reconstruction accuracy while being computationally efficient. Based on this computed loss value, the 3D Gaussian parameters are updated to minimize the difference between the original and reconstructed frames.

[0065] For example, parameters such as position, scale, rotation, and features are adjusted incrementally to improve the reconstruction quality while maintaining the validity constraints of each parameter type. In some implementations, the iteration process continues until a loss value is below a specified threshold, indicating sufficient reconstruction quality, or until a maximum number of iterations is reached.

[0066] FIG. 6 illustrates an example of the video processing apparatus 600 according to aspects of the present disclosure. The video processing apparatus 600 includes an electronic processor 605, a communication interface 610, and a memory 620. The memory 620 stores computer software programs including an encoder 615 and a decoder 650. The encoder 615 includes the gaussian parameter generator 625, the 3D Gaussian generation component 630, the time slicing component 635, the 2D Gaussian extraction component 640, and the rasterization component 645..

[0067] The electronic processor 605 may include a hardware device, such as a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuitD24074WO01 (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof.

[0068] In some examples, the electronic processor 605 is configured to operate a memory array using a memory controller. In other examples, a memory controller is integrated into the electronic processor 605. In some examples, the electronic processor 605 is configured to execute computer-readable instructions stored in the memory 620 to perform various functions. In some aspects, the electronic processor 605 includes special purpose components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0069] The communication interface 610 handles the flow of information between the computer system and external devices or users. For example, the communication interface 610 handles the input from devices such as keyboards, mice, or microphones, and manages output to displays, speakers, or other peripherals. In some examples, the communication interface 610 includes a user interface component.

[0070] The memory 620 includes one or more non-transitory memory devices. Examples of a memory device include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid state memory and a hard disk drive. In some examples, memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause at least one processor of electronic processor 605 to perform various functions described herein.

[0071] In some examples, memory 620 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as an interaction with peripheral components or devices. In some examples, memory 620 includes a memory controller that operates memoryD24074WO01cells of memory 620. For example, the memory controller may include a row decoder, column decoder, or both. In some examples, memory cells within memory 620 store information in the form of a logical state.

[0072] The gaussian parameter generator 625, the 3D Gaussian generation component 630, the time slicing component 635, the 2D Gaussian extraction component 640, the rasterization component 645, and the decoder 650 may be configured to perform functions described with reference to FIGS. 1-5.

[0073] FIG. 7 illustrates experimental results 700 according to embodiments of the present disclosure. The experimental results 700 demonstrate the video fitting and compression capabilities of the video processing apparatus 600.

[0074] In an example, a sample frame in a video sequence shows a test case with significant resolution and complex details. In this example, experiments are performed on the first half of the video sequence from scikit- video, which has 66 frames of resolution. The frame-wise quality statistics are averaged across the entire clip and compared with the smallest version of NeRV reported. The representation is close to NeRV at a fraction of the size.Method Parameters PSNR ↑ MS-SSIM ↑ Ours ( 25 K) 300K 30.12 0.942Our (75K) 900K 31.37 0.959NeRV-S CITE 3.2M 34.21 —

[0075] In another example, the method provided herein is fit to the Ultra Video Group (UVG) which consists of 7 videos with 1920x1080 resolution. The UVG videos are down sampled from 120 FPS to 30 FPS. The first two seconds are selected for fitting. The sample frames from aD24074WO01 UVG video dataset (not shown in FIG. 7) provide another test scenario with different characteristics, where the compressed version consists of 50K Gaussians. The representation of these video frames demonstrates our method’s ability to handle diverse video content.

[0076] The results 715 demonstrate the performance of the video processing apparatus 600 evaluated by computing the PSNR over all N videos at different bitrates corresponding to number of Gaussians: 10K, 20K, 50K, 100K. The experiments reveal that videos with large motion over time (such as camera panning) require longer fitting times to increase performance. For instance, some example videos all reach 30 PSNR, while the rest only reach 25 PSNR given the same number of iterations. These results are obtained while retaining enough Gaussians in high movement areas. These experimental results shown in FIG. 7 demonstrate that the method provided here effectively balances compression ratio and visual quality across different types of video content, with effectiveness in handling complex scenes and motion patterns.

[0077] Aspects of the present disclosure provides some example practical applications in video processing. FIG. 8 illustrates examples of experimental results 800 according to aspects of the present disclosure. The experimental results 800 are test results on diverse video content using the compression and spatially varying resolution capabilities. The sample frames demonstrate compression performance with varying numbers of Gaussians, revealing the balance between compression ratio and visual quality.

[0078] In terms of compression, the representation is naturally compressed since single Gaussians can attend to multiple coordinates. While the representation is sensitive enough to the precision gained with 32-bit floats, experiments show that the parameters can be quantized to 8-bits according to embodiments of the present disclosure, causing approximately less than 35% ofD24074WO01 the original space. Moreover, traditional compression techniques like coin-op 16-bit floats, while scale and rotation require less space, can be incorporated to further reduce storage requirements. For example, in the context of 3D scene representation, researchers have explored bank coding and extreme pruning extended models to reduce the number of Gaussians and space required.

[0079] A particularly significant application is Spatially Varying Resolution (Foveated Streaming). All existing codecs are designed to have a static uniform resolution. This capability is specifically useful for high-resolution and large-display streaming scenarios. Consider wanting to stream a 4K resolution to a large video device where viewers most pixels are in their periphery, they will experience a lower resolution. Traditional codecs may not take advantage of this. In this case, the video processing apparatus 600 can compress in video per level of peripheral attention. For example, in one proof-of-concept demonstration, we compressed the Big Buck Bunny video with Gaussians from the higher or lower bitrate file and stream the combination. In our proof-of-concept demonstration, we select Gaussians from the higher-quality representation if their positions are inside the user focus, and from the lower quality one for the ones outside. This results in -1200 Gaussians from the 10K representation and -400 from the IK representation - resulting in a 40-75% space saving.

[0080] FIG. 9 shows the experimental results 900 using event stream compression methods with probabilistic decoding according to aspects of the present disclosure. In FIG. 9, the experimental results 900 includes a ground truth frame and a compressed frame. The compressed frame is compressed with 50K Gaussians. The experimental results 900 shows how the representation extends to event streams where each event channel is the polarity of positive or events at each pixel and the second channel is the number of negative events. In some examples, fitting technique is used to produce 2D Gaussians describing the events. The decoding processD24074WO01 reconstructs event streams through probabilistic decoding operations. The process generates positive and negative events by sampling from each Gaussian distribution. Event stream reconstruction can be used in applications that involve finite time interpolation in simulation tasks.

[0081] In addition, embodiments of the present disclosure offer denoising capabilities. Since the compression loses high-frequency signal bits, the reconstructions are less noisy than the original signal. This can be useful for cleaning up grainy footage. Accordingly, these practical applications, as demonstrated through FIG. 8 and FIG. 9, demonstrating the versatility and efficiency of the Gaussian-based representation across different video processing scenarios.

[0082] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0083] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include anD24074WO01 electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable readonly memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0084] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0085] A person skilled in the art realizes that the present invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible and considered within the scope of the appended claims. Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims, and which may represent systems, methods, and devices, all arranged in accordance with aspects of the present disclosure.D24074WO01

[0086] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be replaced, amended, or omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0087] EEE1. A method for reconstructing visual content comprises receiving a sequence of visual content frames; encoding the sequence of visual content frames to obtain a sequence of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-timevolume; performing time slicing on the sequence of 3D Gaussians to generate factorized two-dimensional (2D) Gaussian probabilities; and generating reconstructed frames based on the factorized 2D Gaussian probabilities.

[0088] EEE2. The method according to EEE1, wherein the space-time volume comprises two spatial dimensions and one temporal dimension.

[0089] EEE3. The method according to any of EEE1 to EEE2, wherein performing time slicing comprises factorizing a probability density of each 3D Gaussian into a conditional probability in spatial dimensions and a marginal probability in a temporal dimension.

[0090] EEE4. The method according to any of EEE1 to EEE3, wherein generating the reconstructed frames comprises extracting, for each time step, a set of two-dimensional (2D)D24074WO01 Gaussians from the factorized 2D Gaussian probabilities; and computing pixel values by combining contributions from the extracted 2D Gaussians based on their respective probability densities and feature parameters, wherein regions with consistent appearance across multiple frames are represented by 3D Gaussians having larger scale parameters in a temporal dimension.

[0091] EEE5. The method according to any of EEE1 to EEE4, wherein computing pixel values comprises calculating a weighted average of probability densities of each Gaussian at a pixel coordinate multiplied by corresponding feature parameters.

[0092] EEE6. The method according to any of EEE1 to EEE5, wherein the feature parameters comprise at least one of: RGB color values, event polarity indicators, or numbers of positive and negative events for event streams.

[0093] EEE7. The method according to any of EEE1 to EEE6, further comprises culling Gaussians based on at least one of a spatial distance threshold or a temporal probability threshold during frame reconstruction.

[0094] EEE8. The method according to any of EEE1 to EEE7, wherein encoding the sequence of visual content frames comprises representing static regions with 3D Gaussians having larger scale parameters in a temporal dimension; and representing dynamic regions with sequences of 3D Gaussians having smaller scale parameters in the temporal dimension and varying spatial parameters.

[0095] EEE9. A method for training a video representation system comprises receiving a sequence of video frames; generating an initial set of three-dimensional (3D) Gaussian parameters based on the sequence of video frames; and iteratively refining the 3D Gaussian parameters by generating a reconstructed frame using the 3D Gaussian parameters; computing aD24074WO01 loss value between the reconstructed frame and an original frame among the sequence of video frames; and updating the 3D Gaussian parameters based on the computed loss value.

[0096] EEE10. The method according to EEE9, wherein generating a reconstructed frame comprises performing time slicing on 3D Gaussians determined by the 3D Gaussian parameters to generate factorized two-dimensional (2D) Gaussian probabilities; and generating the reconstructed frame based on the factorized 2D Gaussian probabilities.

[0097] EEE11. The method according to any of EEE9 to EEE10, wherein the loss value is computed using L1 loss between pixel values of the reconstructed frame and corresponding pixel values of the original frame.

[0098] EEE12. The method according to any of EEE9 to EEE11, wherein generating the initial set of 3D Gaussian parameters comprises initializing position parameters aligned on a discrete grid; initializing random scale parameters capped by a predetermined threshold; and initializing random rotation parameters normalized to unit length.

[0099] EEE13. An apparatus for video processing comprises an electronic processor; and a memory storing computer executable instructions, wherein the computer executable instructions, when executed, cause the electronic processor to: encode a sequence of visual content frames to obtain a sequence of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a spacetime volume; perform time slicing on the sequence of 3D Gaussians to generate factorized two-dimensional (2D) Gaussian probabilities; and generate reconstructed frames based on the factorized 2D Gaussian probabilities.

[0100] EEE14. The apparatus according to EEE13, wherein the space-time volume comprises two spatial dimensions and one temporal dimension.D24074WO01

[0101] EEE15. The apparatus according to any of EEE13 to EEE14, wherein performing time slicing comprises factorizing a probability density of each 3D Gaussian into a conditional probability in spatial dimensions and a marginal probability in a temporal dimension.

[0102] EEE16. The apparatus according to any of EEE13 to EEE15, wherein generating reconstructed frames comprises extracting, for each time step, a set of two-dimensional (2D) Gaussians from the factorized 2D Gaussian probabilities; and computing pixel values by combining contributions from the extracted 2D Gaussians based on their respective probability densities and feature parameters, wherein regions with consistent appearance across multiple frames are represented by 3D Gaussians having larger scale parameters in a temporal dimension.

[0103] EEE17. The apparatus according to any of EEE13 to EEE16, wherein computing pixel values comprises calculating a weighted average of probability densities of each Gaussian at a pixel coordinate multiplied by corresponding feature parameters.

[0104] EEE18. The apparatus according to any of EEE13 to EEE17, wherein the feature parameters comprise at least one of: RGB color values, event polarity indicators, or numbers of positive and negative events for event streams.

[0105] EEE19. The apparatus according to any of EEE13 to EEE18, wherein the computer executable instructions, when executed, further cause the electronic processor to cull Gaussians based on at least one of a spatial distance threshold or a temporal probability threshold during frame reconstruction.

[0106] EEE20. The apparatus according to any of EEE13 to EEE19, wherein encoding the sequence of visual content frames comprises representing static regions with 3D GaussiansD24074WO01 having larger scale parameters in the temporal dimension; and representing dynamic regions with sequences of 3D Gaussians having smaller scale parameters in a temporal dimension and varying spatial parameters.

[0107] EEE21. A method for decoding visual content comprises receiving an encoded bitstream representing the visual content, the encoded bitstream comprising a sequence of three-dimensional (3D) Gaussians in a space-time volume, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters; performing entropy decoding and dequantization of the sequence of 3D Gaussians; selecting a temporal position in the space-time volume; generating a factorized two-dimensional (2D) Gaussian probability at the selected temporal position by performing time slicing on the sequence of 3D Gaussians; and generating a reconstructed frame by applying the factorized 2D Gaussian probability to the sequence of 3D Gaussians.

[0108] EEE22. The method according to EEE21, wherein the feature parameters comprise spectral band parameters representing multiple frequency bands of the visual content.

[0109] EEE23. The method according to any of EEE21 to EEE22, wherein performing time slicing comprises one of: factorizing a probability in a spatial plane, factorizing a probability in a horizontal-temporal plane, and factorizing a probability in a vertical-temporal plane.

[0110] EEE24. The method according to any of EEE21 to EEE23, wherein the sequence of 3D Gaussians in the encoded bitstream is sequenced based on at least one of: spatial significance in a frame, temporal significance across frames, and contribution to reconstruction quality.D24074WO01

[0111] EEE25. The method according to any of EEE21 to EEE24, wherein performing entropy decoding and dequantization comprises applying differential decoding relative to neighboring 3D Gaussians in the space-time volume.

[0112] EEE26. The method according to any of EEE21 to EEE25, wherein generating the reconstructed frame comprises generating an initial reconstruction using a first subset of the sequence of 3D Gaussians; and progressively refining the initial reconstruction using one or more subsequent subsets of the sequence of 3D Gaussians.

[0113] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0114] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary in made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.D24074WO01

[0115] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

D24074WO01 CLAIMS1. A method for reconstructing visual content, comprising:receiving a sequence of visual content frames;encoding the sequence of visual content frames to obtain a group of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-time volume;performing time slicing on the group of 3D Gaussians to generate factorized two-dimensional (2D) Gaussian probabilities; andgenerating reconstructed frames based on the factorized 2D Gaussian probabilities.

2. The method of claim 1, wherein the space-time volume comprises two spatial dimensions and one temporal dimension.

3. The method of claim 1 or 2, wherein performing time slicing comprises:factorizing a probability density of each 3D Gaussian into a conditional probability in spatial dimensions and a marginal probability in a temporal dimension.

4. The method of any one of claims 1 to 3, wherein generating the reconstructed frames comprises:extracting, for a temporal position, a set of 2D Gaussians from the factorized 2D Gaussian probabilities; andcomputing a pixel value based on the set of 2D Gaussians.D24074WO01 5. The method of claim 4, wherein computing the pixel value comprises:calculating a weighted average of probability densities of each Gaussian at a pixel coordinate multiplied by corresponding feature parameters.

6. The method of any one of claims 1 to 5, wherein the features parameters comprise at least one of: RGB color values, event polarity indicators, or numbers of positive and negative events for event streams.

7. The method of any one of claims 1 to 5, further comprising:culling Gaussians based on at least one of a spatial distance threshold or a temporal probability threshold during frame reconstruction.

8. The method of any one of claims 1 to 7, wherein encoding the sequence of visual content frames comprises:representing static regions with 3D Gaussians having larger scale parameters in a temporal dimension; andrepresenting dynamic regions with sequences of 3D Gaussians having smaller scale parameters in the temporal dimension and varying spatial parameters.

9. A method for training a video representation system, comprising:receiving a sequence of video frames;generating an initial set of three-dimensional (3D) Gaussian parameters based on the sequence of video frames; anditeratively refining the 3D Gaussian parameters by:D24074WO01 generating a reconstructed frame using the 3D Gaussian parameters; computing a loss value between the reconstructed frame and an original frame among the sequence of video frames; andupdating the 3D Gaussian parameters based on the computed loss value.

10. The method of claim 9, wherein generating the reconstructed frame comprises:performing time slicing on 3D Gaussians determined by the 3D Gaussian parameters to generate factorized two-dimensional (2D) Gaussian probabilities; andgenerating the reconstructed frame based on the factorized 2D Gaussian probabilities.

11. The method of any one of claims 9 or 10, wherein the loss value is computed using L1 loss between pixel values of the reconstructed frame and corresponding pixel values of the original frame.

12. The method of any one of claims 9 to 11, wherein generating the initial set of 3D Gaussian parameters comprises:initializing position parameters aligned on a discrete grid;initializing random scale parameters capped by a predetermined threshold; and initializing random rotation parameters normalized to unit length.D24074WO01 13. An apparatus for video processing comprises an electronic processor; and a memory storing computer executable instructions, wherein the computer executable instructions, when executed, cause the electronic processor to:encode a sequence of visual content frames to obtain a group of three-dimensional (3D) Gaussians, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters in a space-time volume;perform time slicing on the group of 3D Gaussians to generate factorized two-dimensional (2D) Gaussian probabilities; andgenerate reconstructed frames based on the factorized 2D Gaussian probabilities.

14. The apparatus of claim 13, wherein the space-time volume comprises two spatial dimensions and one temporal dimension.

15. The apparatus according to claim 13 or 14, wherein performing time slicing comprises factorizing a probability density of each 3D Gaussian into a conditional probability in spatial dimensions and a marginal probability in a temporal dimension.

16. The apparatus according to any of claims 13 to 15, wherein generating reconstructed frames comprises extracting, for each time step, a set of two-dimensional (2D) Gaussians from the factorized 2D Gaussian probabilities; and computing pixel values by combining contributions from the extracted 2D Gaussians based on their respective probability densities and feature parameters, wherein regions with consistent appearance across multiple frames are represented by 3D Gaussians having larger scale parameters in a temporal dimension.D24074WO0117. The apparatus according to any of claims 13 to 16, wherein computing pixel values comprises calculating a weighted average of probability densities of each Gaussian at a pixel coordinate multiplied by corresponding feature parameters.

18. The apparatus according to any of claims 13 to 17, wherein the feature parameters comprise at least one of: RGB color values, event polarity indicators, or numbers of positive and negative events for event streams.

19. The apparatus according to any of claims 13 to 18, wherein the computer executable instructions, when executed, further cause the electronic processor to cull Gaussians based on at least one of a spatial distance threshold or a temporal probability threshold during frame reconstruction.

20. The apparatus according to any of claim 13 to 19, wherein encoding the sequence of visual content frames comprises representing static regions with 3D Gaussians having larger scale parameters in the temporal dimension; and representing dynamic regions with sequences of 3D Gaussians having smaller scale parameters in a temporal dimension and varying spatial parameters.

21. A method for decoding visual content, comprising:D24074WO01 receiving an encoded bitstream representing the visual content, the encoded bitstream comprising a sequence of three-dimensional (3D) Gaussians in a space-time volume, wherein each 3D Gaussian includes position parameters, scale parameters, rotation parameters, and feature parameters;performing entropy decoding and dequantization of the sequence of 3D Gaussians; selecting a temporal position in the space-time volume;generating a factorized two-dimensional (2D) Gaussian probability at the selected temporal position by performing time slicing on the sequence of 3D Gaussians; and generating a reconstructed frame by applying the factorized 2D Gaussian probability to the sequence of 3D Gaussians.

22. The method of claim 21, wherein the feature parameters comprise spectral band parameters representing multiple frequency bands of the visual content.

23. The method of claim 21 or 22, wherein performing time slicing comprises one of: factorizing a probability in a spatial plane, factorizing a probability in a horizontal-temporal plane, and factorizing a probability in a vertical-temporal plane.

24. The method according to any of claims 21 to 23, wherein the sequence of 3D Gaussians in the encoded bitstream is sequenced based on at least one of: spatial significance in a frame, temporal significance across frames, and contribution to reconstruction quality.D24074WO01 25. The method of any one of claims 21 to 24, wherein performing entropy decoding and dequantization comprises:applying differential decoding relative to neighboring 3D Gaussians in the space-time volume.

26. The method of any one of claims 21 to 25, wherein generating the reconstructed frame comprises:generating an initial reconstruction using a first subset of the sequence of 3D Gaussians; andprogressively refining the initial reconstruction using one or more subsequent subsets of the sequence of 3D Gaussians.

27. A computer program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any of claims 1-12 and 20-26.

28. A non-transitory computer-readable storage medium storing the program according to claim 27.

Images