Transmitting 3D gaussian splatting as spheres

Abstract

A method comprising: obtaining (30,31) a 3D Gaussian representative of a part of a 3D scene, the 3D Gaussian being based on a covariance matrix; obtaining (33) a scalar value from the 3D Gaussian; obtaining (34) a simplified representation of the 3D Gaussian wherein the covariance matrix is replaced by an element depending on the scalar value; and, signalling (35) information representative of the simplified representation of the 3D gaussian.

Description

[0001] 2025PF00110

[0002] TRANSMITTING 3D GAUSSIAN SPLATTING AS SPHERES

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to European Application No. 25305374.8, filed March 19, 2025, which is incorporated herein by reference in its entirety.

[0005] BACKGROUND

[0006] The present application generally relates to a method and an apparatus for 3D data compression and processing.

[0007] Advances in 3D capturing and rendering technologies are enabling new applications and services in fields of autonomous driving, cultural heritage archival, immersive telepresence, and virtual / augmented reality.

[0008] One of these new services is the generation of new views (i.e., New View Generation: NVG) of a complex scene from a limited number of captured views.

[0009] Neural Radiance Fields (NeRFs) have proved to be a remarkable method to render new views of a complex scene from a limited number of captured views. During the last three years, a lot of progress has been made to improve the first initial work described in document Mildenhall, B. S. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis, Communications of the ACM, (pp. 99-106). However computational complexity remains a main bottleneck of this method since NeRFs aims at evaluating a luminance of the scene for all points of a 3D volume.

[0010] On the other hand, 3D Gaussian (3DG) splatting has emerged as a new scene representation that is able to render 3D radiance fields with a same quality than NeRF but much faster. Indeed, 3DG splatting avoids unnecessary computation in an empty space in order to decrease training time and achieve real-time rendering.

[0011] 3DG splatting has already been used in the past in the computer graphics community but recently, with the advent of artificial intelligence and the arrival of very powerful GPUs, new learning techniques have become popular, creating a paradigm-shift in the realm of 3D scene representation.

[0012] A main drawback of 3DG splatting (3DGS) is that it requires a large amount of memory for a storage of the Gaussians. In their original paper (Kerbl, B. K. (2023). 3D Gaussian Splatting2025PF00110

[0013] for Real-Time Radiance Field Rendering, ACM Transactions on Graphics (TOG), (pp. 1-14)), the authors report a memory consumption significantly higher than NeRF-based solutions which may be a hurdle for a deployment of 3DGS in consumer devices or low-end GPUs. The solution proposed in the original paper generates a large amount of 3DGS due to a densification process during optimization (cloning and splitting). Furthermore, there is a large number of attributes characterizing each 3DG. All in all, the transmission and manipulation of 3DGs is more complicated than it should be.

[0014] It is desirable to propose solutions to reduce the bulk (i.e., the bitrate) of transmission of the 3DGs.

[0015] BRIEF SUMMARY

[0016] In a first aspect, one or more of the present embodiments provide a method comprising:

[0017] obtaining a 3D Gaussian representative of a part of a 3D scene, the 3D Gaussian being based on a covariance matrix;

[0018] obtaining a scalar value from the 3D Gaussian;

[0019] obtaining a simplified representation of the 3D Gaussian wherein the covariance matrix is replaced by an element depending on the scalar value; and,

[0020] signalling information representative of the simplified representation of the 3D gaussian.

[0021] In a second aspect, one or more of the present embodiments provide a device comprising electronic circuitry configured for:

[0022] obtaining a 3D Gaussian representative of a part of a 3D scene, the 3D Gaussian being based on a covariance matrix;

[0023] obtaining a scalar value from the 3D Gaussian;

[0024] obtaining a simplified representation of the 3D Gaussian wherein the covariance matrix is replaced by an element depending on the scalar value; and,

[0025] signalling information representative of the simplified representation of the 3D Gaussian.

[0026] In an embodiment of the first or second aspect, the covariance matrix is decomposable in a matrixial operation based on a scaling matrix and a rotation matrix and signalling information representative of the simplified representation of the 3D Gaussian comprises signalling the scalar value instead of signalling the scaling matrix.2025PF00110

[0027] In an embodiment of the first or second aspect, the scaling matrix is a diagonal matrix 0 0 \1

[0028] S = I 0 s20 j, and the scalar value is equal to ( L x s2x S3)3or equal to a maximum

[0029]

[0030] \0 0 sj

[0031] of s1?s2and s3.

[0032] In an embodiment of the first or second aspect, responsive to the 3D Gaussian is seen by a direction, the scalar value depends on a semi-major axis and a semi-minor axis of an ellipse resulting from an orthogonal projection in the direction of an ellipsoid representative of the 3D Gaussian on a plan.

[0033] In an embodiment of the first or second aspect, the semi-major axis is equal to a and the 1

[0034] semi-minor axis is equal to b and the scalar value is equal to (a x b)2 or equal to a maximum of a and b.

[0035] In an embodiment of the first or second aspect, signalling information representative of the simplified representation of the 3D Gaussian comprises skipping a signalling of the rotation matrix.

[0036] In an embodiment of the first or second aspect, a syntax element indicates that a simplified representation of the 3D Gaussian is signalled.

[0037] In a third aspect, one or more of the present embodiments provide a method for signalling a plurality of 3D Gaussians representing a 3D scene wherein a 3D Gaussian of the plurality is signalled using the method of the first aspect depending on an eccentricity of the 3D Gaussian or depending on an eccentricity of an ellipse resulting from an orthogonal projection in a direction of an ellipsoid representative of the 3D Gaussian on a plan or depending on a percentage of 3D Gaussians of the plurality already signalled.

[0038] In a fourth aspect, one or more of the present embodiments provide a device for signalling a plurality of 3D Gaussians representing a 3D scene comprising electronic circuitry configured for signalling a 3D Gaussian of the plurality using a simplified representation of the 3D Gaussian, the signalling being performed depending on an eccentricity of the 3D Gaussian or depending on an eccentricity of an ellipse resulting from an orthogonal projection in a direction of an ellipsoid representative of the 3D gaussian on a plan or depending on a percentage of 3D gaussians of the plurality already signalled, a simplified representation of the 3D Gaussian being such that a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value.

[0039] In a fifth aspect, one or more of the present embodiments provide a method comprising:2025PF00110

[0040] obtaining a simplified representation of a 3D gaussian in which a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value; and reconstructing at least a part of a 3D scene using the simplified representation of the 3D Gaussian.

[0041] In a sixth aspect, one or more of the present embodiments provide a device comprising electronic circuitry configured for:

[0042] obtaining a simplified representation of a 3D Gaussian in which a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value; and reconstructing at least a part of a 3D scene using the simplified representation of the 3D Gaussian.

[0043] In an embodiment of the fifth or the sixth aspect, the covariance matrix is decomposable in a matrixial operation based on a scaling matrix and a rotation matrix and wherein the scalar value is signalled in place of the scaling matrix in the simplified representation of the 3D Gaussian.

[0044] In an embodiment of the fifth or the sixth aspect, the scaling matrix is a diagonal matrix ° ° \1

[0045] S = ( 0 s20 ), and the scalar value is equal to ( L x s2x s3)3or equal to a maximum \0 0 s

[0046] of s1?s2and s3.

[0047] In an embodiment of the fifth or the sixth aspect, the scalar value depends on a semi-major axis and a semi -minor axis of an ellipse resulting from an orthogonal projection in a direction of an ellipsoid representative of the 3D Gaussian on a plan.

[0048] BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The following detailed description will be better understood when read in conjunction with the appended drawings, in which there are shown examples of one or more of the multiple embodiments of the present disclosure. It should be understood, however, that the embodiments described herein are not limited to the precise arrangements and instrumentalities shown in the drawings. In the drawings:

[0050] FIG. 1A describes an example of a context in which following embodiments can be implemented;

[0051] FIG. IB is a block diagram illustrating an example system according to one or more embodiments of the present disclosure;

[0052] FIG. 2 is a process diagram illustrating an example 3DG generation process according to some embodiments;2025PF00110

[0053] FIG. 3 illustrates various embodiments of a process signalling a simplified representation ofa3DG;

[0054] FIG. 4 illustrates an orthogonal projection of an ellipsoid E;

[0055] FIG. 5 illustrates the first and second embodiment; and,

[0056] FIG. 6 illustrates various embodiments of a process of receiving a simplified representation ofa3DG.

[0057] DETAILED DESCRIPTION

[0058] In describing the various embodiments of the present disclosure, certain terminology is used herein for convenience only and should not be considered as limiting such embodiments. In the drawings, the same reference numerals are employed for designating the same elements throughout the several figures and the present description.

[0059] FIG. 1A describes an example of a context in which following embodiments can be implemented.

[0060] In FIG. 1 A, a set of cameras are in charge of capturing a plurality of views of a same scene. Images representing the different views are transmitted to a system 11. The system 11 is in charge of generating a representation from the plurality of views allowing generating not only the captured views but new views of the scene. The system 11 then encodes and transmits this encoded representation to a system 13 via a network 12. The system 11 may be a computer, a server or any device capable of generating, encoding and transmitting the representation.

[0061] The communication channel 12 is a wired (for example Internet or Ethernet) or a wireless (for example WiFi, 3G, 4G or 5G) network link.

[0062] The system 13, that could be for example a set top box, receives and decodes the encoded representation to generate at least one view of the scene.

[0063] Each generated view is then transmitted to a display system 15 using a communication channel 14, that could be a wired or wireless network. The display system 15 then displays at least one of the generated views.

[0064] In an embodiment, the system 13 is comprised in the display system 15. In that case, the system 13 and display 15 are comprised in a TV, a computer, a tablet, a smartphone, a headmounted display, or any system allowing virtual reality (VR) / mixed reality (MR) / augmented reality (AR).

[0065] FIG. IB illustrates an example of system 140 implementing the system 11 or the system 13 in which embodiments of the present disclosure can be implemented.2025PF00110

[0066] The system 140 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, vehicle control and / or entertainment system, a head mounted display, and servers. Elements of system 140, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 140 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 140 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 140 is configured to implement one or more of the aspects described in this document.

[0067] The system 140 includes at least one processor 142 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. The processor 142 may be a general-purpose processor or microprocessor, digital signal processor (DSP), one or more microprocessors in association with a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), a state machine, and the like. The processor 142 may include at least one central processing unit (CPU), embedded memory, input and output interfaces. All these types of processors are electronic circuitries.

[0068] Processor 142 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 140 includes at least one memory 144 (e.g., a volatile memory device, and / or anon-volatile memory device). System 140 may include a storage device 148, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 148 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0069] System 140 includes an encoder / decoder module 146 configured, for example, to process raw image data / encoded image data (such as captured views, new views, point clouds, etc) and to provide encoded image data / decoded image data. The encoder / decoder module 146 may include2025PF00110

[0070] its own processor and memory. The encoder / decoder module 146 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 146 can be implemented as a separate element of system 140 or can be incorporated within processor 142 as a combination of hardware and software as known to those skilled in the art.

[0071] Instructions for execution by the processor 142 and / or the encoder / decoder module 146 may be stored in the storage device 148 and subsequently loaded into memory 144 for execution by the processor 142. In some embodiments, one or more of processor 142, memory 144, storage device 148, and encoder / decoder module 146 may store one or more items when performing the processes disclosed herein. Such items may include input raw image data, reconstructed image data, decoded image data or portions thereof, bitstreams, chunks, matrices, variables, operational logic, and intermediate and / or final results from processing of equations, formulas, or operations.

[0072] In some embodiments, memory inside of the processor 142 and / or the encoder / decoder module 146 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 142 or the encoder / decoder module 146) is used for one or more of these functions. The external memory can be the memory 144 and / or the storage device 148, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for coding and decoding operations.

[0073] The system 140 may obtain or receive information via one or more input devices, interfaces, and / or ports as indicated in input block 162. Examples of the input devices include a radio frequency (RF) device for transmitting and / or receiving RF signals over various media, for example, RF signals received over the air from a broadcaster; component video (COMP) inputs; a Universal Serial Bus (USB) input; and / or a High-Definition Multimedia Interface (HDMI) input. Other examples include composite video input (not shown). In some embodiments, the input devices are associated with respective input processing elements, e.g., those generally known in the art. For example, the RF device may be associated with elements suitable for selecting a desired frequency (e.g., selecting or band-limiting a signal) or performing error correction on the signal. The USB and / or HDMI inputs may include respective interface processors and transceivers (or transmitters and receivers) for coupling the system 140 to other devices via USB and / or HDMI2025PF00110

[0074] ports or connections. Various forms of input processing may be implemented, for example, by and / or within a separate input processing device or the processor 142.

[0075] Various elements of system 140 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 164, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0076] The system 140 includes a communication interface 150 that enables wired and / or wireless communication with other devices, e.g., via a communication channel 152 (corresponding to communication channel 12 or 14 in FIG. 1A). The communication interface 150 may include one or more transceivers, modems, network cards and the like. The communication channel 152 may be or include wired and / or wireless mediums.

[0077] In some embodiments, data may be streamed to the system 140 via wired and / or wireless networks. Examples of such wireless networks include cellular, Bluetooth or Wi-Fi (e.g., IEEE 802.11) networks. The wired and / or wireless networks may include one or more base stations (e.g., cellular base stations, access points, etc.), and / or user equipment (e.g. cellular user equipment, stations, etc.), and / or other network elements that communicate with the system 140 via the communication interface 150 and communication channel 152, whereby the system 140 may obtain data streamed from streaming applications (e.g., OTT (Over The Top) services) via various networks, including the Internet. In some embodiments, data is streamed to the system 100 via the input block 105 (e.g., using a set-top box that delivers data via the HDMI connection or the RF connection). In some embodiments, data is received by the system 100 in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0078] The system 140 may provide an output signal to various output devices, including a display 166, speakers 168, and other peripheral devices 170. The display 166 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 166 can be for a television, a tablet, a laptop, a cell phone (mobile phone), head-mounted display, AR, VR, MR systems, or other device or user equipment. The display 166 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 170 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 170 that provide a function based on the output of the system 140.2025PF00110

[0079] In various embodiments, control signals are communicated between the system 140 and the display 166, speakers 168, or other peripheral devices 170 using signaling such as AV. Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 140 via dedicated connections through respective interfaces 154, 156, and 158. Alternatively, the output devices can be connected to system 140 using the communications channel 152 via the communications interface 150. The display 166 and speakers 168 can be integrated in a single unit with the other components of system 140 in an electronic device such as, for example, a television.

[0080] The display 166 and speaker 168 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 162 is part of a separate set-top box. In various embodiments in which the display 166 and speakers 168 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0081] The system 140 may include one or more sensor devices 160. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopic sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors may be used to determine information such as user’s position and orientation. Where the system 140 is used as a control module for an extended reality display (such as control modules), the user’s position and orientation may be used in determining how to render image data such that the user perceives the correct portion of a virtual object or virtual scene from the correct point of view. In the case of a user equipment such as ahead-mounted display devices, the position and orientation of the device itself may be used to determine a position and orientation of a user for the purpose of rendering virtual content. In the case of other display devices, such as a phone, a tablet, a computer monitor, or a television, other inputs may be used to determine the position and orientation of the user for the purpose of rendering content. For example, a user may select and / or adjust a desired viewpoint and / or viewing direction with the use of a touch screen, keypad or keyboard, trackball, joystick, or other input. Where the display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation used for the purpose of rendering content may be selected and / or adjusted based on motion of the display device.

[0082] A User Equipment (UE) may correspond to any extended Reality (XR) device / node which may come in variety of form factors. Typical UE (e.g., XR UE) may include, but not limited to the following: Head Mounted Displays (HMD), optical see-through glasses and video see-through HMDs for Augmented Reality (AR) and Mixed Reality (MR), mobile devices with positional2025PF00110

[0083] tracking and camera, wearables etc. In addition to the above, several different types of XR UE may be envisioned based on XR device functions for e.g., as display, camera, sensors, sensor processing, wireless connectivity, XR / Media processing, and power supply, to be provided by one or more devices, wearables, actuators, controllers and / or accessories. One or more device / nodes / UEs may be grouped into a collaborative XR group for supporting any of XR applications / experience / services.

[0084] During last years, different volumetric representations have been proposed to be used in the context of New View Generation (NVG). NVG generates new views from arbitrary viewing angles of a scene when only a limited number of input views have been captured. In particular, 3D Gaussian (3DG) splatting has emerged as a new scene representation that is able to realistically render volumetric data from a multi-view capture with high quality according to Kerbl, B. K. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering, ACM Transactions on Graphics (TOG), (pp. 1-14).

[0085] Kerbl is understood to explain how to generate, in a reasonable time, a 3DG representation from a discrete set of views of the same static 3D scene, thus enabling high-quality fast rendering of virtual views. Since then, numerous papers have improved the original method and used this new scene representation into a variety of application domains.

[0086] As already mentioned above, a main drawback of 3DG splatting (3DGS) is that it requires a large amount of memory for a storage of the 3DGs and a large bitrate for a transmission of the 3DGs.

[0087] In the following, various embodiments propose to reduce the storage and bitrate by representing 3DGs as isotropic spheres.

[0088] FIG. 2 illustrates an example 3DG generation process 200 according to some embodiments.

[0089] The 3DG generation process of 200 (detailed in Kerbl) is for example implemented by the processor 142 of the system 140 when the system 140 implements the system 11.

[0090] We suppose here that a pre-processing process, not represented, generates a sparse set of 3D points 202 from the plurality of views generated by the set of cameras 10.

[0091] In a step 204, the processor 142 applies an initialization process using a sparse set of 3D points 202. During step 204, the processor 142 generates initial isotropic 3D Gaussians 206 centered at each point from the sparse set of 3D points 202.2025PF00110

[0092] Starting from this initialization process in step 204, an iterative optimization process based on steps 210, 214 and 212 learns a 3DG based representation of the 3D scene from the plurality of view 208 generated, for example, by the set of cameras 10.

[0093] In step 210, the processor 142 applies a projection process using the 3D Gaussians 206 and camera parameters of the plurality of views 208.

[0094] In step 214, the processor 142 applies a process of differentiable tile rasterizering to an output of projection process of step 210. The process of differentiable tile rasterizering of step 214 outputs rendered images at training input views positions of the plurality of views. During step 214, differences between rendered image and training input views are computed and errors are propagated backward to the iterative optimization process through a gradient flow.

[0095] In step 212, a process of adaptive density control performs splitting and cloning operations and outputs a revised version of the 3D Gaussians 206. This revised version is updated in an opposite direction of a gradient (stochastic gradient descent algorithm) and inputted to step 210 for a new iteration.

[0096] The iterative optimization process of step 210, 214 and 212 repeats until the differences between rendered images and training input views are within a threshold of a stop criteria or until a maximum number of iterations is reached.

[0097] After the iterative optimization process is done, step 214 can output an image 216 at any virtual view position.

[0098] Another output of the process of FIG. 2 is the iteratively optimized 3DGs 206.

[0099] Formally, a 3DG is defined as an object in the 3D space as follows:

[0100] G

[0101]

[0102] (%) = (!)

[0103] The object is characterized by its center / r = (x, y, z~), a 3D covariance matrix T, where T can be decomposed into a scaling matrix S (with parameters (i.e., matrix coefficients) s1(s2and s3) and a rotation matrix R (encoded into a unit quaternion q with coefficients q1,q2, q3and q4) as follows:

[0104] Σ = RSSTRT(2)

[0105] Where,Tindicates the transpose.

[0106] One can note that the scaling matrix S is diagonal:

[0107] s10 0

[0108] S = [ 0 s20 j

[0109]

[0110] \0 0 sj2025PF00110

[0111] Besides the geometric description of each 3DG, other attributes are considered such as the opacity a and the color c, which is usually a list of spherical harmonics {cn|n = 1,, N}.

[0112] Then, given a list of 3DGs representing a 3D scene, and an image position of a new view, a synthetized image is generated in the new position via the projection of all 3DGs in the image plane. This operation, also called splatting, corresponds to an a-blending of the different 3DGs. In detail, given a sorted list of 3DGs N, a color of a pixel in the new image is given by the following formula:

[0113] i-1

[0114] C = (3)

[0115]

[0116] iEN j = l

[0117] where q is the learnt color of the i-th Gaussian, a'j = aj gj(x) aj being a;-th Gaussian opacity and <

[0118]

[0119] q7(x) a;-th Gaussian projection on the new image plane.

[0120] FIG. 3 illustrates various embodiments of a process signalling a simplified representation of a 3DG.

[0121] The process of FIG. 3 is for example implemented by the processor 142 of the system 140 when the system 140 implements the system 11.

[0122] In a step 30, the processor 142 obtains a 3DG. For instance, the 3DG is one of the iteratively optimized 3DGs 206 representing a 3D scene. As seen above, this 3DG is represented by the formula (1).

[0123] In a step 31, the processor 142 obtains the covariance matrix Σ from the 3DG given by formula (1).

[0124] In a step 32, the processor 142 obtains the scaling matrix S resulting from the decomposition of the covariance matrix Σ of equation (2).

[0125] In a step 33, the processor 142 obtains a scalar value 5’ from the 3DG. Step 33 is detailed below.

[0126] In a step 34, the processor 142 obtains a simplified representation of the 3DG based on the scalar value 5’ wherein the covariance matrix Σ is replaced by a new covariance matrix Σ'.

[0127] S' = R. S’)2. RT

[0128] The simplified representation of the 3DG is given by the following formula:

[0129] G

[0130]

[0131] (%) = (4)

[0132] In a step 35, the processor 142 signals the simplified representation of the 3DG instead of the 3DG in a bitstream.

[0133] The bitstream may be transmitted, for example, to the system 13.2025PF00110

[0134] In a first embodiment, the scalar value S’ obtained in step 33 by the following formula:

[0135] s = (s1x s2x s3)⅓(5)

[0136] In this first embodiment, only one parameter (the scalar value S') is encoded when encoding the simplified representation of the 3DG (i.e., formula (4) instead of three parameters (s1,s2and s3) when encoding the 3DG (i.e., formula (1)). Hence, A bitrate reduction is obtained.

[0137] When a 3DG is mainly seen by a direction d, its dimension along this direction has a little impact for rendering. This can occur when the cameras used to capture the scene have a limited coverage of the scene, for example when using coplanar placed cameras, or when the 3DG is occluded by other 3DGs for certain directions, limiting the number of directions it can be seen. Thus, the 3DG may be splatted to a 2DG in the direction d.

[0138] In a 3D space, a 3DG can be viewed as an ellipsoid E. A splatting of a 3DG to a 2DG in the direction d may be viewed as an orthogonal projection in the direction d of the ellipsoid E representative of the 3DG on a plan P. In that case a result of the projection is an ellipse Ep.

[0139] FIG. 4 illustrates an orthogonal projection of an ellipsoid E.

[0140] An ellipse may be represented by two dimensions: a representing a semi-major axis and b representing the semi-minor axis. An example of equation representing an ellipse is the following:

[0141] x 2 y 2

[0142] - F — = 1

[0143] a2b2

[0144] In a second embodiment, the scalar value S' is obtained from the semi-major axis a and the semi-minor axis b as follows:

[0145] S' = (a x b)½(6)

[0146] In this second embodiment, again, only one parameter (the scalar value S') is encoded when encoding the simplified representation of the 3DG (i.e., formula (4) instead of three parameters (s1,s2and s3) when encoding the 3DG (i.e., formula (1)).

[0147] FIG. 5 illustrates the first and second embodiment. The simplification of the scale parameters (i.e., of the parameters s1,s2and s3of the scaling matrix S) process is like giving a size to a point.

[0148] In a third embodiment, rather than using a geometric mean as in equations (5) and (6), the maximal value max(s1, s2, s3) or max(a,b) may be used. Any other method may be employed.2025PF00110

[0149] A consequence of the previous embodiments is that no rotation is required anymore, since a sphere is isotropic. Hence, in a fourth embodiment, an additional size reduction is obtained by skipping a signalling of the coefficients q1, q2, q3and q4of the unit quaternion q.

[0150] In that case the covariance matrix Σ is replaced by a scalar value (S')2and the simplified representation of the 3DG is given by the following formula:

[0151] G

[0152]

[0153] (%) = (X-M)

[0154] As changing an ellipsoid into a sphere can introduce distortions when reprojecting the 3DGs to synthesize a view, in a fifth embodiment, it is proposed to apply the first, second, third or fourth embodiment to only a part of the 3DGs. For instance, the first, second, third or fourth embodiment is applied only to the 3DGs that exhibit a small eccentricity, i.e., the ones that are the more like a sphere, and the other are kept unchanged. In an example, the eccentricity e may be obtained as follows:

[0155] m2

[0156] 1“ M

[0157]

[0158] 2J

[0159] where m = min(s1,s2,s3) and M = max(s1,s2,s3). In another example, m = min s12,s22,s32) and M = max(s12,s22,s32). When m and M are close to each other, the eccentricity e tends towards “0”. When m and Mare very different, the eccentricity e tends towards “J”

[0160] Alternatively, the eccentricity e may be computed as follows:

[0161]

[0162] For selecting the 3DGs to simplify (i.e., on which applying the first, second, third or fourth embodiment), either an absolute threshold ta, a relative threshold tror a percentage ptotof the total number of 3DGs may be used. Table TAB1 provides example of process of selecting the 3DGs of a set of N 3DGs to simplify:

[0163] Absolute threshold (e.g. ta= 0.1) for i in range[0, N[:

[0164] if e[i] < ta:

[0165] then simplify 3D Gaussian with index i

[0166]

[0167] 2025PF00110

[0168] Relative threshold (e.g. tr= 0.25) for i in range[0, N[:

[0169] if e[i] < E * tr:

[0170] then simplify 3D Gaussian with index i

[0171] where E = max(e[idx])

[0172] idx

[0173] Percentage (e.g. ptot= 5%) for j in range[0, N[:

[0174] if j < N * Ptot- then simplify 3D Gaussian with index j

[0175] Where j is an index on the sorted list of 3DGs by increasing eccentricity.

[0176]

[0177] Table TAB1

[0178] Having two different types of 3DGs (simplified or traditional) means that the encoder should signal the type of 3DG to the decoder, to use the correct parsing for each one. In a sixth embodiment a symbol is transmitted to signal the type of each 3DG. For instance, a one-bit syntax element type coding the type of 3DG is signalled for each 3DG:

[0179] type=0 for traditional 3DG,

[0180] type=1 for simplified 3DG.

[0181] This additional bit for each 3DG can be balanced by the reduction in size due to the reduction of the parameters (scale parameters, rotation coefficients).

[0182] FIG. 6 illustrates various embodiments of a process of receiving a simplified representation of a 3DG.

[0183] The process of FIG. 6 is for example implemented by the processor 142 of the system 140 when the system 140 implements the system 13.

[0184] In a step 60, the processor 142 obtains a simplified representation of a 3DG. For example, the 3DG belongs to a set of 3DGs representing a 3D scene transmitted by the system 11. The simplified representation is one of the simplified representations described in relation to the previous embodiments.

[0185] In a step 61, the processor 142 reconstructs a part of the 3D scene based on the obtained simplified representation of the 3DG. In an embodiment, the processor 142 determines whether the 3DG is a traditional 3D Gaussian or a simplified 3DG using the one-bit syntax element type.2025PF00110

[0186] One or more embodiments provide a computer program comprising instructions which when executed by one or more processors cause such processors to perform the process of FIG. 3 according to any of the embodiments described above. One or more embodiments also provide a computer readable storage medium having stored thereon instructions for the process of FIG. 3.

[0187] One or more embodiments provide a computer readable storage medium having stored thereon 3DGs generated using the process of FIG. 3. One or more embodiments also provide a method and apparatus for transmitting or receiving 3DGs generated according to the process of FIG. 3.

[0188] The embodiments described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., as a method), the implementation of such features may also be implemented in other forms. An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. Corresponding methods may be implemented in, for example, a processor.

[0189] The various embodiments described herein provide at least the following features, devices or aspects, alone or on any combination, across various claim categories and types:

[0190] i. Encoding, into coded image data, syntax elements that can enable the decoder to decode the coded image data, according to any of the embodiments described herein. ii. A bitstream that includes one or more of the described syntax elements, or variations thereof, whether transmitted, stored, or otherwise made available.

[0191] iii. Creating, transmitting, receiving, and / or decoding of the bitstream.

[0192] iv. An electronic device (e.g., TV, set-top box, mobile phone, tablet, etc.) that tunes a channel to receive a bitstream or that receives such bitstream over the air. The electronic device decodes the syntax elements from the bitstream, and, optionally, displays (e.g., via a monitor or other type of display) a resulting image. Various numeric values are used in the present application. Such specific values are for example purposes and the embodiments described are not limited to these specific values.

[0193] Various methods are described herein, and such methods comprise one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.2025PF00110

[0194] The present disclosure may refer to “determining” various pieces of information. Determining information may include one or more of, for example, estimating, calculating, predicting, or retrieving (e.g., from memory) the information.

[0195] The present disclosure may refer to “accessing” various pieces of information. Accessing information may include one or more of, for example, receiving, retrieving (e.g., from memory), storing, moving, copying, calculating, determining, predicting, or estimating the information. Similarly, the present disclosure may refer to “receiving” various pieces of information. Receiving information may include one or more of, for example, accessing or retrieving (e.g., from memory) the information.

[0196] The present disclosure refers to information, for example, syntax elements, that can be transmitted or stored. Such information can be packaged or arranged in a variety of manners. These manners for example includes manners that are common for system level or application-level standards such as signaling the information into one or more of the following:

[0197] i. session description protocol (SDP), for example as described in RFCs and / or used in conjunction with real-time transport protocol (RTP) transmission.

[0198] ii. hypertext transfer protocol (HTTP) live Streaming (HLS) manifest transmitted over HTTP.

[0199] iii. dynamic adaptive streaming over HTTP (DASH) media presentation description (MPD) descriptors, for example as used in DASH and transmitted over HTTP. iv. RTP header extensions, for example as used during RTP streaming.

[0200] v. International Organization for Standardization (ISO) base media file format, for example, as used in Omnidirectional MediA Format (OMAF).

[0201] As used herein, “signal” and “signaling” refer to, among other things, indicating information to a decoder. For example, in some embodiments the encoder signals a type of 3DG, whereby the same parameter is used for both encoding and decoding. In some embodiments, the signaling may be explicit, such that information (e.g., a particular parameter) is transmitted to the decoder enabling the decoder to use the same particular parameter. In some embodiments, the signaling may be implicit, in that the information (e.g., a particular parameter) is indicated based on other information at or transmitted to the decoder or derived or selected by the decoder based on information available at the decoder. By not transmitting the information (e.g., the particular parameter), a bit savings is thus realized in some embodiments. In some embodiments, one or2025PF00110

[0202] more syntax elements or flags are used to signal information to a decoder. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0203] In some embodiments, signals may be produced that are formatted to carry information that may be stored or transmitted. Such information may include, for example, instructions for performing a method, or data produced by one of the described implementations (e.g., a bitstream of a described embodiment). Such a signal may be formatted, for example, as an electromagnetic wave or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links and may be stored on a processor-readable medium.

[0204] It is to be understood that use of any of the following

[0205]

[0206] “and / or”, and “at least one of’ is intended to encompass all possible selections of listed items, taken either individually or in any combination thereof.

[0207] While specific embodiments have been described in the foregoing description in connection with the accompanying drawings, it should be understood that embodiments described herein are examples only and should not be taken as limiting the scope of the present disclosure or the following claims. Although features and elements are described herein in particular combinations, those of ordinary skill in the art will appreciate that such features or elements may be used alone or in any combination with the other features and elements. It is understood, therefore, that the overall teachings of the present disclosure are not limited to the particular embodiments, implementations, and examples disclosed herein, but are intended to cover variations, modifications, and alternatives as defined by the appended claims and any and all equivalents thereof.

Claims

2025PF00110CLAIMSWhat is claimed is:

1. A method comprising:obtaining (30,31) a 3D Gaussian representative of a part of a 3D scene, the 3D Gaussian being based on a covariance matrix;obtaining (33) a scalar value from the 3D Gaussian;obtaining (34) a simplified representation of the 3D Gaussian wherein the covariance matrix is replaced by an element depending on the scalar value; and,signalling (35) information representative of the simplified representation of the 3D gaussian in a bitstream.

2. A device comprising electronic circuitry configured for:obtaining (30,31) a 3D Gaussian representative of a part of a 3D scene, the 3D Gaussian being based on a covariance matrix;obtaining (33) a scalar value from the 3D Gaussian;obtaining (34) a simplified representation of the 3D Gaussian wherein the covariance matrix is replaced by an element depending on the scalar value; and,signalling (35) information representative of the simplified representation of the 3D Gaussian in a bitstream.

3. The method of claim 1 or the device of claim 2 wherein, the covariance matrix is decomposable (32) in a matrixial operation based on a scaling matrix and a rotation matrix and signalling information representative of the simplified representation of the 3D Gaussian comprises signalling the scalar value instead of signalling the scaling matrix.

4. The method of claim 3 or the device of claim 3 wherein, the scaling matrix is a diagonal matrixh ° ° \1S = ( 0 s20 ), and the scalar value is equal to (S-L x s2x s3)3 or equal to a maximum of s1?\0 0 ss2and s3.2025PF001105. The method of claim 3 or the device of claim 3 wherein, responsive to the 3D Gaussian is seen by a direction (d), the scalar value depends on a semi-major axis and a semi -minor axis of an ellipse resulting from an orthogonal projection in the direction (d) of an ellipsoid representative of the 3D Gaussian on a plan.

6. The method of claim 5 or the device of claim 5 wherein, the semi-major axis is equal to a and the semi-minor axis is equal to b and the scalar value is equal to (a x b' or equal to a maximum of a and b.

7. The method of any previous claim from claim 3 to 6 or the device of any previous claim from claim 3 to 6 wherein, signalling information representative of the simplified representation of the 3D Gaussian comprises skipping a signalling of the rotation matrix.

8. The method according to any previous claim from claim 1, 3 to 7 or the device of any previous claim from claim 2 to 7 wherein, a syntax element indicates that a simplified representation of the 3D Gaussian is signalled.

9. A method for signalling a plurality of 3D Gaussians representing a 3D scene in a bitstream wherein a 3D Gaussian of the plurality is signalled using the method of any previous claim from claim 1, 3 to 8 depending on an eccentricity of the 3D Gaussian or depending on an eccentricity of an ellipse resulting from an orthogonal projection in a direction (d) of an ellipsoid representative of the 3D Gaussian on a plan or depending on a percentage of 3D Gaussians of the plurality already signalled.

10. A device for signalling a plurality of 3D Gaussians representing a 3D scene in a bitstream comprising electronic circuitry configured for signalling a 3D Gaussian of the plurality using a simplified representation of the 3D Gaussian, the signalling being performed depending on an eccentricity of the 3D Gaussian or depending on an eccentricity of an ellipse resulting from an orthogonal projection in a direction of an ellipsoid representative of the 3D gaussian on a plan or depending on a percentage of 3D gaussians of the plurality already signalled, a simplified representation of the 3D Gaussian being such that a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value.

11. A method comprising:obtaining (60), from a bitstream, a simplified representation of a 3D gaussian in which a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value; and reconstructing (61) at least a part of a 3D scene using the simplified representation of the 3D Gaussian.2025PF0011012. A device comprising electronic circuitry configured for:obtaining (60), from a bitstream, a simplified representation of a 3D Gaussian in which a covariance matrix of the 3D Gaussian is replaced by an element based on a scalar value; and reconstructing (61) at least a part of a 3D scene using the simplified representation of the 3D Gaussian.

13. The method of claim 11 or the device of claim 12 wherein, the covariance matrix is decomposable in a matrixial operation based on a scaling matrix and a rotation matrix and wherein the scalar value is signalled in place of the scaling matrix in the simplified representation of the 3D Gaussian.

14. The method of claim 13 or the device of claim 13 wherein, the scaling matrix is a diagonal matrix° ° \1S = ( 0 s20 ), and the scalar value is equal to (.$! x s2x s3)3 or equal to a maximum of s1?\0 0 ss2and s3.

15. The method of claim 13 or the device of claim 13 wherein, the scalar value depends on a semi-major axis and a semi -minor axis of an ellipse resulting from an orthogonal projection in a direction (d) of an ellipsoid representative of the 3D Gaussian on a plan.

Images