Volumetric video streaming using compressed gaussian splatting

Abstract

An image processing method and apparatus is disclosed and includes obtaining a plurality of images of an image capture region captured by a plurality of image capture devices location at different positions around the image capture region, generating a predetermined number of gaussians using the plurality of captured images based on points in a 3D representation of the image capture region, generating an initial number splats for the given scene, wherein each splat includes certain number of the predetermined gaussians, optimizing the number of splats using at least one optimization parameter, compressing the number of optimized splats into a file; and transmitting the compressed filed to at least one external apparatus that decompresses and displays the compressed file to view, one the external apparatus, the 3D image of the image capture region.

Description

TITLEVolumetric Video Streaming Using Compressed Gaussian SplattingCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This PCT application claims the benefit of priority from US Provisional Patent Application Serial No. 63 / 742,366 filed on January 6, 2025, the entirety of which is incorporated herein by reference.BACKGROUNDField

[0002] The present disclosure relates generally to volumetric video capture and transmission to a display device.Description of Related Art

[0003] Streaming high quality volumetric video to user’s devices such as phones / VR headsets has traditionally been done by representing 3D data using meshes (a set of triangles or quads with an accompanying texture file). However, in order for this data to be transmitted, effective compression of the sequence of meshes is required. One technique used for compression of this data includes compressing texture over time by not having a single texture image per frame but rather a texture video which can span multiple frames. Another technique used is by using a library that compresses the data in particular manner such as the library Draco by GOOGLE®. Further compression techniques include temporal compression such that meshes are compressed by establishing correspondence between meshes in the sequence and encoding the changes therebetween.

[0004] Another technique for streaming volumetric video includes transmitted the 3D volumetric data as a point cloud which is a set of colored points used to represent 3D data. The drawback associated with this is the requirement of a very large amount of points in the 3D point cloud to encode high quality 3D data. There are techniques to compress the point cloud prior to transmission but end-to-end solutions that effectively transmit and stream volumetric video data are not yet readily available.SUMMARY

[0005] According to the present disclosure, an image processing method and apparatus is disclosed. The method includes obtaining a plurality of images of an image capture region captured by a plurality of image capture devices location at different positions around the image capture region, generating a predetermined number of gaussians using the plurality of captured images based on points in a 3D representation of the image capture region, generating an initial number splats for the given scene, wherein each splat includes certain number of the predetermined gaussians, optimizing the number of splats using at least one optimization parameter, compressing the number of optimized splats into a file; and transmitting the compressed filed to at least one external apparatus that decompresses and displays the compressed file to view, one the external apparatus, the 3D image of the image capture region.

[0006] In another embodiment, an apparatus is provided that includes one or more memories storing instructions and one or more processors that, upon execution of the stored instructions, configures the one or more processors to perform operations in accordance with any of the embodiments described herein.

[0007] In another embodiment, a server is provided that includes one or more memories storing instructions and one or more processors that, upon execution of the stored instructions, configures the one or more processors to perform operations in accordancewith any of the embodiments described herein.

[0008] In another embodiment, a non-transitory computer readable storage medium is provided that stores instructions that, when executed by one or more processors, configures an apparatus or device to perform one or more methods of any of the embodiments described herein.

[0009] These and other objects, features, and advantages of the present disclosure will become apparent upon reading the following detailed description of exemplary embodiments of the present disclosure, when taken in conjunction with the appended drawings, and provided claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Fig. 1 is a flow diagram detailing an exemplary processing algorithm according the present disclosure.

[0011] Fig. 2 shows illustrates an exemplary operational environment for capturing a plurality of images according to the present disclosure.

[0012] Fig. 3A is an exemplary image generated by the processing algorithm illustrating a first imaging component having a first parameter according to the present disclosure.

[0013] Fig. 3B is an exemplary image generated by the processing algorithm illustrating a first imaging component having a second parameter according to the present disclosure.

[0014] Fig. 4 is an exemplary image generated by the processing algorithm illustrating a second imaging component according to the present disclosure.

[0015] Fig. 5A is an exemplary image generated by the processing algorithm illustrating a third imaging component having a first parameter according to the present disclosure.

[0016] Fig. 5B is an exemplary image generated by the processing algorithm illustrating a third imaging component having a second parameter according to the present disclosure.

[0017] Fig. 6A is an exemplary image generated by the processing algorithm illustrating a fourth imaging component having a first parameter according to the present disclosure.

[0018] Fig. 6B is an exemplary image generated by the processing algorithm illustrating a fourth imaging component having a second parameter according to the present disclosure.

[0019] Fig. 7A is an exemplary image generated by the processing algorithm illustrating a fifth imaging component having a first parameter according to the present disclosure.

[0020] Fig. 7B is an exemplary image generated by the processing algorithm illustrating a fifth imaging component having a second parameter according to the present disclosure.

[0021] Fig. 8A is an exemplary image generated by the processing algorithm illustrating a sixth imaging component having a first parameter according to the present disclosure.

[0022] Fig. 8B is an exemplary image generated by the processing algorithm illustrating a sixth imaging component having a second parameter according to the present disclosure.

[0023] Fig. 9 illustrates a full scene captured by a plurality of image captured devices that use the processing algorithm according to the present disclosure.

[0024] Fig. lOAis an exemplary image generated by a prior art image processing algorithm.

[0025] Fig. 10B is an exemplary image generated by the processing algorithm according to the present disclosure.

[0026] Fig. HAis an exemplary image generated by a prior art image processing algorithm.

[0027] Fig. 1 IB is an exemplary image generated by the processing algorithm according to the present disclosure.

[0028] Fig. 12 is a block diagram representing hardware for capturing and processing images according to the present disclosure.

[0029] Throughout the figures, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the subject disclosure will now be described indetail with reference to the figures, it is done so in connection with the illustrative exemplary embodiments. It is intended that changes and modifications can be made to the described exemplary embodiments without departing from the true scope and spirit of the subject disclosure as defined by the appended claims.DESCRIPTION OF THE EMBODIMENTS

[0030] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It is to be noted that the following exemplary embodiment is merely one example for implementing the present disclosure and can be appropriately modified or changed depending on individual constructions and various conditions of apparatuses to which the present disclosure is applied. Thus, the present disclosure is in no way limited to the following exemplary embodiment and, according to the Figures and embodiments described below, embodiments described can be applied / performed in situations other than the situations described below as examples. Further, where more than one embodiment is described, each embodiment can be combined with one another unless explicitly stated otherwise. This includes the ability to substitute various steps and functionality between embodiments as one skilled in the art would see fit.

[0031] The present disclosure provides an improved method and system for compressing and transmitting volumetric video data from a plurality of image capture devices (cameras) that are arranged around one or more subjects and which captures images of the subjects at from various positions and orientations. The plurality of captured images from the plurality of image capture devices are then combined to generate a three-dimensional image representing the volumetric image, the sequence of which in time, forms volumetric video image data. Because volumetric image data represents a 3D image, streaming high quality 3D image data, particularly associated with large scenes with a plurality of subjects (e.g. asporting even such as a basketball game), using conventional 3D data compression techniques that makes use of a mesh-based approach is challenging. Specifically, the amount of bandwidth needed to transmit the data is large and this limitation requires significant compression of the meshes for it to even be possible. This significant compression of the meshes results in a loss of quality of the 3D image when it is reconstructed (e.g. decompressed) for playback. This loss of quality is due to the way that a mesh is represented which makes compression without loss of quality particularly difficult. Moreover, this lower quality is particularly visible when the 3D data includes multiple subjects.

[0032] The present disclosure provides a method for generating, from a plurality of 2D images captured by a plurality of image capture devices, volumetric video data representing a three dimensional image of a given scene captured at a given time, and using that 3D image to generate a plurality of groupings of data that have similar characteristics and / or other information, which can then be compressed for transmission to an external apparatus (e.g. mobile phone, computer, tablet, etc.) where these groupings are reconstructed into the original three dimensional image for viewing by a user. In certain instances, the scene being captured may include one or more individuals positioned at an image capture location and surrounded by the image capture devices. In other instances, the one or more individuals positioned in an image capture region may be holding or otherwise making use of one or more objects that are captured simultaneously with images of the individuals themselves.

[0033] An exemplary algorithm according to the present disclosure is described in Fig. 1. This algorithm represents computer executable instructions (e.g. one or more programs) that are stored in a memory and are executed by one or more processors of a single or plurality of computing devices such as information processing apparatuses and / or servers. In certain embodiments, aspects of this algorithm may be performed in a particular device such as an image capture device. In other embodiments, aspects of this algorithm may be performed inan edge device such as a server that performs one or more preprocessing operations described herein that modify data in original form to generate intermediate data which is then further processed for compression and transmission to external devices for decompression and viewing thereof.

[0034] According to the algorithm described in Fig. 1, a first image capture step is performed in SI 02 in order to obtain a plurality of images of a single scene (e.g. image capture location) from a plurality of different positions and orientations in order to generate 3D image data from the plurality of captured 2D images.

[0035] Because the 3D volumetric video data is generated from a plurality of 2D images, an image capture system is provided at a given image capture location. The image capture system captures a plurality of 2D images that include one or both still images and video images of one or more subjects and their particular location. These plurality of images are used to generate the 3D image data. In one embodiment, the image capture system uses a plurality of cameras that capture one or more target subjects at a plurality of angles. In this embodiment, each of the plurality of cameras are disposed at known locations and orientations that surround the image capture region. Each image capture device that is part of the image capture system includes a capture resolution at each of these vicwpoints / anglcs sufficient to effectively capture the texture of the subject. In one embodiment, the image capture system includes up to 100 image capture devices positioned at various angles and at various heights in order to capture a plurality of angles and viewpoints of the capture area. For example, if the image capture region is an arena floor such as a basketball court, the number of image capture devices disposed about this type of image capture region may be 100 or more. In another embodiment, if the image capture region includes one or a few individuals, the number of image capture devices may be 20 or less. This is merely exemplary, and the number of image capture devices included in the image capture system depends on the size and shape and location of the image capture area. For example, in a casewhere the image capture area is a basketball court, the number of cameras included in the image capture system is significantly more than if the image capture area is a karate studio which is a much smaller scale thus requiring less image capture devices than a professional sports stadium / arena. While not all image capture devices in the system need to capture the same part of the entire volume to be captured, but the plurality of cameras must sufficiently capture the entire volume even if a single camera only captures a part of the entire volume. In operation, image data that is captured at a same time by each of the plurality of image capture devices is used in order to provide sufficient data to generate the three-dimensional volumetric video data image that will be transmitted and rendered on a display device.

[0036] In one exemplary embodiment, an image capturing system 200 according to the present disclosure may be deployed with a plurality of image capture devices 202a - 202k may capture an image capture region 204 illustrated by the arc 204 having dashed lines. Within the image capture region 204, in this example, a playing floor and respective overhead volume of a basketball court including one or more imaging targets 206 (e.g. individuals and / or objects) in a sports arena is being captured. This exemplary use case is depicted in Fig. 2.

[0037] When capturing images for the puiposc of generating volumetric video of imaging targets, individual image capture devices, referenced generically herein using reference numeral 202, are positioned at known locations to capture images from many possible imaging viewpoints such that the two dimensional image data captured can be processed to generate three dimensional images of the imaging target(s). In this embodiment the image capture devices 202a, 202b, 202j and 202k represent a first set of image capture devices distributed around a first area of the imaging region 204. In this embodiment, the first area of the imaging region 204 is a floor level of a basketball court with a subset of these cameras capturing each end of the court in sufficient resolution. In one embodiment, the resolution at which images are captured are the same by each image capture device. In anotherembodiment, the resolution at which images are captured may be variable and depend on one or more of the type of image capture device and it’s available resolution and / or a position of the respective image capture device. The imaging system 200 of Fig. 2 depicts a second set of image capture device positioned to capture images of the image capture region 204 from a second region having an image capturing perspective and orientation different from the perspective and orientation captured by the first set of image capture devices. In this example, the second set of image capture devices are disposed in upper levels of the arena and include image captured devices 202c, 202g and 202i and provides a more top-down angular view of the image capture region 204 that includes the imaging target(s) 206 (e.g. players and action). A third set of image capture devices positioned to capture images of the image capture region 204 from a third region having an image capturing perspective and orientation different from the perspective and orientation captured by both the first set of image capture devices and second set of imaging capture devices. In this example, the third set of image capture devices are disposed directly overheard of the image capture region 204 provides a downward view of the image capture region 204 that includes the imaging target(s) 206 (e.g. players and action). In this example, the third set image capture devices 202d, 202c, 202f and 202h and arc positioned extending downward from the ceiling of the arena looking straight down on the court to sufficiently capture the top of the players.

[0038] As shown in Fig. 2, the number of imaging capturing devices 202, the described sets of image capture devices and their capture regions and the nature and type of image capturing regions 204 are described for purposes of example only and are intended to facilitate the understanding of how the system according to the present disclosure operates.

[0039] The positioning of image capture devices 202 in the image capture system 200 are fixed and known such that image data (e.g. information) being captured by each respective image capture device includes spatial information enabling the capture image data to bespatially understood such that portions of the captured images (2D images) can be used to create the 3D image data. The physical position of each image capture device (e.g. camera(s)) is known as the extrinsic camera parameters. Additionally, the optical characteristics of the cameras are known (or can be determined via a calibration process). These parameters such as optical center, focal length, and lens distortion are known as intrinsic parameters of the camera.

[0040] In some embodiments the image captured devices 202 are synchronized such that they acquire images at a same time and at the same frame rate ensuring that each image captured from each of the plurality of cameras is taken of the exact same moment. However, other embodiments that consider volume reconstraction as both a spatial and temporal process may handle asynchronously captured images. Other embodiments time the capture of the cameras to coincide with the frequency of the arena illumination to capture images at certain points on the periodic fluctuation of the illumination (e.g. fluctuation due to AC power or other electrical or physical processes involved in the illumination subsystem).

[0041] To reduce the amount of data collected and stored, some embodiments of the capture system 200 provide pcr-camcra processing (sometimes on edge, e.g. near the camera device) to mask out image background and only provide segmented crops or masked frames containing the subject of interest (for example segmenting the basketball players on the court from the background and floor). This can greatly reduce the amount of data needed to be transmitted per frame from the plurality of capture devices to a central processing center where further processing (described below) may be performed to reconstruct the captured volume in a way that can be effectively and efficiently transmitted to the end-user (viewer) of the volumetric content as described below.

[0042] The images being captured by the plurality of image capture devices 202 are transmitted to one or more servers 208 that store the processing algorithm described herein.The one or more servers 208 includes instructions for coordinating and causing image capture operations to be performed as well as receiving and parsing captured image data from each of the plurality of image captured devices 202. The captured image data received by the one or more servers each represent different partial views of the image capture region 204 having spatial information associated therewith which is obtained based on the known position and orientation of the respective image capture device that captured the respective partial view of the image capture region. As will be described hereinafter, the one or more servers 208 use the received captured partial views of the image capture region 204 to generate three-dimensional volumetric video data representative of the image capture region 204.

[0043] Turning back to Fig. 2, step 104 of the algorithm according to the present disclosure is executed. Step 104 represents a reconstruction algorithm that is performed that obtains all of the images from the image capture devices 202 that have been captured in order to generate a plurality of groups of data that include images from one or more of the images captured devices of the image capture system 200.

[0044] The groups of image data generated are gaussian splats that are comprised of a plurality of pieces of image data captured by one or more of the image capture devices 202 of the image capture system 200. Gaussian Splatting is a method for creating 3D visual representations by using small, fuzzy dots (e.g. pieces of information) called "gaussians" to build up a detailed picture of one or more aspects of a 3D image. Each piece of image data is combined into a single group (e.g. splat) and includes color information and light information. Because there is a plurality of image capture devices 202 capturing images at a same time from different perspectives, the pieces of image data can overlap one another until there is sufficient pieces of image data that are able to be reconstructed to create a 3D image of the scene. Each of the individual pieces of image data (gaussians), when combined into the group represent tiny, overlapping spots of color and light. Using the gaussians isanalogous to painting an image, not with precise brushstrokes, but overlapping spots of color and light. Each gaussian is a blob that represents a piece of the scene and includes one or more properties including position, size, and intensity. The gaussians of partial image data are splatted (e.g. grouped) by creating a 2D representation of the plurality of gaussians for a given image. In the case of video image data, these are performed on a frame-by-frame basis.

[0045] These gaussians can be layered and adjusted to reconstruct highly realistic scenes when viewed from different angles. In other words, this digital image data representation is computed and processed to generate image data layers and represents depth for use in generating the 3D image. An analogy is an artist layering paint to create depth. However, because each gaussian represents such a tiny piece of the image, computation is required to generate the layering of the gaussians. This technique is particularly useful in areas like computer graphics, augmented reality, and virtual reality, where rendering realistic images quickly and efficiently is important. Using a plurality of images of an image capture area 204 and generating one or more gaussians advantageously provides high quality 3D data with a relatively small number of splats (e.g. groups) which allows for higher compression without image quality loss.

[0046] The generation of gaussian for each splat and the number of gaussian splats corresponding to a particular image scene is generated using a trained machine learning model. The present disclosure makes use of a modified version of the Spatfacto model. The modifications occur during the training process required to train the machine learning model to process image data into gaussian splats. However, this is merely exemplary and any module that performs similar processing to generate gaussian in the described manner may be used.

[0047] According to the present disclosure the model used to generate the gaussian splats is based on optimization of a collection of “splats” (tiny 3D blobs) to accurately represent ascene captured in a plurality of 2D images by a plurality of image capture devices. The first step in training the machine learning model that ultimately is used to generate the gaussian splats is an initialization step. In this initialization step, an initial set of splats are used. The initial splats may be derived from a 3D point cloud (e.g. an image scan of a scene) or an approximation of the desired geometry for the 3D output. Each of the initial splats have one or more splat properties including, but not limited to position information, size information, orientation information, color information, lighting information, etc.

[0048] In a second training step for training the machine learning model, training is performed using a custom initialization that includes a point cloud. The model being initialized with a point cloud would use a sparse point cloud reproduction algorithm to obtain reconstruction point cloud information. In one embodiment, because the image capture system is defined and covers a known image capture region, an image capture region specific point cloud grid that covers the entire image capture region and its volume is used. In the training process that makes use of the image capture specific point cloud region, portions of the point cloud that intersect with subjects within the image capture region stay and transform and portions that correspond to empty space disappear and not used.

[0049] In a third training step, iterative projection and comparison operations arc performed whereby a particular scene is rendered from various camera angles by projecting the splats onto 2D images and these renderings are compared to the plurality of images used as input data using a loss function which measures well the splats would reproduce the input image(s).

[0050] In a fourth training step, an optimization operation is performed to optimize one or more model parameters for the model which will then be used in real-time operation on input images obtained from the plurality of image capture devices 202 to generate the gaussians for a given image capture region 204. For a given set of input images, a random number of gaussian splats are generated that correspond to whatever subjects (e.g. imagingtarget(s) 206) are present in the image capture region 204. These get rendered and compared to a true image and then, depending on a closeness value, parameters are modified such as position, size, number etc. and then re-rendered and compared to the true image and that continues until the best parameters for a given type of image are determined. In one embodiment, the optimization operation includes adjusting one or more properties of each splat (e.g. position, size, etc.) to minimize the difference between the rendered images and the input data using, for example, gradient-based optimization techniques. This optimization operation is iteratively performed to repeat the projection and optimization over made iterations thereby gradually improving the fidelity of the scene.

[0051] In another embodiment, optimization operations include dynamically adapting the splats based on the input image data and the determined optimization parameters. Dynamically adapting splats includes joining processing (e.g. merging) whereby two splats that are close together and represent similar properties (e.g. similar color, size and orientation) are merged into a single splat. Merging of splats is an improvement because it simplifies the representation of the scene by reducing the number of splats which improves the speed that the scene is rendered for display while maintaining accuracy in areas of smooth or uniform detail. Dynamically adapting splats also includes splitting splats into more than one splat. When a single splat struggles to represent the complexity of a region (e.g., fine textures or edges), it may be split into multiple smaller splats. Splitting allows for better resolution and detail in areas where precision is needed. The dynamic adjustment of splats is driven by the loss function which encourages the splats to evolve in ways that improve the overall quality of the rendered scene while balancing efficiency. Representing three-dimensional image using gaussian splats uses groups of pieces of image data (blocs) that are dynamically reshaped, merged and divided, as needed, to create a detailed and accurate representation of a 3D scene captured using multiple 2D image capture devices.

[0052] In another embodiment, optimization operation includes constraining a number of splats to a final splat count thereby preventing the splitting of splats to minimize a final splat count for a given scene. To be able to transmit the 3D gaussians to an external device on which it will be rendered and viewed, the file size should be below a predetermined file size threshold. In short, the file should have a small file size. One of the ways to do this is to limit the splats to a maximum number during the training process which leads to very similar quality at a much smaller size. Consider the images shown in Figs 3A and 3B which are line drawings representative of a 3D volumetric video generated according to the algorithm of the present disclosure each created with a different number of generated splats used in creation of the volumetric video.

[0053] For example, the image in Fig. 3A is generated and includes 20K splats yielding a file size of 1.3 MB while the image in Fig. 3B on the right include 50K splats yielding a file size of 3.2 MB. While the image in Fig. 3B is slightly higher quality, Fig. 3A yields an image having sufficient image quality. Because the image in Fig. 3A is generating using less number of splats as compared to the image in Fig. 3B, the data cost savings is important in both transmission and time to render the data from the splats to generate the image. Therefore, the optimization operation parameters arc constrained so that gaussian splitting operation no longer occurs after a maximum number of splats has been reached preventing future splitting and reducing the total number of splats for a given scene.

[0054] Turning back to Fig. 1, the algorithm performs a compression step in SI 06 such that compression processing is performed to compress the splats corresponding to a given scene for transmission to an external apparatus. The compression of a set of splats that represent a scene captured in the image capture region 204 is important to be able to stream this data to end users. This is particularly relevant when considering communication constraints associated network transmission speeds. For reference, Table 1 illustrates typical bandwidth requirements for various streaming platforms:Table 1

[0055] The current median download speed of internet connections, in the US, which is 240 Mbps. Therefore, in a case where the 3D data was uncompressed and included 20,000 splats per frame with a desired frame rate of 30 frames per second, the result requires throughput of substantially 312 Mbps which is calculated by 1.3 (MB per frame) * 30 (frame count) * 8 (Megabytes to Megabits) = 312 Mbps. This a quite a bit higher than even the most demanding streamable content at the time of filing.

[0056] Compression processing performed in step SI 06 reduces a size of the image frame having the determined number of splats. This compression processing is performed using a compression library that compress 3D image data structured as meshes or point clouds but which is modified to compress splat data. The modification includes tuning an amount that each individual component of a splat is compressed using a quantization which is a method to simplify floating-point numbers by converting them into lower-precision values which take up less space. Quantization tuning is performed by identifying the range of values you need to represent, e.g. 0.0 to 1.0. The selected range depends on the data being represented but is set to the minimum and maximum value of that field. The identified range is divided into a set number of steps, such as 256 steps for 8-bit precision and each value in the range is mapped to the nearest step. For example, a value of 0.53 is rounded to step 128 out of 256. Therefore, the simplified value is saved. As such, instead of storing the exact originalvalue, you store its step number, which takes up less space which can later be converted back to an approximate value close to the original. For example, consider an original range to includes values 0.0 to 1.0 and having 256 steps. As such, a value such as 0.5 gets mapped to the middle step, step 128.

[0057] By applying this quantization compression processing to a frame with a given number of splats, the components of the splat are considered, and compression is applied to each component accordingly. An individual Splat is made up of the following, and, depending on the dataset, we tune the quantization level to compress it with minimal quality loss.

[0058] A first component of a given splat are Means. Means are the JNI7. position of the center of the splat, and they determine where a given splat is positioned in 3D space. This value is sensitive to quantization in large scenes but less so in small scenes. As shown in Fig. 4, for example, the image it is quantized with 7 bits (128 possible values), and the quality of the splat degrades dramatically such that the imaging targets cannot be visualized in a meaningful way.

[0059] Another component of the splat is the scale. The scale is the size of the splat. Scale is also sensitive to quantization as illustrated in Figs. 5 A and 5B which results in visualization of edges, for example in letters and / or numbers, are impacted. This can be seen when comparing Fig. 5A which has a 4-bit scale that depicts undesirable artifacts around the letters of the B for Boston and Fig. 5B which has an 8 bit scale and yields much sharper edges.

[0060] A further component of the splats is a rotation value. Rotation can cause some very noticeable artifacts when quantized too far, however it can still be quantized such that the resulting quantization achieves a maximum error of less than 2 degrees with 7 bits (128) values is enough. Consider Fig. 6A which illustrates rotation scaled at 3 bits and Fig. 6B which illustrates rotation scaled at 7 bits whereby Fig. 6B yields a superior image.

[0061] Another component of the splat is the opacity which is a value that can undergo significant compression without introducing artifacts that would degrade the image being rendered on the external apparatus. Fig. 7A illustrates opacity level having a 2-bit compression whereas Fig. 7B illustrates the same image compressed at 16 bit. There is no noticeable difference indicating that this component can be greatly compressed to reduce the size of the image frame.

[0062] Another component of the splat is a color information. Each splat has color expressed as RGB values. While this can be quantized, human perception is very sensitive to color changes so the compression of the color information can only be quantized so much before it impacts the viewing quality of the rendered 3D image data on an external apparatus. In Fig. 8A, a color values are quantized using a 4-bit quantization. This yields splotchy color as indicated by each of the imaging targets having hashing. However, in Fig. 8B, the color values are quantized using 6-bit quantization and includes color that is closer to the original captured content illustrated by the hashing only being present at locations indicating that human perception would perceive the proper color for the image frame being rendered on the external device.

[0063] To determine exactly what quantization is used in the compression step S106, a grid search is performed over an initial set of quantization levels for each image and the determined quantization is one that is visually closest as determined by image data values in the image and which has the smallest file size (e.g. file size below a threshold file size). Applying this compression technique allows for the splats to be compressed down substantially 200 KB per splat for 3D data when a the number of splats to be used is limited to 20K splats. This results in the bandwidth requirement being 0.2 (MB per frame) * 30 (frames per second) * 8 (Megabytes to Megabits) = 48 Mbps. This is in line with current bandwidth requirements for streaming of next generation immersive content.

[0064] Turning back to Fig. 1, once the image frame includes the desired number of splats and compressed according to the above operations in steps S104 and S106, image frame data is transmitted via a communication network to one or more external devices in step SI 08. Executing on each external device is a playback application that receives the compressed image frame data including the splats representative of the captured image region, decompresses the image frames, in real-time, and causes the 3D image data to be displayed on a display of the external device. In one embodiment, the external apparatus is a mobile phone. In another embodiment, the external apparatus is a head mounted display device (e.g. Apple Vision Pro). In order to playback the data, the playback application performs operations including decompressing the compressed splat, rendering an the environment built from a traditional mesh and saved depth map and render the splats over the environment created by the traditional mesh.

[0065] Decompression is performed with DracoSwift which handles real time decompression of splat data as well as removing the quantization. Rendering of the environment is done in Metal using Apple libraries and their MetalKit framework. Rendering meshes using MetalKit is a standard practice and the rendering of the splats is done in Metal using MctalSplattcr which is modified to take, as input, the depth map from MetalKit and only render splats where an object is not occluding the imaging target as can be seen in Fig. 9 where the basket occludes the players.

[0066] In greater detail, the receipt of the compressed image frames for decompression and rendering for 3D volumetric video display on a mobile device includes the following operations. Each splat is encoded using to compress the following characteristics for each splat:MeansScaleRotationColors

[0067] In decompressing the encoded frame, a same library that was used to encode these values is used to read the values for each of these characteristic components. Prior to rendering each splat only the rotation parameter needs to be modified because, to compress splats the rotation parameter is encoded as Euler angles instead of quaternions. This allows rotation to be represented with 3 numbers (pitch, yaw, roll) instead of 4 numbers (scalar part + 3 imaginary parts) thereby minimizing space requirements. This conversion processing is executed using the following formulas:1. Compute the half angles :o cl = cos(roll / 2)si = sin(roll / 2)(Roll: rotation around the X-axis)o c2 = cos(pitch / 2)s2 = sin(pitch / 2)(Pitch: rotation around the Y-axis)o c3 = cos(yaw / 2)s3 = sin(yaw / 2)(Yaw: rotation around the Z-axis)2. Calculate the quaternion components:O w = cl * c2 * c3 + si * s2 * s3o x = sl * c2 * c3 - cl * s2 * s3o y = cl * s2 * c3 + sl * c2 * s3o z = cl * c2 * s3 - si * s2 * c33. Combine into the quaternion:o q = (w, x, y, z)

[0068] At this point, the splat is decompressed, and for each frame to be played back there is a set of splats to render. To playback the volumetric video, two parallel processes are running. The first is responsible for fetching and decompressing a frame into memory so it's ready to be displayed, the second is used to display the frame and ensure we are showing the right frame at the right time.

[0069] The first fetching and decompressing processing checks if the frame queue, which represents frames that are ready to be shown, has sufficient space to support new frames being added thereto. The frame queue will store a predetermined number of frames therein as all the data is stored in memory and there is a need to limit how much data is stored therein. The compressed splats are read from the frame queue and decompressed into a decompressed splat data. The decompressed splat data is placed into the frame queue in the correct position based on a frame index that provides a correct order that frames are to be shown and, decompression occurs via multiple processing threads, decompressed frames from different processing threads may become out of sync. To determine the correct position we look at the frame number and ensure it is in order in the queue.

[0070] The second process plays back the frames in the queue. This process runs every time the screen wants to refresh and checks if enough time has elapsed to show a new frame. For example, a screen may run at 120 fps but splat data may be only at 30 fps, so in that case we would only update the splat to render every 4 frames. When a predetermined time has elapsed to show a new frame, switching the splat to render from the current splat being rendered to a next splat in the frame queue is performed and the next splat is removed from the frame queue. If the predetermined time has not elapsed, the current splat remains the one being decompressed and rendered. The splat is rendered from the current camera viewpoint. This is done using MetalSplatter which a library that uses camera position information and generates a camera viewpoint and a splat as input and renders it to the screen.

[0071] Gaussian Splats provide an improvement by providing image data to be rendered which is compressed with less quality loss as well as showing less artifacts in the rendering as compared to high quality reconstructed meshes. The improvement is visualized in exemplary images in Figs. 10A and 10B where the image in Fig. 10A is volumetric video rendered based on compressed meshes and Fig. 10B is the same frame of volumetric video rendered using the gaussian splats of the present disclosure. In Fig. 10B, the image quality is improved as compared to Fig. 10A which includes artifacts that are pronounced and visualized via the less-defined boarder of the imaging target. While Fig. 10B may include a predetermined amount of blurriness on the edges, Fig. 10A has more noticeable artifacts. Gaussian splats provide higher quality especially when compressed, this is where we see the biggest advantages compared to a mesh as can be seen when considering Fig. HA compared to Fig. 11B. Fig. HA is an image frame rendered based on a compressed mesh and Fig. 1 IB is an equivalently compressed splat on the right. In the compressed splat generated according to the algorithm described herein, the harsh edges of the mesh are gone and the overall texture of the splat looks much better.

[0072] A block diagram illustrating the hardware described hereinabove that performs the described operations is shown in Fig. 12. Fig. 12 illustrates the hardware of the 3D image processing system described above. The information processing apparatus (e.g. server 208) (shown in Fig. 2) is selectively connected to a communication network 210 such as a wired network, a wireless network, a LAN, a WAN, a MAN, and a PAN. A plurality of image capture devices 202n, where n is an integer greater than 2, is communicatively connected to the information processing apparatus (e.g. server 208) and provides captured image data to the information processing apparatus (e.g. server 208) for processing according to the algorithms described herein to output, via an external apparatus 212, the rendered volumetric video data. While not shown, the plurality of image capture devices 202n may not be in direct communication with the server but also may be connected via the same ordifferent communication network. Also, in some embodiments communicate occurs via other wired or wireless channels. The information processing apparatus (e.g. server 208) includes one or more processors 209, one or more I / O components 213, and storage 211. Also, the hardware components communicate via one or more buses or other electrical connections. Examples of buses include a universal serial bus (USB), an IEEE 1394 bus, a PCI bus, an Accelerated Graphics Port (AGP) bus, a Serial AT Attachment (SATA) bus, and a Small Computer System Interface (SCSI) bus.

[0073] The one or more processors 209 include one or more central processing units (CPUs), which may include one or more microprocessors (e.g., a single core microprocessor, a multi-core microprocessor); one or more graphics processing units (GPUs); one or more tensor processing units (TPUs); one or more application-specific integrated circuits (ASICs); one or more field-programmable-gate arrays (FPGAs); one or more digital signal processors (DSPs); or other electronic circuitry (e.g., other integrated circuits). The I / O components 213 include communication components (e.g., a graphics card, a network-interface controller) that communicate with the head mount display apparatus, the network and other input or output devices (not illustrated), which may include a keyboard, a mouse, a printing device, a touch screen, a light pen, an optical-storage device, a scanner, a microphone, a drive, and a game controller (e.g., a joystick, a gamepad).

[0074] The storage 211 includes one or more computer- readable storage media. As used herein, a computer-readable storage medium includes an article of manufacture, for example a magnetic disk (e.g., a floppy disk, a hard disk), an optical disc (e.g., a CD, a DVD, a Blu-ray), a magneto-optical disk, magnetic tape, and semiconductor memory (e.g., a non-volatile memory card, flash memory, a solid-state drive, SRAM, DRAM, EPROM, EEPROM). The storage, which may include both ROM and RAM, can store computer-readable data or computer-executable instractions.

[0075] The information processing apparatus (e.g. server 208) includes splat processing module 215 (or application) that performs the functions described hereinabove. A module includes logic, computer-readable data, or computer-executable instructions. In the embodiment shown herein, the modules are implemented in software (e.g., Assembly, C, C++, C#, Java, BASIC, Perl, Visual Basic, Python, Swift). However, in some embodiments, the modules are implemented in hardware (e.g., customized circuitry) or, alternatively, a combination of software and hardware. When the modules are implemented, at least in part, in software, then the software can be stored in the storage.

[0076] At least some of the above-described devices, systems, and methods can be implemented, at least in part, by providing one or more computer-readable media that contain computer-executable instructions for realizing the above-described operations to one or more computing devices that are configured to read and execute the computerexecutable instructions. The systems or devices perform the operations of the abovedescribed embodiments when executing the computer-executable instructions. Also, an operating system on the one or more systems or devices may implement at least some of the operations of the above-described embodiments.

[0077] Furthermore, some embodiments use one or more functional units to implement the above-described devices, systems, and methods. The functional units may be implemented in only hardware (e.g., customized circuitry) or in a combination of software and hardware (e.g., a microprocessor that executes software).

[0078] Additionally, some embodiments of the devices, systems, and methods combine features from two or more of the embodiments that are described herein. Also, as used herein, the conjunction “or” generally refers to an inclusive “or,” though “or” may refer to an exclusive “or” if expressly indicated or if the context indicates that the “or” must be an exclusive “or.”

[0079] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

Claims

ClaimsWe claim,1. An image processing method comprising :obtaining a plurality of images of an image capture region captured by a plurality of image capture devices location at different positions around the image capture region;generating a predetermined number of gaussians using the plurality of captured images based on points in a 3D representation of the image capture region;generating an initial number splats for the given scene, wherein each splat includes certain number of the predetermined gaussians;optimizing the number of splats using at least one optimization parameter; compressing the number of optimized splats into a file; andtransmitting the compressed filed to at least one external apparatus that decompresses and displays the compressed file to view, one the external apparatus, the 3D image of the image capture region.

2. The method according to claim 1 , further comprisingrendering a scene corresponding to the image capture region using images by projecting the generated splats onto 2D images; andcomparing the renderings to the set of obtained plurality of images using a loss function;adjusting at least one property of the splat to minimize a difference between the rendered images and the obtained plurality of images.

3. The method according to any of the above claims, further comprisingdynamically adapting the initial number of splats to a target number of splats based on at least one property of each of the initial splats.

4. The method according to any of the above claims, wherein dynamically adapting the initial number of splats includes merging two closely related splats into a single splat.

5. The method according to any of the above claims, wherein dynamically adapting the initial number of splats includes separating two splats determined to not be representative of a particular region of the original image.

6. The method according to any of the above claims, further comprising limiting a target number of splats to be generated thereby minimizing size of an image frame.

7. The method according to any of the above claims, further comprising compressing one or more component of each respective splat by quantizing values of each of the one or more components.

8. A server comprisingone or more memories storing instructions andone or more processors that, upon execution of the stored instructions, is configured to:obtain a plurality of images of an image capture region captured by a plurality of image capture devices location at different positions around the image capture region;generate a predetermined number of gaussians using the plurality of captured images based on points in a 3D representation of the image capture region;generate an initial number splats for the given scene, wherein each splat includes certain number of the predetermined gaussians;optimize the number of splats using at least one optimization parameter; compress the number of optimized splats into a file; andtransmit the compressed filed to at least one external apparatus that decompresses and displays the compressed file to view, one the external apparatus, the 3D image of the image capture region.

9. The server according to claim 8, wherein execution of the stored instructions further configures to one or more processors torender a scene corresponding to the image capture region using images by projecting the generated splats onto 2D images; andcompare the renderings to the set of obtained plurality of images using a loss function;adjust at least one property of the splat to minimize a difference between the rendered images and the obtained plurality of images.

10. The server according to any of claims 8 and 9, wherein execution of the stored instructions further configures to one or more processors todynamically adapt the initial number of splats to a target number of splats based on at least one property of each of the initial splats.

11. The server according to claim 10, wherein execution of the stored instructions further configures to one or more processors to dynamically adapt the initial number of splats by merging two closely related splats into a single splat.

12. The server according to claim 10, wherein execution of the stored instructions further configures to one or more processors to dynamically adapt the initial number of splats by separating two splats determined to not be representative of a particular region of the original image.

13. The server according to any of claims 8 - 12, wherein execution of the stored instructions further configures to one or more processors to minimize a size of an image frame by limiting a target number of splats to be generated.

14. The server according to any of claims 8 - 12, wherein execution of the stored instructions further configures to one or more processors to compress one or more component of each respective splat by quantizing values of each of the one or more components.

15. A system comprising:a plurality of image capture devices that capture images of an image capture region a server according to any of claims 8 - 14.

16. The system according to claim 15, further comprising an external apparatus that receives, via a communication network, image generated by the server according to any of claims 8 - 14, and causes the image to be displayed on a display of the external apparatus.

17. A non- transitory computer readable storage medium storing instructions that, upon execution by one or more processors of a computing device, causes the computing device to execute the image processing method according to any of claims 1 - 7.

Images