Real-time rendering method using point cloud data

Abstract

Disclosed is a real-time rendering method using point cloud data, which is performed by a computing apparatus of the present disclosure. The method may comprise the steps of: acquiring a plurality of frames by capturing a dynamic scene; converting, for each frame, geometric information of a target object into point cloud data; performing interpolation processing between two frames by using the point cloud data; and rendering on the basis of the interpolated data.

Description

Real-time rendering method using point cloud data

[0001] The present invention relates to a real-time rendering method, and more specifically, to a real-time rendering method utilizing point cloud data.

[0002] Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) are attracting attention as technologies that generate and render 3D scenes by creating views of uncaptured angles from 2D images that observe a specific object or scene from multiple viewpoints.

[0003] NeRF is a method in which a neural network learns the radiance field of 3D space to predict the color and density for specific coordinates in 3D space, and 3DGS is a method in which points in 3D space are represented by a Gaussian distribution, and the location, size, direction, color, etc. of the Gaussians are learned to directly construct the scene.

[0004] In-Camera Visual Effects (ICVFX) is produced by projecting a virtual environment onto an LED wall in real time, with actors performing in front of it. In other words, ICVFX is a technology that combines virtual backgrounds with physical sets for filming in real time. Because visual effects are implemented in real time during the filming phase, it reduces post-production time and allows actors to perform as if they were in a real environment.

[0005] NeRF provides high-quality results but has limitations for real-time rendering tasks due to long rendering times. 3DGS enables real-time rendering but is optimized for static scenes, making it difficult to apply to ICVFX environments that require continuous scenes.

[0006] Korean Patent No. 10-2688669 (Registration Date: July 22, 2024) discloses a method for obtaining an omnidirectional 3D mesh model sequence from a mixed reality-based multi-view video sequence.

[0007] The present disclosure aims to optimize the continuous rendering of a dynamic scene that changes over time.

[0008] Meanwhile, the technical problem that the present disclosure aims to solve is not limited to the technical problem mentioned above, and various technical problems may be included within the scope obvious to a person skilled in the art from the contents described below.

[0009] A real-time rendering method utilizing point cloud data is disclosed according to an embodiment of the present disclosure for realizing the aforementioned objectives. The method may include the steps of capturing a dynamic scene to acquire a plurality of frames, converting geometric information of a target object into point cloud data for each frame, performing interpolation processing between two frames using the point cloud data, and rendering based on the interpolated data.

[0010] In one embodiment, the step of capturing the dynamic scene to acquire the plurality of frames may include the step of acquiring synchronized data from the plurality of cameras and the step of classifying the synchronized data into the same frame number.

[0011] In one embodiment, the step of generating the geometric information of the target object into the point cloud data for each frame may include the step of generating the Gaussian point cloud by converting the geometric information of the target object for each frame into a Gaussian distribution, and the step of generating the Gaussian point cloud data by optimizing the parameters of each Gaussian point.

[0012] In one embodiment, the method further includes the step of converting the Gaussian point cloud data of each frame into a feature vector, and the step of performing interpolation processing between the two frames may include the step of generating intermediate data between the two frames using the converted feature vector.

[0013] In one embodiment, the step of rendering based on the interpolated data may include the step of generating new Gaussian point cloud data based on the intermediate data and the step of rendering using the new Gaussian point cloud data.

[0014] In one embodiment, the method further includes the step of caching the transformed feature vector, and the step of performing interpolation processing between the two frames may include the step of generating intermediate data between the two frames using the cached feature vector.

[0015] In one embodiment, the step of generating intermediate data between the two frames using the cached feature vector may include the step of identifying identical Gaussian points between the two frames and the step of generating intermediate data between the two frames based on the identification result.

[0016] In one embodiment, the step of generating intermediate data between the two frames based on the identification result may further include the step of matching Gaussian points determined to be identical according to the identification result, and, if there are Gaussian points that are not matched to each other, the step of generating a new Gaussian point or gradually removing the Gaussian point.

[0017] In one embodiment, the method may further include the steps of aligning point clouds of two frames, comparing the aligned point clouds to evaluate similarity between the two frames, and caching point cloud data of the frame determined to have high similarity based on the evaluation result.

[0018] In one embodiment, the step of rendering based on the interpolated data may include the step of rendering by combining the interpolated data and the cached point cloud data.

[0019] A computer program stored on a computer-readable storage medium is disclosed in accordance with an embodiment of the present disclosure for realizing the aforementioned objectives. When the computer program is executed by at least one processor, the at least one processor is made to perform the following operations, wherein the operations may include the operation of capturing a dynamic scene to acquire a plurality of frames, the operation of converting geometric information of a target object into point cloud data for each frame, the operation of performing interpolation processing between two frames using the point cloud data, and the operation of rendering based on the interpolated data.

[0020] In one embodiment, the operation of capturing the dynamic scene to acquire the plurality of frames may include the operation of acquiring synchronized data from the plurality of cameras and the operation of classifying the synchronized data into the same frame number.

[0021] In one embodiment, the operation of generating the geometric information of the target object into the point cloud data for each frame may include the operation of generating the geometric information of the target object into a Gaussian point cloud by converting it into a Gaussian distribution for each frame, and the operation of generating the Gaussian point cloud data by optimizing the parameters of each Gaussian point.

[0022] In one embodiment, the operation further includes an operation of converting the Gaussian point cloud data of each frame into a feature vector, and the operation of performing interpolation processing between the two frames may include an operation of generating intermediate data between the two frames using the converted feature vector.

[0023] In one embodiment, the operation of rendering based on the interpolated data may include the operation of generating new Gaussian point cloud data based on the intermediate data and the operation of rendering using the new Gaussian point cloud data.

[0024] A computing device is disclosed according to one embodiment of the present disclosure for realizing the aforementioned objectives. The device comprises at least one processor and a memory, wherein the at least one processor may be configured to capture a dynamic scene to acquire a plurality of frames, convert geometric information of a target object into point cloud data for each frame, perform interpolation processing between two frames using the point cloud data, and render based on the interpolated data.

[0025] In one embodiment, the at least one processor may be further configured to acquire synchronized data from a plurality of cameras and classify the synchronized data into the same frame number.

[0026] In one embodiment, the at least one processor may be further configured to convert the geometric information of the target object into a Gaussian distribution for each frame to generate a Gaussian point cloud, and to generate Gaussian point cloud data by optimizing the parameters of each Gaussian point.

[0027] In one embodiment, the at least one processor may be further configured to convert the Gaussian point cloud data of each frame into a feature vector and to generate intermediate data between the two frames using the converted feature vector.

[0028] In one embodiment, the at least one processor may be further configured to generate new Gaussian point cloud data based on the intermediate data and to render using the new Gaussian point cloud data.

[0029] The present disclosure can optimize the real-time rendering of dynamic scenes, thereby enabling the natural handling of changes in time and space. Consequently, it has the effect of enabling natural scene transitions.

[0030] Meanwhile, the effects of the present disclosure are not limited to those mentioned above, and various effects may be included within the scope obvious to a person skilled in the art from the contents described below.

[0031] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0032] FIG. 2 is a schematic diagram showing a neural network according to one embodiment of the present disclosure.

[0033] FIG. 3 is a flowchart for explaining a real-time rendering method using point cloud data according to one embodiment of the present disclosure.

[0034] FIG. 4 is a drawing for explaining a data collection method according to one embodiment of the present disclosure.

[0035] FIG. 5 is a flowchart for explaining a data cleaning method according to one embodiment of the present disclosure.

[0036] FIG. 6 is a conceptual diagram for explaining Gaussian splatting according to one embodiment of the present disclosure.

[0037] FIG. 7 is a block diagram illustrating a data processing method according to one embodiment of the present disclosure.

[0038] FIG. 8 is a flowchart illustrating a frame-based Gaussian splatting technique according to one embodiment of the present disclosure.

[0039] FIG. 9 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0040] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0041] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0042] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0043] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0044] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0045] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0046] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0047] In the present disclosure, network functions, artificial neural networks, and neural networks may be used interchangeably.

[0048]

[0049] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0050] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0051] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0052] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0053] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0054] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0055] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0056] In addition, the network unit (150) presented in this specification may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0057] In the present disclosure, the network unit (150) can be configured regardless of the communication mode, such as wired and wireless, and can be configured as various communication networks such as a Local Area Network (LAN), a Personal Area Network (PAN), and a Wide Area Network (WAN). In addition, the network may be a known World Wide Web (WWW) and may utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

[0058] The technologies described in this specification can be used not only in the networks mentioned above but also in other networks.

[0059]

[0060] FIG. 2 is a schematic diagram showing a neural network according to one embodiment of the present disclosure.

[0061] Throughout this disclosure, neural network, neural network model, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

[0062] In this case, within the neural network model, one or more nodes connected via links can form a relative relationship between an input node and an output node. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the relationship between an input node and an output node can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0063] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight (at this time, parameters and weights may be used interchangeably throughout this disclosure). The weight may be variable and may be varied by a user or an algorithm to enable the neural network model to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value may be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0064] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0065] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0066] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0067] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0068] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network allows for the identification of latent structures in data. That is, it is possible to identify the latent structures of photos, text, videos, voice, and music (e.g., what objects are present in a photo, what the content and emotions of the text are, what the content and emotions of the voice are, etc.). Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0069] In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to the input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be placed between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrically with respect to the input layer). The autoencoder may perform non-linear dimensionality reduction. The number of input and output layers may correspond to the dimension after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may have a structure in which the number of nodes decreases as it moves away from the input layer. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (e.g., more than half of the input layer).

[0070] A neural network model including a neural network can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training of a neural network model may be a process of applying knowledge to the neural network model to perform a specific action.

[0071] Neural network models can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the model, calculating the error between the model's output and the target for the training data, and updating the weights of each node in the model by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In the case of supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in the case of unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training point is labeled with a category. Labeled training data is input into the neural network model, and the error can be calculated by comparing the model's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network model's output. The calculated error is backpropagated in the neural network model (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network model can be updated. The amount of change in the connection weights of each updated node can be determined by the learning rate. The computation of the neural network model on input data and the backpropagation of the error can constitute an epoch. The learning rate can be applied differently depending on the number of iterations of the neural network model's epoch. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0072] In the training of neural network models, training data is generally a subset of real-world data (i.e., the data intended to be processed using the trained neural network model). Consequently, there may exist epochs where errors decrease on the training data but increase on the real data. Overfitting is a phenomenon where the model learns excessively on the training data, leading to an increase in errors on real-world data. For example, a neural network model trained on yellow cats might fail to recognize cats other than yellow ones as cats, which can be considered a type of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and batch normalization can be applied.

[0073]

[0074] FIG. 3 is a flowchart for explaining a real-time rendering method using point cloud data according to one embodiment of the present disclosure.

[0075] The present disclosure relates to a method for optimizing the continuous rendering of a dynamic scene that changes over time in an ICVFX environment requiring continuous scenes. To address this, a processor may apply Gaussian splatting on a frame-by-frame basis by utilizing frame-based Gaussian splatting. Through this, the Gaussian data of each frame can be naturally synchronized. In other words, since the processor can naturally process changes in time and space, it has the effect of enabling natural scene transitions.

[0076] Referring to FIG. 3, the processor can capture a dynamic scene and acquire multiple frames (S110). The processor can acquire synchronized data from multiple cameras and classify the synchronized data into the same frame number to form a single frame.

[0077] The processor can convert the geometric information of the target object into point cloud data for each frame (S120). The processor can estimate the 3D structure of the scene in each frame and generate a 3D point cloud of the geometric information of the estimated 3D structure.

[0078] The processor can perform interpolation processing between two frames using point cloud data (S130). The processor can convert the point cloud data into a feature vector and generate intermediate data between the two frames using the feature vector.

[0079] The processor can render based on the interpolated data (S140). The processor can generate a new point cloud based on the intermediate data, which is the interpolated data, and perform rendering using the new point cloud.

[0080]

[0081] Hereinafter, embodiments of the present disclosure will be described in more detail.

[0082] 1. Data Collection

[0083] FIG. 4 is a drawing for explaining a data collection method according to one embodiment of the present disclosure.

[0084] Referring to FIG. 4, the processor can capture dynamic scenes from various angles using multiple cameras (210-260). To ensure that all cameras capture at the same moment, the multiple cameras (210-260) can be synchronized using a time code synchronization technique. For example, the time code synchronization technique may be a hardware-based synchronization that aligns the time codes of all cameras using a synchronization device such as Genlock. Alternatively, the time code synchronization technique may be a software-based synchronization that aligns the time codes recorded in the metadata of the captured data by comparing them.

[0085] The processor can acquire synchronized data from a plurality of cameras (210-260). For example, the processor can acquire first synchronized data (D1) at a first time point, second synchronized data (D2) at a second time point, third synchronized data (D3) at a third time point, and fourth synchronized data (D4) at a fourth time point. The first synchronized data (D1) may include data acquired from the first camera (210) at a first time point, data acquired from the second camera (220) at a first time point, data acquired from the third camera (230) at a first time point, data acquired from the fourth camera (240) at a first time point, data acquired from the fifth camera (250) at a first time point, and data acquired from the sixth camera (260) at a first time point.

[0086] The processor can classify synchronized data into the same frame number. For example, the processor can classify the first synchronized data (D1) into the first frame (t1), the second synchronized data (D2) into the second frame (t2), the third synchronized data (D3) into the third frame (t3), and the fourth synchronized data (D4) into the fourth frame (t4). Data classified into the same frame number can serve as a basis for accurately reproducing a single dynamic scene from multiple angles during a subsequent rendering operation. Thus, consistency in positional changes or movements occurring in the dynamic scene can be maintained.

[0087] In the present disclosure, data classified by the same frame number may be understood to constitute a single frame. That is, a single frame data may be data captured from different angles and classified by the same frame number.

[0088]

[0089] 2. Data Cleansing

[0090] FIG. 5 is a flowchart for explaining a data cleaning method according to one embodiment of the present disclosure.

[0091] Referring to FIG. 5, the processor can extract frame-by-frame Gaussian data (S210). The processor can generate 3D scene data using synchronized data constituting each frame and convert the frame-by-frame generated 3D scene data into a Gaussian distribution. The processor can optimize the Gaussian distribution to effectively represent detailed features of the scene and convert it into Gaussian point cloud data for each frame.

[0092] The processor can extract feature vectors of Gaussian data (S220). The processor can summarize Gaussian point cloud data to generate feature vectors such as location, size, and density.

[0093] The processor can determine similarity between frames (S230). The processor can evaluate similarity between two frames when the scene changes and identify repeated scenes.

[0094] The processor can perform caching optimization (S240). The processor can pre-cachise feature vectors of all frames and utilize them for interpolation and rendering. Additionally, the processor can further cache PLY data of frames determined to have high similarity and retrieve the cached frame data whenever necessary without additional computation to perform rendering tasks. Therefore, unnecessary computations can be reduced, and rendering time and resources can be saved.

[0095]

[0096] Hereinafter, the data cleaning method of the present disclosure will be described in more detail.

[0097] <Extract Gaussian Data by Frame>

[0098] The processor can generate 3D scene data using synchronized data that constitutes each frame. The processor can obtain frame-based Gaussian splatting data by converting the 3D scene data generated per frame into a Gaussian distribution.

[0099] FIG. 6 is a conceptual diagram for explaining Gaussian splatting according to one embodiment of the present disclosure.

[0100] Generally, Gaussian splatting refers to a method of representing a 3D scene using numerous Gaussian blobs. Here, a Gaussian blob refers to a spherical (or elliptical) object that spreads smoothly outward from a single point in 3D space, as illustrated in FIG. 6. Each blob may have attributes such as position, scale, color, opacity, and anisotropy. An image representing the scene can be created by projecting the Gaussian blobs existing in 3D space onto a 2D screen.

[0101] 3D Gaussian splatting transforms the 3D structure of a scene into a Gaussian distribution and can optimize the parameters of Gaussian points based on a static scene. Therefore, 3D Gaussian splatting is difficult to apply to the continuous rendering of dynamic scenes.

[0102] The frame-based Gaussian splatting of the present disclosure can convert 3D scene data generated frame by frame into a Gaussian distribution independently for each frame. That is, a new Gaussian distribution can be generated for each frame by considering objects or camera movements that change over time. Therefore, frame-based Gaussian splatting data can be applied to real-time rendering of dynamic scenes.

[0103] Specifically, the processor can generate geometric information of a target object as a 3D point cloud for each frame. The geometric information may include the contours, feature points, location information, etc. of the target object. For example, the processor can utilize Structure-from-Motion (SfM) to estimate the 3D structure of the scene in each frame and generate geometric information of the estimated 3D structure as a 3D point cloud.

[0104] The processor can convert the above 3D point cloud into a Gaussian distribution and represent it as a Gaussian point cloud. Each Gaussian point can be represented by a Multivariate Normal Distribution. The Multivariate Normal Distribution can be represented using a mean vector and a covariance matrix. Here, the mean vector can represent the central location of the Gaussian points in 3D space, and the covariance matrix can represent the magnitude, directionality, and density of the Gaussian points.

[0105] The processor can optimize parameters such as the position, size, orientation, and density of each Gaussian point. For example, by learning to reduce the distance between Gaussian points and the actual 3D point cloud, the processor can adjust the centers of the Gaussian points to align as closely as possible with the actual 3D surface. Additionally, the processor can adjust the size of the Gaussians to suit the data, using small Gaussians in areas with high detail and large Gaussians in simple areas of the scene. Furthermore, the processor can calculate normal vectors from the 3D point cloud and adjust the Gaussian covariance matrix to follow that direction, thereby ensuring that the Gaussians are positioned along the slope of the object. Moreover, the processor can adaptively control density by placing more Gaussians in important areas, such as boundaries or feature-rich regions, while removing unnecessary Gaussians.

[0106] The processor can optimize the Gaussian distribution by utilizing a loss function such as [Equation 1].

[0107] [Mathematical Formula 1]

[0108] L = L Chamfer + λ1L Size + λ2L Orientation + λ3L Density

[0109] Here, L Chamferrepresents the chamfer distance for optimizing the position of the Gaussian, and L Size is a factor for optimizing the magnitude of the Gaussian, and L Orientation is a factor for optimizing the directionality of the Gaussian, and L Density is a factor for optimizing density, and λ 1, λ 2, Each λ3 can be used as a weight value to adjust the importance of each element.

[0110] Optimized Gaussian distributions can effectively represent the detailed features of a scene. The Gaussian point cloud generated from each frame in frame-based Gaussian splatting can be converted into a PLY (polygon) file format and stored in memory on a frame-by-frame basis. Since the Gaussian point cloud maintains independence between frames, it facilitates editing or additional processing of specific frames in subsequent operations. Furthermore, the Gaussian point cloud can be directly utilized in subsequent rendering tasks, enabling efficient processing.

[0111]

[0112] <Extraction of Feature Vectors from Gaussian Data>

[0113] FIG. 7 is a block diagram illustrating a data processing method according to one embodiment of the present disclosure.

[0114] Referring to FIG. 7, the processor (310) can read the PLY file of each frame stored in memory (330) and load Gaussian point cloud data into RAM (Random Access Memory; 320). The Gaussian point cloud data of each frame can be represented as follows.

[0115] G t = {(x i , y i , z i , size i , density i )}

[0116] The processor can summarize Gaussian point cloud data and convert it into feature vectors. The processor can summarize Gaussian point cloud data to generate feature vectors such as location, magnitude, and density. The feature vector of each frame can be represented as follows.

[0117] Vt = (mean(x), mean(y), mean(z), std(size), std(density))

[0118] The feature vectors of all frames can be cached and utilized for interpolation and rendering. That is, the processor (310) stores the feature vectors in a cache (312), and when performing interpolation and rendering, the CPU (311) or GPU can retrieve them from the cache (312) to increase the computation speed.

[0119]

[0120] <Assessment of Similarity Between Frames>

[0121] The processor can evaluate the similarity between two frames and identify repeated scenes when the scene changes. At this time, to reduce errors caused by changes in camera position or scene movement, the processor can correct for differences in position and orientation between the two frames. For example, the processor can align the Gaussian point clouds of two frames by registering them using the Iterative Closest Point (ICP) algorithm.

[0122] The processor can evaluate the similarity between two frames by comparing aligned Gaussian point clouds between the two frames. For example, the processor can calculate the average distance between aligned point clouds using the Champer Distance algorithm. The calculated average distance is called the Champer distance and can be calculated as shown in [Equation 2].

[0123] [Mathematical Formula 2]

[0124]

[0125] The processor can determine that the smaller the Champer distance (d), the higher the similarity between the two data. Data with high similarity can be considered as data that can be used repeatedly. For example, if the Champer distance is below a threshold, the frame can be considered as a repeating scene.

[0126] Resources can be wasted when recalculating repetitive scenes or specific frames whenever they are needed. Therefore, PLY data for frames deemed to have high similarity can be pre-cached, allowing rendering tasks to be performed by retrieving the cached frame data without additional computation whenever required. Consequently, unnecessary computations can be reduced, saving rendering time and resources.

[0127]

[0128] <Cache Optimization>

[0129] As described above, the processor caches the feature vectors of all frames and can additionally cache PLY data for recurring scenes with high similarity. Therefore, this caching strategy can significantly reduce the amount of computation during the rendering process, thereby improving performance optimization and resource management efficiency.

[0130]

[0131] 3. Frame Interpolation Processing

[0132] If the Gaussian points of a fast-moving object undergo significant changes only within specific frames, interpolation techniques can be applied to adjust their position and size so that they change gradually across consecutive frames. This allows movements or changes occurring in dynamic scenes to flow naturally, enabling viewers to experience smooth transitions.

[0133] The processor can achieve a natural connection between frames by utilizing linear interpolation techniques to adjust the location, magnitude, and density of Gaussian distributions between two frames. The processor can generate intermediate data between two frames by utilizing cached feature vectors. Therefore, it can maintain consistency in data processing and perform efficient interpolation.

[0134] First, given the feature vectors Vt1 and Vt2 of two frames (t1, t2), the feature vector Vt of the interpolated frame t can be calculated as in [Equation 3].

[0135] [Mathematical Formula 3]

[0136] Vt = (1-α)Vt1 + αVt2, α=

[0137] Here, α represents the interpolation coefficient at time t and can have a value between 0 and 1. t1 and t2 can represent the start frame and the end frame.

[0138] The feature vector Vt of the interpolated frame t can be expressed as follows.

[0139] Vt = (mean(x), mean(y), mean(z), std(size), std(density))

[0140] The processor can identify identical Gaussian points between two frames by utilizing feature vectors (Vt1, Vt2). The processor calculates the distance between Gaussian points using a nearest neighbor matching algorithm and determines that the closest Gaussian points are identical. If the distance between Gaussian points is below a threshold, the processor determines that the Gaussian points between the two frames are identical. The processor can match the Gaussian points determined to be identical. The processor can apply a linear interpolation formula to ensure that the location, size, and density of the matched Gaussian points are smoothly connected.

[0141] Position interpolation can calculate an intermediate value using the position information of the first frame and the second frame, as shown in [Equation 4].

[0142] [Mathematical Formula 4]

[0143] mean t (x) = (1-α)·mean t1 (x) + α·mean t2 (x)

[0144] Size interpolation can be calculated as in [Equation 5].

[0145] [Mathematical Formula 5]

[0146] Std t (size) = (1-α)·std t1 (size) + α·std t2 (size)

[0147] Density interpolation can be calculated as in [Equation 6].

[0148] [Mathematical Formula 6]

[0149] Std t (density) = (1-α)·std t1 (density) + α·std t2 (density)

[0150] If the distance between Gaussian points exceeds a threshold, the processor can determine that the Gaussian points between the two frames are different. For unmatched Gaussian points, the processor can generate new Gaussian points or remove them by gradually reducing their size and density. For example, Gaussian points present only at t2 can be newly generated at t1 to maintain continuity. Alternatively, Gaussian points present only at t1 can be naturally removed by gradually reducing their size and density.

[0151] According to the embodiments of the present disclosure, cached feature vectors can be utilized as is, thereby preventing additional data processing and maximizing efficiency.

[0152]

[0153] 4. Gaussian Data Reconstruction

[0154] The processor can generate a new Gaussian point cloud based on intermediate data between two frames, which is interpolated data. The processor can convert the new Gaussian point cloud into the PLY format so that it can be used by the rendering engine.

[0155]

[0156] 5. Rendering

[0157] The processor can output reconstructed Gaussian point cloud data in real time to a display such as an LED WALL. The reconstructed Gaussian point cloud data may include Gaussian point cloud data corresponding to the interpolated data. For example, the processor can provide natural and seamless screen transitions by rendering a combination of Gaussian point cloud data corresponding to the interpolated data and Gaussian point cloud data corresponding to the repeating scene data.

[0158]

[0159] FIG. 8 is a flowchart illustrating a frame-based Gaussian splatting technique according to one embodiment of the present disclosure.

[0160] Referring to FIG. 8, the processor can collect camera data (S310). The processor can use multiple cameras to capture dynamic scenes from various angles and synchronize all camera data along the time axis to align them frame by frame.

[0161] The processor can refine the captured data (S320). The processor can convert the captured data into a 3D point cloud and represent it as a Gaussian distribution. The processor can summarize the Gaussian data to generate feature vectors such as position, size, and density. The processor can evaluate similarity between frames using Champer distance and identify repetitive scenes. The processor can cache the feature vectors of all frames and additionally cache PLY data of repetitive scenes.

[0162] The processor can perform real-time interpolation processing (S230). The processor can generate an intermediate vector between two frames based on cached feature vectors. The processor can ensure a natural connection between frames by linearly interpolating the location, magnitude, density, etc. of a Gaussian distribution.

[0163] The processor can reconstruct Gaussian data (S240). The processor can generate a new Gaussian point cloud based on the interpolated data. The processor can convert the new Gaussian point cloud into PLY format.

[0164] The processor can render the reconstructed Gaussian point cloud data to the display in real time (S250). The processor can combine the interpolated data and the repetitive scene data to provide natural and seamless screen transitions.

[0165] The frame-based Gaussian splatting technique of the present disclosure is a technology that supports real-time scene navigation and natural rendering in ICVFX, VR, AR, and game environments, and can provide smooth scene transitions and consistent visual quality.

[0166]

[0167] Meanwhile, a computer-readable medium storing a data structure is disclosed according to an embodiment of the present disclosure.

[0168] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data to solve specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of the device's resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0169] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0170] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form loops in a graph data structure.

[0171] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, the term neural network will be used consistently. A data structure may include a neural network. Furthermore, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the configurations described above, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated during the computational process of the neural network, and is not limited to the foregoing. A computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0172] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0173] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0174] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0175] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0176] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0177]

[0178] FIG. 9 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0179] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0180] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0181] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0182] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0183] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0184] An exemplary environment for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1104).

[0185] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0186] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0187] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0188] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0189] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0190] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0191] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0192] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0193] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0194] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0195] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0196] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0197] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0198] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0199] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0200] As described above, the relevant details have been described in the best mode for carrying out the invention.

Claims

1. A real-time rendering method utilizing point cloud data performed by a computing device, wherein A step of capturing a dynamic scene to acquire multiple frames; A step of converting the geometric information of the target object into point cloud data for each frame; A step of performing interpolation processing between two frames using the above point cloud data; and A step of rendering based on the above interpolated data including, method.

2. In Paragraph 1, The step of capturing the dynamic scene and acquiring the plurality of frames is A step of acquiring synchronized data from multiple cameras; and Step of classifying the above synchronized data into the same frame number including, method.

3. In Paragraph 1, The step of generating the geometric information of the target object into the point cloud data for each of the above frames is: A step of converting the geometric information of the target object into a Gaussian distribution for each of the above frames to generate a Gaussian point cloud; and Step to generate Gaussian point cloud data by optimizing the parameters of each Gaussian point including, method.

4. In Paragraph 3, The above method is, Step of converting the Gaussian point cloud data of each frame into a feature vector Includes more, The step of performing interpolation processing between the two frames above is, A step of generating intermediate data between the two frames using the transformed feature vector; including, method.

5. In Paragraph 4, The step of rendering based on the interpolated data above is, A step of generating new Gaussian point cloud data based on the above intermediate data; and Rendering step using the new Gaussian point cloud data mentioned above including, method.

6. In Paragraph 4, The above method is, The method further includes the step of caching the transformed feature vector, The step of performing interpolation processing between the two frames above is, A step comprising generating intermediate data between the two frames using the cached feature vector, method.

7. In Paragraph 6, The step of generating intermediate data between the two frames using the cached feature vector is: A step of identifying identical Gaussian points between the two frames; and A step comprising generating intermediate data between the two frames based on the above identification result, method.

8. In Paragraph 7, The step of generating intermediate data between the two frames based on the above identification result is, A step of matching Gaussian points determined to be identical based on the above identification result; If there are Gaussian points that do not match each other, the method further includes the step of generating new Gaussian points or progressively removing said Gaussian points. method.

9. In Paragraph 1, The above method is, Step of aligning the point clouds of two frames; A step of evaluating similarity between the two frames by comparing the matched point clouds; and Step of caching point cloud data of frames determined to have high similarity based on evaluation results including, method.

10. In Paragraph 9, The step of rendering based on the interpolated data above is, A step of rendering by combining the interpolated data and the cached point cloud data. including, method.

11. A computer program stored on a computer-readable storage medium, wherein when the computer program is executed by at least one processor, the at least one processor is made to perform the following operations, and the operations are: The operation of capturing a dynamic scene to acquire multiple frames; An operation to convert the geometric information of a target object into point cloud data for each frame; An operation to perform interpolation processing between two frames using the above point cloud data; and Rendering operation based on the above interpolated data including, A computer program stored on a computer-readable storage medium.

12. In Paragraph 11, The operation of capturing the above dynamic scene and acquiring the above plurality of frames is, The operation of acquiring synchronized data from multiple cameras; and The operation of classifying the above synchronized data into the same frame number including, A computer program stored on a computer-readable storage medium.

13. In Paragraph 11, The operation of generating the geometric information of the target object into the point cloud data for each of the above frames is, The operation of converting the geometric information of the target object into a Gaussian distribution for each of the above frames to generate a Gaussian point cloud; and The operation of generating Gaussian point cloud data by optimizing the parameters of each Gaussian point including, A computer program stored on a computer-readable storage medium.

14. In Paragraph 13, The above operation is, The operation of converting the Gaussian point cloud data of each of the above frames into feature vectors Includes more, The operation of performing interpolation processing between the two frames above is, The operation of generating intermediate data between the two frames using the transformed feature vector above. including, A computer program stored on a computer-readable storage medium.

15. In Paragraph 14, The operation of rendering based on the above interpolated data is, The operation of generating new Gaussian point cloud data based on the above intermediate data; and Rendering operation utilizing the above new Gaussian point cloud data including, A computer program stored on a computer-readable storage medium.

16. As a computing device, At least one processor; and memory Includes, The above at least one processor is, Capture a dynamic scene to acquire multiple frames, and For each frame, convert the geometric information of the target object into point cloud data, and Using the above point cloud data, interpolation processing is performed between two frames, and, Configured to render based on the above interpolated data, device.

17. In Paragraph 16, The above at least one processor is, Acquire synchronized data from multiple cameras, and Additionally configured to classify the above synchronized data into the same frame number, device.

18. In Paragraph 16, The above-mentioned at least one processor is, For each of the above frames, the geometric information of the target object is converted into a Gaussian distribution to generate a Gaussian point cloud, and additionally configured to generate Gaussian point cloud data by optimizing the parameters of each Gaussian point, device.

19. In Paragraph 18, The above-mentioned at least one processor is, Convert the Gaussian point cloud data of each of the above frames into feature vectors, and Further configured to generate intermediate data between the two frames using the transformed feature vector above, device.

20. In Paragraph 19, The above-mentioned at least one processor is, Based on the above intermediate data, new Gaussian point cloud data is generated, and Additionally configured to render using the above-mentioned new Gaussian point cloud data, device.

Images