Method for stabilizing image by utilizing plurality of pieces of data

Abstract

A method by which one or more processors of a computing device increases image accuracy, according to one embodiment of the present disclosure, is disclosed. The method may comprise the steps of: acquiring first data related to a position of an image; acquiring second data related to movement in the image; supplementing the second data on the basis of the first data and supplementing the first data on the basis of the second data so as to acquire combined data; and acquiring a final image on the basis of the combined data and the image.

Description

Method to stabilize video using multiple data

[0001] The present disclosure relates to a method for stabilizing an image using multiple data, and more specifically, to a method for stabilizing an image by utilizing multiple data together to mutually compensate for the loss between first data, which is accurate but may experience data loss in the middle, and second data, which is useful for accurately identifying the movement of an object by providing real-time data but has the problem that the measured value gradually differs from the actual value over time.

[0002] In existing video and content production environments, the process of stabilizing camera images has played a crucial role in digital content production. Specifically, because the various camera images used in video production environments do not have identical distortion rates and positional information, there was a problem where the quality of the captured images deteriorated due to inaccurate afterimages or shaking. Furthermore, there was a limitation in that data acquired from different sensors included in the camera images for tracking were difficult to utilize together because their timecodes differed. For example, while 6DOF tracking information obtained through an optical tracking system can be used for tracking by acquiring the object's rotation (roll, pitch, yaw) and position (x,y,z coordinates), there was a problem in that it could not acquire rapidly changing rotation and acceleration information.

[0003] Therefore, unlike conventional camera tracking methods, there is a growing need for a method to stabilize video more accurately and efficiently by compensating for the loss of main tracking data with sub-tracking data and compensating for the gaps in sub-tracking data with main tracking data.

[0004] Meanwhile, although the present disclosure is derived at least based on the technical background examined above, the technical problem or objective of the present disclosure is not limited to solving the problems or disadvantages examined above. That is, in addition to the technical issues examined above, the present disclosure can cover various technical issues related to the contents described below.

[0005] The present disclosure relates to a method for stabilizing an image using multiple data, and more specifically, the problem is to stabilize an image by utilizing multiple data together to mutually compensate for the loss between first data, which is accurate but may experience data loss in the middle, and second data, which is useful for accurately identifying the movement of an object by providing real-time data but has the problem that the measured value gradually differs from the actual value over time.

[0006] In addition, the present disclosure states that the first data loss has an error rate due to environmental factors due to the limitations of the first data technology itself, in addition to the reasons mentioned above, and can be compensated for based on the second data operated by a different technology, and since the error in the technology itself for acquiring the second data also occurs in a different part from the first data, the parts where there is mutual loss between the data can be mutually compensated for.

[0007] Meanwhile, the technical problem that the present disclosure aims to solve is not limited to the technical problem mentioned above, and various technical problems may be included within the scope obvious to a person skilled in the art from the contents described below.

[0008] A method performed by a computing device according to one embodiment of the present disclosure for realizing the aforementioned task is disclosed. The method may include: a step of acquiring first data related to a position of an image; a step of acquiring second data related to a movement of the image; a step of acquiring combined data by supplementing the second data based on the first data and supplementing the first data based on the second data; and a step of acquiring a final image based on the combined data and the image.

[0009] Alternatively, the step of acquiring first data related to the position of an image may include at least one of the step of acquiring first-1 data related to the rotation of an object included in the image; or the step of acquiring first-2 data related to the position of an object included in the image.

[0010] Alternatively, the step of acquiring second data related to the movement of the image may include at least one of the following: acquiring second-1 data related to the acceleration of an object included in the image; acquiring second-2 data related to the angular velocity of an object included in the image; or acquiring second-3 data related to the orientation of an object included in the image.

[0011] Alternatively, the step of acquiring first data related to the position of the image may further include the step of acquiring first time series data for the first data, and the step of acquiring second data related to the movement of the image may further include the step of acquiring second time series data for the second data.

[0012] Alternatively, the step of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data may include: a step of obtaining first corrected data based on the first data, the first time series data and the second time series data; a step of obtaining second corrected data based on the second data, the first time series data and the second time series data; and a step of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data.

[0013] Alternatively, the step of obtaining a first corrected data based on the first data, the first time series data, and the second time series data comprises: a step of obtaining combined time series data by matching the time series of the first time series data and the second time series data; and a step of obtaining a first corrected data based on the first data and the combined time series data, and the step of obtaining a second corrected data based on the second data, the first time series data, and the second time series data may include a step of obtaining a second corrected data based on the second data and the combined time series data.

[0014] Alternatively, the step of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data may include at least one of the steps of: supplementing an error accumulated over time for the second corrected data based on the first corrected data to obtain the combined data; or supplementing a data loss included in the first corrected data based on the second corrected data to obtain the combined data.

[0015] Alternatively, the step of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data may further include the step of performing filtering on the combined data to obtain filtered combined data.

[0016] Alternatively, the step of obtaining filtered combined data by performing filtering on the combined data may include at least one of the steps of: obtaining filtered combined data by performing filtering on the combined data based on a Kalman filter; or obtaining filtered combined data by performing filtering on the combined data based on a complementary filter.

[0017] Alternatively, the step of acquiring a final image based on the combined data and the image may include the step of acquiring the final image by correcting shaking of an object included in the image based on the combined data.

[0018] Alternatively, the step of acquiring a final image based on the combined data and the image may include the step of acquiring the final image by correcting the viewpoint for a portion of the image based on the combined data.

[0019] A computer program stored on a computer-readable storage medium is disclosed in accordance with one embodiment of the present disclosure for realizing the aforementioned tasks. When the computer program is executed on one or more processors, the one or more processors are configured to perform operations to increase the accuracy of an image, and the operations may include: an operation to acquire first data related to the position of the image; an operation to acquire second data related to the movement of the image; an operation to acquire combined data by supplementing the second data based on the first data and supplementing the first data based on the second data; and an operation to acquire a final image based on the combined data and the image.

[0020] Alternatively, the operation of acquiring first data related to the position of an image may include at least one of the operation of acquiring first-1 data related to the rotation of an object included in the image; or the operation of acquiring first-2 data related to the position of an object included in the image.

[0021] Alternatively, the operation of acquiring second data related to the movement of the image may include at least one of the following: an operation of acquiring second-1 data related to the acceleration of an object included in the image; an operation of acquiring second-2 data related to the angular velocity of an object included in the image; or an operation of acquiring second-3 data related to the orientation of an object included in the image.

[0022] Alternatively, the operation of acquiring first data related to the position of the image may further include the operation of acquiring first time series data for the first data, and the operation of acquiring second data related to the movement of the image may further include the operation of acquiring second time series data for the second data.

[0023] Alternatively, the operation of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data may include: the operation of obtaining first corrected data based on the first data, the first time series data and the second time series data; the operation of obtaining second corrected data based on the second data, the first time series data and the second time series data; and the operation of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data.

[0024] Alternatively, the operation of obtaining a first corrected data based on the first data, the first time series data, and the second time series data includes: the operation of obtaining combined time series data by matching the time series of the first time series data and the second time series data; and the operation of obtaining a first corrected data based on the first data and the combined time series data, and the operation of obtaining a second corrected data based on the second data, the first time series data, and the second time series data may include the operation of obtaining a second corrected data based on the second data and the combined time series data.

[0025] Alternatively, the operation of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data may include at least one of the operation of supplementing an error accumulated over time for the second corrected data based on the first corrected data to obtain the combined data; or supplementing a data loss included in the first corrected data based on the second corrected data to obtain the combined data.

[0026] Alternatively, the operation of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data may further include the operation of performing filtering on the combined data to obtain filtered combined data.

[0027] Alternatively, the operation of obtaining filtered combined data by performing filtering on the combined data may include at least one of the operation of obtaining filtered combined data by performing filtering on the combined data based on a Kalman filter; or obtaining filtered combined data by performing filtering on the combined data based on a complementary filter.

[0028] Alternatively, the operation of acquiring a final image based on the combined data and the image may include the operation of acquiring the final image by correcting shaking of an object included in the image based on the combined data.

[0029] Alternatively, the operation of acquiring a final image based on the combined data and the image may include the operation of acquiring the final image by correcting the viewpoint for a part of the image based on the combined data.

[0030] A computing device according to one embodiment of the present disclosure for realizing the aforementioned tasks is disclosed. The device comprises at least one processor; and a memory, wherein the at least one processor may be configured to acquire first data related to a position of an image; acquire second data related to a movement of the image; supplement the second data based on the first data and supplement the first data based on the second data to acquire combined data; and acquire a final image based on the combined data and the image.

[0031] The present disclosure relates to a method for stabilizing an image using multiple data. More specifically, the image can be stabilized more accurately and efficiently by utilizing multiple data together to mutually compensate for the loss between first data, which is accurate but may experience data loss in the middle, and second data, which is useful for accurately identifying the movement of an object by providing real-time data but has the problem that the measured value gradually differs from the actual value over time.

[0032] Meanwhile, the effects of the present disclosure are not limited to those mentioned above, and various effects may be included within the scope obvious to a person skilled in the art from the contents described below.

[0033] FIG. 1 is a block diagram of a computing device for stabilizing an image using a plurality of data according to one embodiment of the present disclosure.

[0034] FIG. 2 is a schematic diagram showing a network function according to one embodiment of the present disclosure.

[0035] FIG. 3 is a flowchart illustrating a method for stabilizing an image using a plurality of data according to one embodiment of the present disclosure.

[0036] FIG. 4a is a schematic diagram illustrating a process for acquiring first data related to a position in an image according to one embodiment of the present disclosure.

[0037] FIG. 4b is a schematic diagram illustrating the process of acquiring second data related to the movement of an image according to one embodiment of the present disclosure.

[0038] FIG. 5 is a schematic diagram illustrating a process of obtaining combined data by supplementing second data based on first data and supplementing first data based on second data according to one embodiment of the present disclosure.

[0039] FIG. 6 is a schematic diagram illustrating the process of obtaining a final image based on combined data and an image according to one embodiment of the present disclosure.

[0040] FIG. 7 is a schematic diagram illustrating the process of obtaining the final image by correcting the viewpoint of a portion of the image based on combined data according to one embodiment of the present disclosure.

[0041] FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0042] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0043] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0044] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0045] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0046] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0047] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0048] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0049] In the present disclosure, network functions, artificial neural networks, and neural networks may be used interchangeably.

[0050]

[0051] FIG. 1 is a block diagram of a computing device for stabilizing an image using a plurality of data according to one embodiment of the present disclosure.

[0052] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0053] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0054] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for training a neural network model. The processor (110) may perform calculations for training a neural network model, such as processing input data for training in deep learning (DL), extracting features from input data, calculating errors, and updating weights of the neural network model using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the training of the neural network model. For example, a CPU and a GPGPU can work together to process the training of a neural network model and the classification of data using the neural network model. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the training of a neural network model and the classification of data using the neural network model. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0055] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0056] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0057] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0058] Additionally, the network unit (150) presented in this disclosure may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0059] In the present disclosure, the network unit (150) can be configured regardless of the mode of communication, such as wired and wireless, and can be configured as various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). In addition, the network may be a known World Wide Web (WWW) and may utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The technologies described in the present disclosure may also be used in other networks mentioned above.

[0060]

[0061] FIG. 2 is a schematic diagram showing a network function according to one embodiment of the present disclosure.

[0062] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

[0063] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0064] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight can be variable and can be varied by the user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0065] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0066] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0067] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0068] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0069] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network allows for the identification of latent structures in data. That is, it is possible to identify the latent structures of photos, text, videos, voice, and music (e.g., what objects are present in a photo, what the content and emotions of the text are, what the content and emotions of the voice are, etc.). Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0070] In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to the input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be placed between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrically with respect to the input layer). The autoencoder may perform non-linear dimensionality reduction. The number of input and output layers may correspond to the dimension after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may have a structure in which the number of nodes decreases as it moves away from the input layer. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (e.g., more than half of the input layer).

[0071] Neural networks can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The training of a neural network may be the process of applying knowledge to the neural network to perform a specific action.

[0072] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training point is labeled with a category. The labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node being updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0073] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed by the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a form of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0074]

[0075] FIG. 3 is a flowchart illustrating a method for stabilizing an image using a plurality of data according to one embodiment of the present disclosure.

[0076] A computing device (100) according to one embodiment of the present disclosure may directly acquire "information for stabilizing an image using multiple data" or receive it from an external system. The external system may be a server, a database, etc., that stores and manages information for stabilizing an image using multiple data. The computing device (100) may use the information directly acquired or received from the external system as "input data for stabilizing an image using multiple data."

[0077] According to one embodiment of the present disclosure, a computing device (100) can acquire first data related to the position of an image (S110). For example, the computing device (100) can acquire first-1 data related to the rotation of an object included in the image, or acquire first-2 data related to the position of an object included in the image. For example, the computing device (100) can acquire the first-1 data composed of roll, pitch, and yaw angles rotating around each axis included in the image, and can acquire the first-2 data representing movement (position) in the x, y, and z axis directions. Meanwhile, the computing device (100) can acquire first time-series data for the first data. In this regard, the first data related to the position of the image may be collected at a different frequency from the second data related to the movement of the image, which will be described later, and in this case, the time-series between the data are different, so the first data cannot be utilized in the mutual complementation process. Accordingly, the computing device (100) can acquire first time-series data for the first data in order to accurately match the first data and the second data to be described later on the time axis. Meanwhile, the first data related to the position of the image can be utilized in the process of the computing device (100) supplementing the second data to be described later, and a specific explanation thereof will be provided below through FIG. 5.

[0078] According to one embodiment of the present disclosure, a computing device (100) can acquire second data related to the movement of the image in step S110 (S120). Specifically, the computing device (100) can acquire second-1 data related to the acceleration of an object included in the image, acquire second-2 data related to the angular velocity of an object included in the image, and acquire second-3 data related to the direction of an object included in the image. Meanwhile, the computing device (100) can acquire second time-series data for the second data. In this regard, the second data related to the movement of the image may be collected at a different frequency from the first data related to the position of the image, and in this case, the time series between the first and second data are different, so the second data cannot be utilized in the mutual complementation process. Therefore, the computing device (100) can acquire second time-series data for the second data in order to accurately match the second data and the first data on the time axis. Meanwhile, the second data related to the movement of the above image can be utilized in the process in which the computing device (100) supplements the first data related to the position of the above image, and a specific explanation thereof will be described later through FIG. 5.

[0079] According to one embodiment of the present disclosure, a computing device (100) can obtain combined data by supplementing the second data obtained through step S120 based on the first data obtained through step S110, and supplementing the first data based on the second data (S130). At this time, the computing device (100) can obtain first corrected data based on the first data, the first time series data, and the second time series data, and can obtain second corrected data based on the second data, the first time series data, and the second time series data. In this regard, if the time series of the first data and the second data are different, it is difficult for the computing device (100) to perform the process of supplementing the second data based on the first data or supplementing the first data based on the second data. Accordingly, the computing device (100) can obtain combined time series data by matching the time series of the first time series data and the second time series data, and can obtain the first corrected data based on the first data and the combined time series data. Likewise, the computing device (100) can obtain the second corrected data based on the second data and the combined data, and the first corrected data and the second corrected data may mean data having the same frequency.

[0080] Subsequently, the computing device (100) can obtain combined data by supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data. For example, the computing device (100) can obtain the combined data by supplementing the error accumulated over time for the second corrected data based on the first corrected data, and can obtain the combined data by supplementing the data loss included in the first corrected data based on the second corrected data. Meanwhile, the computing device (100) can obtain filtered combined data by performing filtering on the combined data to increase the accuracy of the combined data. For example, the computing device (100) can obtain filtered combined data by performing filtering on the combined data based on a Kalman filter, or obtain filtered combined data by performing filtering on the combined data based on a Complementary Filter. However, in the filtering process of the present disclosure, the embodiments described above are not limited to the filters and various examples may be utilized. Meanwhile, the computing device (100) may be utilized in the process of obtaining a final image based on the image using the acquired combined data, and a detailed explanation thereof will be provided below through FIG. 6.

[0081] According to one embodiment of the present disclosure, a computing device (100) can obtain a final image based on the combined data obtained through step S130 and the image from step S110 (S140). For example, the computing device (100) can obtain the final image by correcting the shaking of an object included in the image based on the combined data. As another example, the computing device (100) can obtain the final image by correcting the viewpoint of a part of the image based on the combined data. Specifically, the computing device (100) can obtain the final image by correcting the viewpoint of the inner frustum region of the image captured by the camera without changing the entire image. In this regard, since the computing device (100) can obtain information regarding the absolute position of the image through the first data and obtain information regarding the relative movement of the image through the second data, the computing device (100) can obtain the final image based on the combined data in which the first data and the second data are mutually complementary, thereby enabling the acquisition of a stable image by utilizing both the advantages of the first data and the second data even in environments where real-time performance is required or where sensor data is limited. Meanwhile, a detailed explanation of the process by which the computing device (100) obtains the final image based on the acquired combined data is described below through FIGS. 6 and 7.

[0082]

[0083] FIG. 4a is a schematic diagram illustrating a process for acquiring first data related to the position of an image according to one embodiment of the present disclosure, and FIG. 4b is a schematic diagram illustrating a process for acquiring second data related to the movement of an image according to one embodiment of the present disclosure.

[0084] First, referring to FIG. 4a, a computing device (100) can acquire first data (10) related to the position of an image. At this time, the image may refer to a result captured by a camera, and the image may include at least one object. Specifically, the computing device (100) can acquire first-1 data (10-1) related to the rotation of an object included in the image, or first-2 data (10-2) related to the position of an object included in the image. For example, the computing device (100) can acquire the first-1 data (10-1) composed of roll, pitch, and yaw angles rotating around each axis included in the image, and can acquire the first-2 data (10-2) indicating movement (position) in the x, y, and z axis directions. At this time, the roll may refer to the degree of rotation around the x-axis, the pitch may refer to the degree of rotation around the y-axis, and the yaw angle may refer to the degree of rotation around the z-axis. Additionally, the computing device (100) may acquire first time-series data for the first data (10). In this regard, the first data (10) related to the position of the image may be collected at a different frequency from the second data related to the movement of the image to be described later, and in this case, the time series between the data is different, so the first data (10) cannot be utilized in the mutual complementation process. Therefore, the computing device (100) may acquire first time-series data for the first data (10) in order to accurately match the first data (10) and the second data to be described later on the time axis. Meanwhile, the first data (10) related to the location of the above image can be utilized in the process of the computing device (100) supplementing the second data to be described later, and a specific explanation thereof is described later through FIG. 5.

[0085] Referring to FIG. 4b thereafter, the computing device (100) can obtain second data (20) related to the movement of the image. Specifically, the computing device (100) can obtain second-1 data (20-1) related to the acceleration of an object included in the image, second-2 data (20-2) related to the angular velocity of an object included in the image, and second-3 data (20-3) related to the direction of an object included in the image. For example, the computing device (100) can obtain second-1 data (20-1) related to the acceleration of an object included in the image by estimating movement (change in position) based on gravity and acceleration through an accelerometer, and can obtain second-2 data (20-2) related to the angular velocity of an object included in the image by measuring angular velocity (change in roll, pitch, yaw) through a gyroscope. Alternatively, the computing device (100) may obtain second-third data (20-3) by utilizing a geomagnetic sensor to measure the movement (acceleration, rotation) of an object included in the image. However, the second data (20) may include various examples related to the movement of the image, not limited to the example of FIG. 4b. In this regard, since the second data (20) includes information on the relative movement of the image, the computing device (100) can obtain a more stable image by utilizing the second data (20) in the process of correcting image distortion that occurs when the camera moves or shakes. Meanwhile, the computing device (100) may obtain second time-series data for the second data (20). In this regard, the second data (20) related to the movement of the image may be collected at a different frequency from the first data (10) related to the position of the image, and in this case, the time series between the first data (10) and the second data (20) is different, so the second data (20) cannot be utilized in the mutual complementation process.Accordingly, the computing device (100) can acquire second time-series data for the second data (20) in order to accurately match the second data (20) and the first data (10) on the time axis. Meanwhile, the second data (20) related to the movement of the image can be utilized in the process of the computing device (100) supplementing the first data (10) related to the position of the image, and a specific explanation thereof will be described later through FIG. 5.

[0086]

[0087] FIG. 5 is a schematic diagram illustrating a process of obtaining combined data by supplementing second data based on first data and supplementing first data based on second data according to one embodiment of the present disclosure.

[0088] Referring to FIG. 5, the computing device (100) can obtain combined data (30) by supplementing the obtained second data (20) based on the obtained first data (10) and supplementing the first data (10) based on the second data (20). At this time, the computing device (100) can obtain first corrected data (10') based on the first data (10), the first time series data, and the second time series data. Additionally, the computing device (100) can obtain second corrected data (20') based on the second data (20), the first time series data, and the second time series data. In this regard, when the time series of the first data (10) and the second data (20) are different, the computing device (100) finds it difficult to perform the process of supplementing the second data (20) based on the first data (10) or supplementing the first data (10) based on the second data (20). Therefore, the computing device (100) can obtain combined time series data by synchronizing the time series of the first time series data and the second time series data, and can obtain the first corrected data (10') based on the first data (10) and the combined time series data. Similarly, the computing device (100) can obtain the second corrected data (20') based on the second data (20) and the combined data, and the first corrected data (10') and the second corrected data (20') may mean data with the same frequency.

[0089] Subsequently, the computing device (100) can obtain the combined data (30) through a mutual complementation process in which the second corrected data (20') is supplemented based on the first corrected data (10'), and the first corrected data (10') is supplemented based on the second corrected data (20'). For example, the computing device (100) can obtain the combined data (30) by supplementing the error accumulated over time for the second corrected data (20') based on the first corrected data (10'), and can obtain the combined data (30) by supplementing the data loss included in the first corrected data (10') based on the second corrected data (20'). In this regard, the first corrected data (10') is accurate in identifying the position of the object and the position of the image compared to the second corrected data (20'), but it does not contain information about relative movement, so data loss may occur in the middle, and it may have the disadvantage of being sensitive to changes in lighting. In addition, the second corrected data (20') can be updated in real-time at a high frequency compared to the first data (10'), so it can provide data at a much faster speed than a camera and correct shaking that occurs when the object being filmed moves, so it has the advantage of tracking the object more clearly, but as time accumulates, a drift problem may occur in which the actual value gradually differs from the data.Accordingly, the computing device (100) can compensate for data loss or instability that may occur in the first corrected data (10') with the second corrected data (20') that can be updated in real time, and can compensate for drift problems in the second corrected data (20') by utilizing the first corrected data (10'). The combined data (30) obtained through this is a result in which the disadvantages of each of the two data (10' and 20') are mutually compensated, and the computing device (100) can obtain a more accurate result in the image stabilization process or camera tracking process by utilizing the combined data (30).

[0090] Meanwhile, the computing device (100) can obtain filtered combined data by performing filtering on the combined data (30) to increase the accuracy of the combined data (30). For example, the computing device (100) can obtain filtered combined data by performing filtering on the combined data (30) based on a Kalman filter, or obtain filtered combined data by performing filtering on the combined data (30) based on a complementary filter. In this regard, if the computing device (100) utilizes a Kalman filter in the filtering process for the combined data (30), it can calculate a more accurate estimate based on the reliability of the two data sources (10 and 20). Alternatively, if the computing device (100) utilizes a complementary filter in the filtering process for the combined data (30), since the gyroscope has high short-term accuracy and the accelerometer has high long-term accuracy, there is an advantage of simple implementation and low computational load by using gyroscope and accelerometer data complementarily. However, in the filtering process of the present disclosure, the embodiments described above are not limited to the filters and various examples may be utilized. Meanwhile, the computing device (100) may be utilized in the process of obtaining a final image based on the image using the acquired combined data (30), and a detailed explanation thereof will be provided below through FIG. 6.

[0091]

[0092] FIG. 6 is a schematic diagram illustrating the process of obtaining a final image based on combined data and an image according to one embodiment of the present disclosure.

[0093] Referring to FIG. 6, the computing device (100) can obtain a final image (11') based on the acquired combined data (30) and the acquired image (11). At this time, the image (11) may include an image acquired through a camera during the content shooting process, or an image pre-stored in a database. For example, the computing device (100) can obtain the final image (11') by correcting the shaking of an object included in the image (11) based on the combined data (30). In this regard, the first data (10) used in the process of acquiring the combined data (30) may include information regarding the absolute position of the image (11), and the second data (20) may include information regarding the relative movement of the image (11). Therefore, the computing device (100) can perform stabilization of the image (11) by utilizing the combined data (30) in which the first data (10) and the second data (20) are mutually complementary, and thereby acquire the final image (11'). This allows for the acquisition of a stabilized image by utilizing all the advantages of the first data (10) and the second data (20), which provide different types of information, even in environments where real-time performance is required or where sensor data is limited, and can significantly improve the accuracy and stability of camera tracking.

[0094]

[0095] FIG. 7 is a schematic diagram illustrating the process of obtaining the final image by correcting the viewpoint of a portion of the image based on combined data according to one embodiment of the present disclosure.

[0096] Referring to FIG. 7, the computing device (100) can obtain the final image by correcting the viewpoint (32-2) for a part area (32-1) of the image (11) based on the acquired combined data (30). Specifically, the computing device (100) can obtain the final image by correcting the viewpoint for the inner frustum area (32-1 and 32-2) of the image captured by the camera without rendering the entire image output to the background in a virtual production environment, and by performing rendering only for the inner frustum correction area (32-2) of the image among the entire image (11) through an image renderer machine (31). In this regard, the first data (10) used in the process of acquiring the combined data (30) may include information regarding the absolute position of the image (11), and the second data (20) may include information regarding the relative movement of the image (11). Therefore, the computing device (100) can acquire a rapidly stabilized image by utilizing the combined data (30) in which the first data (10) and the second data (20) are mutually complementary, and by performing stabilization on the inner frustum region (32-1 to 32-2), which is the shooting area of ​​the camera in the image (11), and by performing rendering only on this area to acquire the final image (11'), thereby utilizing all the advantages of the first data (10) and the second data (20) that provide different types of information even in environments where real-time performance is required or where sensor data is limited, and can significantly improve the accuracy and stability of camera tracking.

[0097]

[0098] According to one embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed. A data structure may refer to the organization, management, and storage of data that enables efficient access and modification of the data. A data structure may refer to the organization of data for solving specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may be defined as physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connection relationships between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored in a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include a set of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while using minimal resources of the computing device. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0099] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0100] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form a loop in a graph data structure.

[0101] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, the term neural network will be used consistently. A data structure may include a neural network. Furthermore, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the configurations described above, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated during the computational process of the neural network, and is not limited to the foregoing. A computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0102] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0103] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0104] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0105] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0106] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0107]

[0108] FIG. 8 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0109] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0110] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0111] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0112] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0113] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0114] An exemplary environment (1100) for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit (1104).

[0115] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0116] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0117] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0118] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0119] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0120] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0121] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0122] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0123] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0124] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0125] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0126] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0127] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0128] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0129] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0130] As described above, the relevant details have been described in the best mode for carrying out the invention.

Claims

1. A method for increasing the accuracy of an image, performed by one or more processors of a computing device, A step of acquiring first data related to the position of the image; A step of acquiring second data related to the movement of the above image; A step of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data; and A step of obtaining a final image based on the combined data and the image above; including, method.

2. In Paragraph 1, The step of acquiring first data related to the location of the above image is, A step of acquiring 1-1 data related to the rotation of an object included in the above image; or A step of obtaining first- and second data related to the location of an object included in the above image; including at least one of, method.

3. In Paragraph 1, The step of acquiring second data related to the movement of the above image is, A step of acquiring 2-1 data related to the acceleration of an object included in the above image; A step of acquiring 2-2 data related to the angular velocity of an object included in the above image; or A step of obtaining second- and third data related to the orientation of an object included in the above image; including at least one of, method.

4. In Paragraph 1, The step of acquiring first data related to the location of the above image is, The method further includes the step of obtaining first time series data for the first data above, and The step of acquiring second data related to the movement of the above image is, A further step of acquiring second time series data for the second data, method.

5. In Paragraph 1, The step of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data is: A step of obtaining first corrected data based on the first data, the first time series data, and the second time series data; A step of obtaining second corrected data based on the second data, the first time series data, and the second time series data; and A method comprising the step of supplementing the second corrected data based on the first corrected data, and supplementing the first corrected data based on the second corrected data to obtain combined data. method.

6. In Paragraph 5, The step of obtaining first corrected data based on the first data, the first time series data, and the second time series data is, A step of obtaining combined time series data by matching the time series of the first time series data and the second time series data; and It includes the step of obtaining first corrected data based on the first data and the combined time series data, and The step of obtaining second corrected data based on the second data, the first time series data, and the second time series data is: A step comprising obtaining second corrected data based on the second data and the combined time series data, method.

7. In Paragraph 5, The step of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data is: A step of obtaining the combined data by compensating for the error accumulated over time for the second corrected data based on the first corrected data; or A step of obtaining the combined data by supplementing the data loss included in the first corrected data based on the second corrected data; including at least one of, method.

8. In Paragraph 5, The step of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data is: A method further comprising the step of obtaining filtered combined data by performing filtering on the combined data above. method.

9. In Paragraph 8, The step of obtaining filtered combined data by performing filtering on the combined data above is: A step of obtaining filtered combined data by performing filtering on the combined data based on a Kalman filter; or A step of obtaining filtered combined data by performing filtering on the combined data based on a complementary filter; including at least one of, method.

10. In Paragraph 1, The step of obtaining a final image based on the combined data and the image is, A method comprising the step of obtaining the final image by correcting shaking of an object included in the image based on the combined data. method.

11. In Paragraph 1, The step of obtaining a final image based on the combined data and the image is, A step comprising obtaining the final image by correcting the viewpoint for a portion of the image based on the combined data above. method.

12. A computer program stored on a computer-readable storage medium, wherein, when the computer program is executed by one or more processors, the one or more processors are configured to perform operations to improve the accuracy of an image, and said operations are: An operation to acquire first data related to the position of the image; An operation to acquire second data related to the movement of the above image; An operation of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data; and An operation to acquire a final image based on the combined data and the image above; including, A computer program stored on a computer-readable storage medium.

13. In Paragraph 12, The operation of acquiring first data related to the position of the above image is, An operation to acquire 1-1 data related to the rotation of an object included in the above image; or An operation to acquire first-second data related to the location of an object included in the above image; including at least one of, A computer program stored on a computer-readable storage medium.

14. In Paragraph 12, The operation of acquiring second data related to the movement of the above image is, An operation to acquire 2-1 data related to the acceleration of an object included in the above image; An operation to acquire 2-2 data related to the angular velocity of an object included in the above image; or An operation to acquire second- and third data related to the orientation of an object included in the above image; including at least one of, A computer program stored on a computer-readable storage medium.

15. In Paragraph 12, The operation of acquiring first data related to the position of the above image is, The operation of acquiring first time series data for the first data further includes, The operation of acquiring second data related to the movement of the above image is, Further including the operation of acquiring second time series data for the second data above, A computer program stored on a computer-readable storage medium.

16. In Paragraph 12, The operation of supplementing the second data based on the first data and supplementing the first data based on the second data to obtain combined data is, An operation to obtain first corrected data based on the first data, the first time series data, and the second time series data; An operation of obtaining second corrected data based on the second data, the first time series data, and the second time series data; and A method comprising the operation of supplementing the second corrected data based on the first corrected data, and supplementing the first corrected data based on the second corrected data to obtain combined data. A computer program stored on a computer-readable storage medium.

17. In Paragraph 16, The operation of obtaining first corrected data based on the first data, the first time series data, and the second time series data is, The operation of obtaining combined time series data by matching the time series of the first time series data and the second time series data; and The operation includes obtaining first corrected data based on the first data and the combined time series data, and The operation of obtaining second corrected data based on the second data, the first time series data, and the second time series data is, The operation of obtaining second corrected data based on the second data and the combined time series data, A computer program stored on a computer-readable storage medium.

18. In Paragraph 16, The operation of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data is, An operation to obtain the combined data by compensating for the error accumulated over time with respect to the second corrected data based on the first corrected data; or An operation to obtain the combined data by supplementing the data loss included in the first corrected data based on the second corrected data; including at least one of, A computer program stored on a computer-readable storage medium.

19. In Paragraph 16, The operation of supplementing the second corrected data based on the first corrected data and supplementing the first corrected data based on the second corrected data to obtain combined data is, Further including the operation of obtaining filtered combined data by performing filtering on the combined data above. A computer program stored on a computer-readable storage medium.

20. In Paragraph 19, The operation of obtaining filtered combined data by performing filtering on the above combined data is, An operation to obtain filtered combined data by performing filtering on the above combined data based on a Kalman filter; or An operation to obtain filtered combined data by performing filtering on the above combined data based on a complementary filter; including at least one of, A computer program stored on a computer-readable storage medium.

21. In Paragraph 12, The operation of acquiring a final image based on the combined data and the image is, A method comprising the operation of obtaining the final image by correcting shaking of an object included in the image based on the combined data. A computer program stored on a computer-readable storage medium.

22. In Paragraph 12, The operation of acquiring a final image based on the combined data and the image is, The operation of obtaining the final image by correcting the viewpoint for a part area of ​​the image based on the combined data above, A computer program stored on a computer-readable storage medium.

23. As a computing device, At least one processor; and memory Includes, The above-mentioned at least one processor is, Acquire first data related to the location of the image; Acquire second data related to the movement of the above image; Based on the first data above, the second data is supplemented, and based on the second data, the first data is supplemented to obtain combined data; and Configured to acquire a final image based on the combined data and the image above, Computing device.

Images