Video encoding / decoding method and device, and latent space learning method

The image encoding/decoding method efficiently processes diverse visual information by dividing it into semantic units in a latent space, addressing inefficiencies in existing technologies through a unified structure and model, enhancing encoding/decoding flexibility and performance.

WO2026240614A1PCT designated stage Publication Date: 2026-11-19UNIVERSITY INDUSTRY COOPERATION GROUP OF KYUNG HEE UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2026/096956
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2026-05-13
Filing Date
2026-05-13
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing technologies are inefficient in handling diverse visual information formats and consumption purposes, requiring separate encoding and decoding devices optimized for specific types and uses, leading to suboptimal deployment and operation.

Method used

An image encoding/decoding method and apparatus that utilizes an artificial neural network to divide visual information into common and independent semantic units in a latent space, allowing for efficient encoding and decoding with a single structure and model weights, using identifier mapping, neural network-based decoding, and recombination techniques.

Benefits of technology

Enables high-efficiency visual information encoding/decoding across diverse data forms and purposes, improving operational efficiency and flexibility in processing various types of visual information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2026096956_19112026_PF_FP_ABST
    Figure KR2026096956_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A video encoding / decoding method and device, and a latent space learning method are provided. The video decoding method may comprise the steps of: determining, on the basis of identifier mapping information, a bitstream region to be decoded in a bitstream; decoding the determined bitstream region so as to reconstruct split latent representations; recombining semantic-unit latent representations from the split latent representations; and converting the recombined semantic-unit latent representations into a target space.
Need to check novelty before this filing date? Find Prior Art

Description

Image encoding / decoding method, device, and latent space learning method

[0001] The present invention relates to a method for encoding / decoding visual information, an apparatus, and a method for learning a latent space. Specifically, the present invention relates to a method and apparatus for encoding / decoding visual information that performs encoding and decoding by dividing input visual information into common and independent information of semantic units in a latent space through an artificial neural network-based transformation and recombining them.

[0002] Recently, the utilization of various types of visual information, including 3D / 4D formats such as point clouds, meshes, Gaussian splatting, and holography, as well as video and images, has been rapidly increasing. Along with this, the purposes for consuming collected visual information are also diversifying beyond human visual perception to include artificial neural network-based machine vision tasks (object classification, detection, region segmentation, etc.). As the types of visual information and consumption purposes become more diverse, research to achieve optimal encoding efficiency is being conducted independently; however, deploying and operating multiple encoding and decoding devices, each optimized for specific visual information and consumption purposes, separately on terminals and servers is highly inefficient. Accordingly, there is a need for high-efficiency visual information encoding / decoding technology that can accommodate diverse data forms and multiple purposes while operating with a single structure and single model weights.

[0003] The present invention aims to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0004] In addition, the present invention aims to provide a latent space learning method according to the present invention.

[0005] An image decoding method according to an embodiment of the present invention may include the steps of: determining a bitstream region to be decoded among bitstreams based on identifier mapping information; decoding the determined bitstream region to restore a partitioned latent representation; recombining semantic unit latent representations from the partitioned latent representations; and converting the recombined semantic unit latent representations into a destination space.

[0006] In a video decoding method, the step of determining a bitstream region to be decoded may determine a bitstream region optimized for at least one of machine vision and human vision as the bitstream region to be decoded.

[0007] In a video decoding method, the identifier mapping information may be at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

[0008] In a video decoding method, the method may further include the step of decoding a partitioned latent representation based on at least one of a NAL unit-based independent coding method and a unique syntax-based independent coding method.

[0009] In an image decoding method, the segmented latent representation can be restored based on at least one of a neural network-based segmented latent representation decoding method, an algorithm-based segmented latent representation decoding method, and a standard codec-based segmented latent representation decoding method.

[0010] In a video decoding method, the step of recombining semantic unit latent representations can be performed based on at least one combination means among channel unit recombination and spatial unit recombination.

[0011] In an image decoding method, the step of converting a recombined semantic unit latent representation into a target space can be performed based on at least one of a CNN-based conversion and an MLP-based conversion.

[0012] A video encoding method according to one embodiment of the present invention may include the steps of: converting input time information to generate a latent representation; dividing the generated latent representation to generate latent representations divided by common information and independent information between semantic unit time information; and independently encoding the divided latent representations based on identifier mapping information to generate a bitstream.

[0013] In a video encoding method, semantic unit visual information may be at least one of visual information having a higher-level semantics and visual information optimized for the purpose of use.

[0014] In a video encoding method, the step of generating a latent representation by converting input visual information can be performed based on at least one of a CNN-based conversion and an MLP-based conversion.

[0015] In a video encoding method, the step of dividing a generated latent representation to generate latent representations divided by common information and independent information between semantic unit visual information can be performed based on at least one of channel unit division and spatial unit division.

[0016] In a video encoding method, the step of encoding a divided latent representation to generate a bitstream can generate a bitstream composed of independent regions based on at least one of a NAL unit-based independent encoding method and a unique syntax-based independent encoding method.

[0017] In a video encoding method, the identifier mapping information may be at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

[0018] In a video encoding method, the step of encoding a segmented latent representation can be performed based on at least one of a neural network-based segmented latent representation encoding, an algorithm-based segmented latent representation encoding, and a standard codec-based segmented latent representation encoding.

[0019] A latent space learning method according to one embodiment of the present invention may include the steps of: generating a latent representation by converting input visual information; dividing the converted latent representation to generate a latent representation divided by common information and independent information between semantic unit visual information; converting the divided latent representation into a target space; calculating a loss function for each semantic unit for the converted visual information; calculating a semantic unit-contrast loss function based on a linear combination of the calculated semantic unit loss function and predefined parameters; and updating neural network parameters based on the calculated semantic unit-contrast loss function.

[0020] In a latent space learning method, the step of generating a latent representation by transforming input visual information can be performed based on at least one transformation method among CNN-based transformation and MLP-based transformation.

[0021] In a latent space learning method, the step of transforming a partitioned latent representation into a target space can be performed based on at least one transformation method among CNN-based transformation and MLP-based transformation.

[0022] In a latent space learning method, the step of calculating a loss function for each semantic unit of transformed visual information can be performed based on at least one of a loss function for maximizing cognitive quality and a loss function for maximizing machine vision task performance.

[0023] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0024] According to the present invention, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency may be provided.

[0025] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0026] FIG. 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present invention is applied.

[0027] FIG. 2 is a block diagram showing the configuration according to one embodiment of a decoding device to which the present invention is applied.

[0028] FIG. 3 is a schematic diagram showing a video coding system to which the present invention can be applied.

[0029] FIG. 4 is a diagram showing the configuration of a visual information encoding device and a decoding device based on latent expression partitioning and recombination according to an embodiment of the present invention.

[0030] FIG. 5 is a diagram illustrating semantic unit visual information according to an embodiment of the present invention.

[0031] FIG. 6 is a diagram showing a multi-layer common / independent latent space conversion means according to an embodiment of the present invention.

[0032] FIG. 7 is a diagram illustrating the process of encoding and decoding a subdivided unit latent representation according to an embodiment of the present invention.

[0033] FIG. 8 is a drawing illustrating an example of a means for converting a potential space and restoring a target space according to an embodiment of the present invention.

[0034] FIG. 9 is a drawing illustrating an example of a means for converting a potential space and restoring a target space according to an embodiment of the present invention.

[0035] FIG. 10 is a diagram illustrating a channel-unit latent space partitioning method according to an embodiment of the present invention.

[0036] FIG. 11 is a diagram illustrating a method for dividing potential space into spatial units according to an embodiment of the present invention.

[0037] FIG. 12 is a diagram illustrating a channel-unit latent space recombination method according to one embodiment of the present invention.

[0038] FIG. 13 is a diagram illustrating a spatial unit latent space recombination method according to one embodiment of the present invention.

[0039] FIG. 14 is a drawing illustrating a latent space recombination method according to one embodiment of the present invention.

[0040] FIG. 15 is a diagram illustrating a visual information encoding means based on latent expression segmentation and recombination according to an embodiment of the present invention.

[0041] FIG. 16 is a diagram illustrating a bitstream structure and a parsing method configured independently based on a Network Abstraction Layer (NAL) unit according to an embodiment of the present invention.

[0042] FIG. 17 is a diagram illustrating a bitstream structure and parsing method independently encoded based on a unique syntax element (Syntax) according to an embodiment of the present invention.

[0043] FIG. 18 is a diagram illustrating bitstream header-based identifier mapping information according to an embodiment of the present invention.

[0044] FIG. 19 is a diagram illustrating table-based identifier mapping information according to another embodiment of the present invention.

[0045] FIG. 20 is a diagram illustrating a neural network-based segmented unit latent representation encoding and decoding means according to an embodiment of the present invention.

[0046] FIG. 21 is a diagram illustrating an algorithm-based partitioned unit latent representation encoding and decoding means according to an embodiment of the present invention.

[0047] FIG. 22 is a drawing for explaining a means for determining a decoding range for a single purpose according to an embodiment of the present invention.

[0048] FIG. 23 is a diagram illustrating a means for determining a decoding range according to multiple purposes according to an embodiment of the present invention.

[0049] FIG. 24 is a schematic diagram illustrating a latent space learning means based on a semantic unit-contrast loss function between partitioned latent representations according to an embodiment of the present invention.

[0050] FIG. 25 is a diagram illustrating a specific embodiment for calculating a semantic unit-contrast loss function between partitioned latent representations according to one embodiment of the present invention.

[0051] FIG. 26 is a drawing for explaining a video editing means based on a segmented unit latent representation and the concept thereof according to an embodiment of the present invention.

[0052] FIG. 27 is a diagram illustrating an image editing structure that restores a target space by replacing a specific object included in a bitstream with another object according to an embodiment of the present invention.

[0053] FIG. 28 is a diagram illustrating an image editing structure that saves an edited bitstream by deleting a specific object included in the bitstream according to another embodiment of the present invention.

[0054] FIG. 29 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[0055] FIG. 30 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

[0056] FIG. 31 is a flowchart illustrating a latent space learning method according to one embodiment of the present invention.

[0057] FIG. 32 is a diagram illustrating an exemplary content streaming system to which an embodiment according to the present invention can be applied.

[0058] The present invention is susceptible to various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be provided illustratively for clearer explanation. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. Furthermore, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0059] In the present invention, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0060] The components shown in the embodiments of the present invention are illustrated independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for the convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the present invention as long as they do not deviate from the essence of the invention.

[0061] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, some components of this invention may not be essential components performing an essential function in this invention, but may be optional components merely for enhancing performance. This invention may be implemented by including only the components essential to embody the essence of this invention, excluding components used solely for performance enhancement, and a structure including only the essential components, excluding optional components used solely for performance enhancement, is also included within the scope of this invention.

[0062] In the embodiments, the term "at least one" may mean one of a number of 1 or more, such as 1, 2, 3, and 4. In the embodiments, the term "a plurality of" may mean one of a number of 2 or more, such as 2, 3, and 4.

[0063] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted; similar reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0064] Glossary of Terms

[0065] In the following, “image” may refer to a single picture constituting a video, or it may refer to the video itself. For example, “encoding and / or decoding of an image” may mean “encoding and / or decoding of an image”, and may also mean “encoding and / or decoding of one of the images constituting the video”.

[0066] In the following, "video" and "video" may be used interchangeably with the same meaning. Additionally, the target image may be an image to be encoded and / or an image to be decoded. Furthermore, the target image may be an input image fed into an encoding device and an input image fed into a decoding device. Here, the target image may have the same meaning as the current image.

[0067] In the following, the encoder and the image encoding device may be used interchangeably.

[0068] In the following, the decoder and the image decoder may be used interchangeably with each other.

[0069] In the following, "image," "picture," "frame," and "screen" may be used interchangeably with the same meaning.

[0070] In the following, “target block” may be an encoding target block that is the target of encoding and / or a decoding target block that is the target of decoding. Additionally, the target block may be a current block that is the target of current encoding and / or decoding. For example, “target block” and “current block” may be used interchangeably.

[0071] In the following description, "block" and "unit" may be used interchangeably. Additionally, to distinguish it from a block, "unit" may refer to a block containing a luminance (Luma) component block and a corresponding chroma (Chroma) component block. For example, a Coding Tree Unit (CTU) may consist of a single luminance component (Y) coding tree block (CTB) and two chroma component (Cb, Cr) coding tree blocks associated with it.

[0072] In the following, “sample,” “pixel,” and “pixel” may be used interchangeably with the same meaning. Here, a sample may represent a basic unit constituting a block.

[0073] In the following, “inter” and “between screens” may be used interchangeably with the same meaning.

[0074] In the following, “intra” and “in-screen” may be used interchangeably with the same meaning.

[0075]

[0076] FIG. 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present invention is applied.

[0077] The encoding device (100) may be an encoder, a video encoding device, or an image encoding device. The video may include one or more images. The encoding device (100) may sequentially encode one or more images.

[0078] Referring to FIG. 1, the encoding device (100) may include an image segmentation unit (110), an intra prediction unit (120), a motion prediction unit (121), a motion compensation unit (122), a switch (115), a subtractor (113), a converter (130), a quantization unit (140), an entropy encoding unit (150), an inverse quantization unit (160), an inverse converter (170), an adder (117), a filter unit (180), and a reference picture buffer (190).

[0079] Additionally, the encoding device (100) can generate a bitstream containing encoded information through encoding of an input image and can output the generated bitstream. The generated bitstream can be stored on a computer-readable recording medium or streamed via a wired / wireless transmission medium.

[0080] The video segmentation unit (110) can divide the input video into various forms to increase the efficiency of video encoding / decoding. That is, the input video consists of multiple pictures, and a single picture can be processed by hierarchically dividing it for compression efficiency, parallel processing, etc. For example, a single picture can be divided into one or more tiles or slices and then divided again into multiple CTUs (Coding Tree Units). Alternatively, a single picture can first be divided into multiple sub-pictures defined as groups of rectangular slices, and each sub-picture can be divided into the said tiles / slices. Here, the sub-pictures can be utilized to support the function of partially and independently encoding / decoding and transmitting the picture. Since multiple sub-pictures can each be restored individually, they have the advantage of being easy to edit in applications that configure multi-channel inputs into a single picture. In addition, the tiles can be divided horizontally to create bricks. Here, a brick can be utilized as the basic unit of parallel processing within a picture. Additionally, a single CTU can be recursively partitioned into a Quadtree (QT), and the terminal node of the partition can be defined as a Coding Unit (CU). The CU can be divided into a Prediction Unit (PU) and a Transform Unit (TU) to perform prediction and transformation. Meanwhile, the CU can be utilized as the prediction unit and / or the transformation unit itself. Here, for flexible partitioning, each CTU can be recursively partitioned into a Multi-Type Tree (MTT) as well as a Quadtree (QT). The partitioning of the CTU into a Multi-Type Tree can begin at the terminal node of the QT, and the MTT can be composed of a Binary Tree (BT) and a Triple Tree (TT).For example, the MTT structure can be classified into vertical binary splitting mode (SPLIT_BT_VER), horizontal binary splitting mode (SPLIT_BT_HOR), vertical ternary splitting mode (SPLIT_TT_VER), and horizontal ternary splitting mode (SPLIT_TT_HOR). Additionally, when splitting, the minimum block size (MinQTSize) of the quad tree for the luminance block can be set to 16x16, the maximum block size (MaxBtSize) of the binary tree to 128x128, and the maximum block size (MaxTtSize) of the triple tree to 64x64. Furthermore, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the triple tree can be set to 4x4, and the maximum depth (MaxMttDepth) of the multi-type tree can be set to 4. Additionally, to increase the encoding efficiency of the I slice, a dual tree can be applied that uses different CTU splitting structures for the luminance and chrominance components. On the other hand, in P and B slices, the luminance and color difference CTBs (Coding Tree Blocks) within the CTU can be divided into a single tree that shares a coding tree structure.

[0081] The encoding device (100) may perform encoding on an input image in an intra mode and / or inter mode. Alternatively, the encoding device (100) may perform encoding on an input image in a third mode other than the intra mode and inter mode (e.g., IBC mode, Palette mode, etc.). However, if the third mode has functional characteristics similar to the intra mode or inter mode, it may be classified as an intra mode or inter mode for convenience of explanation. In the present invention, the third mode will be classified and described separately only when a specific explanation of the third mode is required.

[0082] When intra mode is used as the prediction mode, the switch (115) can be switched to intra, and when inter mode is used as the prediction mode, the switch (115) can be switched to inter. Here, intra mode may mean an intra-frame prediction mode, and inter mode may mean an inter-frame prediction mode. The encoding device (100) can generate a prediction block for an input block of an input image. Additionally, after the prediction block is generated, the encoding device (100) can encode a residual block using the residual of the input block and the prediction block. The input image may be referred to as the current image that is the subject of current encoding. The input block may be referred to as the current block that is the subject of current encoding or the encoding target block.

[0083] When the prediction mode is an intra mode, the intra prediction unit (120) may use a sample of a block that has already been encoded / decoded around the current block as a reference sample. The intra prediction unit (120) may perform spatial prediction for the current block using the reference sample and generate prediction samples for the input block through spatial prediction. Here, intra prediction may mean intra-frame prediction.

[0084] In the intra prediction method, non-directional prediction modes such as DC mode and Planar mode, and directional prediction modes (e.g., 65 directions) may be applied. Here, the intra prediction method can be expressed as an intra prediction mode or an intra-frame prediction mode.

[0085] When the prediction mode is an inter mode, the motion prediction unit (121) can search for the region that best matches the input block from the reference image during the motion prediction process and derive a motion vector using the searched region. At this time, the search region can be used as the region. The reference image can be stored in the reference picture buffer (190). Here, the reference image can be stored in the reference picture buffer (190) when encoding / decoding of the reference image is processed.

[0086] The motion compensation unit (122) can generate a prediction block for the current block by performing motion compensation using a motion vector. Here, inter-prediction may mean inter-frame prediction or motion compensation.

[0087] The motion prediction unit (121) and motion compensation unit (122) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. To perform inter-frame prediction or motion compensation, based on the encoding unit, it can determine whether the motion prediction and motion compensation method of the prediction unit included in the corresponding encoding unit is a Skip Mode, Merge Mode, Advanced Motion Vector Prediction (AMVP) Mode, or Intra Block Copy (IBC) Mode, and can perform inter-frame prediction or motion compensation according to each mode.

[0088] In addition, based on the above-mentioned inter-frame prediction method, the AFFINE mode of sub-PU-based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and GPM (Geometric Partitioning Mode) mode of PU-based prediction may be applied. Furthermore, to improve the performance of each mode, HMVP (History based MVP), PAMVP (Pairwise Average MVP), CIIP (Combined Intra / Inter Prediction), AMVR (Adaptive Motion Vector Resolution), BDOF (Bi-Directional Optical-Flow), BCW (Bi-predictive with CU Weights), LIC (Local Illumination Compensation), TM (Template Matching), OBMC (Overlapped Block Motion Compensation), etc. may be applied.

[0089] Among these, AFFINE mode is a technology used in both AMVP and MERGE modes and also offers high encoding efficiency. Conventional video coding standards have the disadvantage of failing to properly compensate for real-world movements, such as zoom in / out and rotation, because they perform Motion Compensation (MC) by considering only the translation of blocks. To address this, a 4-parameter affine motion model using two control point motion vectors (CPMV) and a 6-parameter affine motion model using three control point motion vectors can be applied to inter-prediction. Here, CPMV is a vector representing one of the affine motion models of the top-left, top-right, or bottom-left of the current block.

[0090] The subtractor (113) can generate a residual block using the difference between the input block and the prediction block. The residual block may also be referred to as a residual signal. The residual signal may represent the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming, quantizing, or both transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be a residual signal in block units.

[0091] The transformation unit (130) can generate a transform coefficient by performing a transform on the remaining block and output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the remaining block. When a transform skip mode is applied, the transformation unit (130) may skip the transform on the remaining block.

[0092] A quantized level can be generated by applying quantization to a conversion coefficient or a residual signal. In the following embodiments, the quantized level may also be referred to as a conversion coefficient.

[0093] For example, a 4x4 luminance residual block generated through intra-frame prediction can be transformed using a Discrete Sine Transform (DST)-based basis vector, while the remaining residual blocks can be transformed using a Discrete Cosine Transform (DCT)-based basis vector. Additionally, the transformation blocks for a single block can be divided into a quad tree form using Residual Quad Tree (RQT) technology, and after performing transformation and quantization on each transformation block divided by RQT, a coded block flag (cbf) can be transmitted to increase coding efficiency in the case where all coefficients become zero.

[0094] As another alternative, the Multiple Transform Selection (MTS) technique can be applied to perform transformations using multiple transformation bases selectively. In other words, instead of splitting a CU into TUs via RQT, a function similar to TU splitting can be performed using the Sub-block Transform (SBT) technique. Specifically, SBT is applied only to inter-frame prediction blocks and, unlike RQT, can split the current block into ½ or ¼ sizes in the vertical or horizontal direction and perform a transformation on only one of the blocks. For example, if split vertically, a transformation can be performed on the leftmost or rightmost block, and if split horizontally, a transformation can be performed on the topmost or bottommost block.

[0095] In addition, Low Frequency Non-Separable Transform (LFNST), a secondary transform technique that further transforms the residual signal converted to the frequency domain through DCT or DST, can also be applied. LFNST performs additional transformation on the 4x4 or 8x8 low-frequency region in the upper left corner, thereby allowing the residual coefficients to be concentrated in the upper left corner.

[0096] The quantization unit (140) can generate a quantized level by quantizing a transformation coefficient or residual signal according to a quantization parameter (QP, Quantization parameter) and can output the generated quantized level. At this time, the quantization unit (140) can quantize the transformation coefficient using a quantization matrix.

[0097] For example, a quantizer using QP values ​​from 0 to 51 can be used. Alternatively, if the image size is larger and higher coding efficiency is required, QP values ​​from 0 to 63 can be used. Additionally, a Dependent Quantization (DQ) method using two quantizers instead of a single one can be applied. DQ performs quantization using two quantizers (e.g., Q0, Q1), but can be applied so that the quantizer to be used for the next transform coefficient is selected based on the current state through a state transition model, even without signaling information regarding the use of a specific quantizer.

[0098] The entropy encoding unit (150) can generate a bitstream and output a bitstream by performing entropy encoding according to a probability distribution on values ​​calculated by the quantization unit (140) or coding parameter values ​​calculated during the encoding process. The entropy encoding unit (150) can perform entropy encoding on information regarding a sample of an image and information for decoding an image. For example, information for decoding an image may include syntax elements, etc.

[0099] When entropy coding is applied, a small number of bits are allocated to symbols with a high probability of occurrence and a large number of bits are allocated to symbols with a low probability of occurrence, thereby representing the symbols and reducing the size of the bit sequence for the symbols to be encoded. The entropy coding unit (150) may use encoding methods such as exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) for entropy coding. For example, the entropy coding unit (150) may perform entropy coding using a Variable Length Coding (VLC) table. In addition, the entropy encoding unit (150) may perform arithmetic encoding using the derived binarization method, probability model, and context model after deriving a binarization method of the target symbol and a probability model of the target symbol / bin.

[0100] In this regard, when applying CABAC, in order to reduce the size of the probability table stored in the decoder, the table probability update method may be changed to a table update method using a simple formula. In addition, two different probability models may be used to obtain more accurate symbol probability values.

[0101] The entropy encoding unit (150) can convert a 2-dimensional block form coefficient into a 1-dimensional vector form through a transform coefficient scanning method to encode a transform coefficient level (quantized level).

[0102] Coding parameters may include information (flags, indexes, etc.) that is encoded in the encoding device (100) and signaled to the decoding device (200), such as syntax elements, as well as information derived during the encoding process or decoding process, and may refer to information required when encoding or decoding images.

[0103] Here, signaling a flag or index may mean that in an encoder, the corresponding flag or index is entropy encoded and included in a bitstream, and in a decoder, the corresponding flag or index is entropy decoded from the bitstream.

[0104] The encoded current image can be used as a reference image for other images processed later. Accordingly, the encoding device (100) can restore or decode the encoded current image again, and can store the restored or decoded image as a reference image in the reference picture buffer (190).

[0105] The quantized level can be dequantized in the dequantization unit (160) and inverse transformed in the inverse transform unit (170). The dequantized and / or inverse transformed coefficients can be added to the prediction block through the adder (117). A reconstructed block can be generated by adding the dequantized and / or inverse transformed coefficients and the prediction block. Here, the dequantized and / or inverse transformed coefficients refer to coefficients for which at least one of dequantization and inverse transformation has been performed, and may refer to the reconstructed residual block. The dequantization unit (160) and the inverse transform unit (170) can be performed as the reverse process of the quantization unit (140) and the transformation unit (130).

[0106] The restoration block may pass through a filter section (180). The filter section (180) may apply a deblocking filter, Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), Bilateral filter (BIF), LMCS (Luma Mapping with Chroma Scaling), etc., to the restoration sample, restoration block, or restoration image as a whole or part of the filtering technique. The filter section (180) may also be referred to as an in-loop filter. In this case, the term in-loop filter is also used as a name that excludes LMCS.

[0107] Deblocking filters can remove block distortion occurring at the boundaries between blocks. To determine whether to perform deblocking, the decision to apply the filter to the current block can be made based on samples contained in a few columns or rows within the block. When applying a deblocking filter to a block, different filters can be applied depending on the required deblocking filtering intensity.

[0108] To compensate for encoding errors using a sample adaptive offset, an appropriate offset value can be added to the sample value. The sample adaptive offset can correct the offset from the original image on a sample-by-sample basis for the deblocked image. One method may be to divide the samples included in the image into a certain number of regions, determine the region to be offset, and apply the offset to that region, or to apply the offset by considering the edge information of each sample.

[0109] A bilateral filter (BIF) can also correct the offset from the original image on a sample-by-sample basis for the deblocked image.

[0110] An adaptive loop filter can perform filtering based on a comparison of the reconstructed image and the original image. After dividing the samples included in the image into predetermined groups, a filter to be applied to each group can be determined, thereby performing filtering differently for each group. Information regarding whether to apply an adaptive loop filter can be signaled per coding unit (CU), and the shape and filter coefficients of the adaptive loop filter to be applied may vary depending on each block.

[0111] In LMCS (Luma Mapping with Chroma Scaling), Luma mapping (LM) refers to remapping luminance values ​​through a piece-wise linear model, and Chroma scaling (CS) refers to a technique that scales the residual values ​​of the chrominance component according to the average luminance value of the predicted signal. In particular, LMCS can be utilized as an HDR correction technique that reflects the characteristics of HDR (High Dynamic Range) video.

[0112] The restored block or restored image that has passed through the filter unit (180) can be stored in the reference picture buffer (190). The restored block that has passed through the filter unit (180) may be part of the reference image. That is to say, the reference image may be a restored image composed of the restored blocks that have passed through the filter unit (180). The stored reference image may subsequently be used for inter-frame prediction or motion compensation.

[0113] FIG. 2 is a block diagram showing the configuration according to one embodiment of a decoding device to which the present invention is applied.

[0114] The decoding device (200) may be a decoder, a video decoding device, or an image decoding device.

[0115] Referring to FIG. 2, the decoding device (200) may include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), an intra prediction unit (240), a motion compensation unit (250), an adder (201), a switch (203), a filter unit (260), and a reference picture buffer (270).

[0116] The decoding device (200) can receive a bitstream output from the encoding device (100). The decoding device (200) can receive a bitstream stored in a computer-readable recording medium or a bitstream stream streamed through a wired / wireless transmission medium. The decoding device (200) can perform decoding on the bitstream in intra mode or inter mode. Additionally, the decoding device (200) can generate a restored image or a decoded image through decoding and can output the restored image or the decoded image.

[0117] If the prediction mode used for decoding is intra mode, the switch (203) can be switched to intra. If the prediction mode used for decoding is inter mode, the switch (203) can be switched to inter.

[0118] The decoding device (200) can decode the input bitstream to obtain a reconstructed residual block and generate a prediction block. Once the reconstructed residual block and the prediction block are obtained, the decoding device (200) can generate a reconstructed block to be decoded by adding the reconstructed residual block and the prediction block. The block to be decoded may be referred to as the current block.

[0119] The entropy decoding unit (210) can generate symbols by performing entropy decoding according to the probability distribution of the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the inverse process of the entropy encoding method described above.

[0120] The entropy decoding unit (210) can convert a one-dimensional vector-shaped coefficient into a two-dimensional block-shaped coefficient through a conversion coefficient scanning method to decode a conversion coefficient level (quantized level).

[0121] The quantized level can be dequantized in the dequantization unit (220) and inversely transformed in the inverse transformation unit (230). The quantized level can be generated as a restored residual block as a result of performing dequantization and / or inverse transformation. At this time, the dequantization unit (220) can apply a quantization matrix to the quantized level. The dequantization unit (220) and the inverse transformation unit (230) applied to the decoding device can apply the same technology as the dequantization unit (160) and the inverse transformation unit (170) applied to the aforementioned encoding device.

[0122] When an intra mode is used, the intra prediction unit (240) can generate a prediction block by performing a spatial prediction on the current block using sample values ​​of already decoded blocks around the block to be decoded. The intra prediction unit (240) applied to the decoding device can apply the same technology as the intra prediction unit (120) applied to the aforementioned encoding device.

[0123] When an inter mode is used, the motion compensation unit (250) can generate a prediction block by performing motion compensation on the current block using a motion vector and a reference image stored in the reference picture buffer (270). The motion compensation unit (250) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. To perform motion compensation, it can be determined whether the motion compensation method of the prediction unit included in the corresponding encoding unit is a skip mode, merge mode, AMVP mode, or current picture reference mode based on the encoding unit, and motion compensation can be performed according to each mode. The motion compensation unit (250) applied to the decoder can apply the same technology as the motion compensation unit (122) applied to the aforementioned encoding unit.

[0124] The adder (201) can generate a restored block by adding the restored residual block and the prediction block. The filter unit (260) can apply at least one of the following to the restored block or the restored image: an inverse-LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter. The filter unit (260) applied to the decoder can apply the same filtering technology as the filter unit (180) applied to the aforementioned encoding device.

[0125] The filter unit (260) can output a restored image. The restored block or the restored image can be stored in a reference picture buffer (270) and used for inter-frame prediction. The restored block that has passed through the filter unit (260) may be part of the reference image. That is to say, the reference image may be a restored image composed of the restored blocks that have passed through the filter unit (260). The stored reference image may subsequently be used for inter-frame prediction or motion compensation.

[0126] FIG. 3 is a schematic diagram showing a video coding system to which the present invention can be applied.

[0127] A video coding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may transmit encoded video and / or image information or data to the decoding device (20) via a digital storage medium or network in the form of a file or streaming.

[0128] An encoding device (10) according to one embodiment may include a video source generation unit (11), an encoding unit (12), and a transmission unit (13). A decoding device (20) according to one embodiment may include a receiving unit (21), a decoding unit (22), and a rendering unit (23). The encoding unit (12) may be called a video / image encoding unit, and the decoding unit (22) may be called a video / image decoding unit. The transmission unit (13) may be included in the encoding unit (12). The receiving unit (21) may be included in the decoding unit (22). The rendering unit (23) may include a display unit, and the display unit may be composed of a separate device or an external component.

[0129] The video source generation unit (11) can acquire video / image through a process of capturing, synthesizing, or generating video / image. The video source generation unit (11) may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / image, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and can generate video / image (electronically). For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0130] The encoding unit (12) can encode the input video / image. The encoding unit (12) can perform a series of procedures such as prediction, conversion, and quantization for compression and encoding efficiency. The encoding unit (12) can output the encoded data (encoded video / image information) in the form of a bitstream. The detailed configuration of the encoding unit (12) can also be configured in the same way as the encoding device (100) of FIG. 1 described above.

[0131] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit (21) of the decoding device (20) via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit (22).

[0132] The decoding unit (22) can decode a video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (12). The detailed configuration of the decoding unit (22) can also be configured to be identical to the decoding device (200) of FIG. 2 described above.

[0133] The rendering unit (23) can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0134]

[0135] The visual information that can be used as input in the present invention may be in various data forms, including point clouds, meshes, Gaussian splatting, and holography. First, a point cloud can be represented as a set of points containing information in a three-dimensional space, and each point may include additional attributes such as color information and reflectance information along with position information (X, Y, Z) in the three-dimensional space. Typically, each point may be represented as a three-dimensional vector in the three-dimensional space or expanded into a vector form of three or more dimensions by means of additional attributes. As a specific example, in the case of a point cloud composed of N points containing color information for red, green, and blue (RGB) channels, each point can be represented as a six-dimensional vector including three-dimensional position information and three-channel color information, and finally, the entire point cloud can be represented in the form of a two-dimensional matrix having an N x 6 shape by integrating the information of each point.

[0136] A mesh can represent 3D space as a set of polygons connected by edges and vertices. As the number of polygons increases, the shape of the object can be represented closer to a curved surface, and a mesh can represent each polygon using at least one of the following: vertices, edges, or faces. Depending on which information is included, a mesh that combines vertex and face information is called a face-vertex mesh, while a mesh that includes point, line, and face information is called a winged-edge mesh. Mathematically, each vertex (v i ) is a position in 3D space as R 3 It can be expressed as an element of space, and a line segment (e i ) is the index number of the two vertices as R 2 It can be expressed as an element of space, and the surface (f i) is the set of each line segment as R 3 It can be expressed as elements of space, etc.

[0137] Among Gaussian splatting methods, 3D Gaussian splatting refers to a method for representing static stereoscopic image information using a Gaussian distribution and rendering it in real time. The Gaussian distribution constituting the stereoscopic information is called a 3D Gaussian, and each 3D Gaussian (g i ) may include at least one of position (μ) information, covariance matrix (Σ) information, opacity (α) information, and color (c) information as a feature. For example, each 3D Gaussian (g i ) is R 59 As an element of space, it can have a total of 59 characteristics. These 59 characteristics can be composed of 3 Position (μ) information, 7 Covariance Matrix (Σ) information, 1 Opacity (α) information, and 48 Color (c) information. In this case, the Covariance Matrix (Σ) information can be decomposed into 3-dimensional scale information and quaternion information representing 4-dimensional rotation, while the Color (c) information can be decomposed into 3 dimensions corresponding to the DC component of Spherical Harmonics (SH) and 45 dimensions corresponding to the higher-order component (based on SH degree 3). However, the 3-dimensional Gaussian (g i The types and dimensions of the characteristics possessed by ) may vary depending on the degree of the applied SH, the implementation method of Gaussian splatting, whether additional attributes (e.g., normal vectors, etc.) are included, and the present invention relates to a 3D Gaussian (g iIt is not limited to a specific characteristic configuration or dimension of ). In addition, 4D Gaussian splatting is a method for rendering dynamic stereoscopic image information in real time, and may be in the form of a 3D Gaussian with a time axis (t) added. Here, rendering means generating a 2D image that a user can view from stereoscopic image information.

[0138] Holography is a technology that reproduces complete three-dimensional images by handling the amplitude and phase information of light to resolve the vergence-accommodation conflict problem inherent in conventional stereoscopic methods. It acquires a hologram by recording interference patterns generated when object light reflected from an object interferes with a reference light, and then reproduces a three-dimensional image by illuminating this hologram with reference light. Digital holography technology acquires three-dimensional image data using image acquisition devices such as Charge-Coupled Device (CCD) cameras and reproduces holographic images using display devices such as Spatial Light Modulators (SLMs). Spatial Light Modulators can refer to any device that modulates light based on position, including beam projectors and shadow art utilizing shadows. Representative technologies for directly acquiring three-dimensional information of actual objects include self-interference technology and optical scanning holography. Additionally, Computer Generated Holography (CGH) can synthesize computer-generated holographic images by numerically simulating the propagation of light waves. Meanwhile, a hologram composed solely of phase information from complex signal data can be called a phase-only hologram.

[0139] Artificial neural network models that perform machine vision tasks such as object classification, recognition, detection, region segmentation, and tracking, or image processing tasks such as resolution enhancement and frame interpolation, can be composed of feature map extraction means that extract features from input data and task execution means that perform tasks based on the extracted features. As a specific example, object detection and classification machine tasks can be performed using the Fast R-CNN artificial neural network model. Furthermore, in the case of the Mask R-CNN model structure, which is frequently used for object region segmentation, a Feature Pyramid Network (FPN) can be used as a feature map extraction means, and a Region Proposal Network (RPN) and ROI Heads can be used as task execution means.

[0140] The feature map extracted by the feature map extraction means can be represented as a two-dimensional array having width and height. Since the feature map of a single layer consists of multiple channels, the multilayer feature map of each layer can be represented as a three-dimensional array having width, height, and the number of channels as dimensions. Specifically, when the feature map of layer k is denoted as Fk, Fk can be represented as a three-dimensional array Fk[x][y][c], where x and y represent the horizontal and vertical positions of the feature values, and c represents the channel index. In the case of a Feature Pyramid Network (FPN), the multilayer feature map P k or C k It is possible to extract feature maps for five layers, such as P2, P3, P4, P5, and P6. These multilayer feature maps have the characteristic that as the layers of the neural network deepen, the width and height become progressively smaller than the size of the input image. For example, the feature maps extracted through the FPN of Mask R-CNN can consist of 256 channels for each layer.

[0141] A feature map encoding process is essential when the feature map extraction means and the task execution means exist on physically separated, distinct devices (e.g., a mobile device and a cloud server), or when, even if they exist within a single device, the extracted feature map must be stored in a separate storage device before being utilized. For example, a ResNet-34 model structure may be located on a mobile device to extract features as a feature map extraction means, while multiple task execution means located on a cloud server may perform inference tasks such as image region segmentation, disparity map estimation, or image restoration based on the received feature map. In this way, feature map encoding is performed to minimize the amount of feature map data that needs to be transmitted or stored, while simultaneously minimizing the degradation of machine task performance.

[0142] In a feature map encoding system, a feature map extracted through a feature map extraction means can be converted into a format suitable for encoding through processes such as quantization, padding, scaling, or rearrangement via a feature map conversion means (Format Conversion). The converted feature map is compressed and encoded through a feature map encoder and transmitted in the form of a bitstream, and is restored again through a feature map decoder. At this time, existing image encoder and decoder-based structures such as HEVC or VVC, or artificial neural network (NN)-based image encoder and decoder structures, may be used as the encoder and decoder. The restored feature map can be re-converted through a feature map inverse conversion means to become a format similar to the originally extracted feature map and finally input into a task execution means. Furthermore, in order for the system to flexibly support various methods, information regarding parameters or extraction methods used in the feature map extraction process, as well as information regarding parameters or transformation methods used in the feature map transformation process, can also be encoded through a feature map encoder and finally transmitted together to a means for performing tasks and a means for reverse transformation, such as a cloud server.

[0143] In a neural network-based image encoding method that can be widely applied to the compression process of visual information, the term "image" can refer to all forms of data that humans and machines can recognize and analyze, including not only natural images acquired through a camera but also computer graphics, holographic images, feature map images extracted through neural networks, and ultrasound images. The neural network-based image encoding process may include the following steps. First, the encoder converts the input image to be encoded into a latent representation through an encoding neural network. Second, each component of the latent representation is quantized. Third, the quantized latent representation is compressed through entropy encoding based on a learnable latent representation probability model and transmitted to the decoder in the form of a bitstream. Fourth, the decoder restores the received bitstream through entropy decoding based on the same latent representation probability model as the encoder. Fifth, the decoding neural network receives the restored latent representation as input, performs an inverse transformation, and produces the final restored image. Sixth, the parameters of neural network models can be iteratively learned and updated through loss functions and backpropagation algorithms.

[0144] Meanwhile, image editing for visual information according to the present invention may encompass the process of performing object editing or context editing on various forms of input images, including not only 2D images such as photographs or videos but also 3D and 4D-based data such as the aforementioned point clouds, meshes, and Gaussian splatting. Object editing refers to editing methods that involve directly adding, deleting, or compositing and replacing objects, such as specific people or things, within an image. For example, a specific person in a 2D image photograph may be erased or a new object may be composited, and in 3D / 4D point cloud data, the shape itself may be changed by removing or supplementing parts of a specific model. In contrast, context editing is not limited to specific objects but may include overall operations such as changing the background (scene), hue, brightness, etc., of the entire image. For example, in 2D images or videos, the overall tone can be reset by adjusting contrast, color correction, noise removal, and applying High Dynamic Range (HDR) effects, and in 3D / 4D data, the overall atmosphere of the video can be newly adjusted by changing the illumination of the terrain or space or replacing the background layer.

[0145] The term "Original Space" as defined in this invention may refer to a unique vector space capable of directly representing visual information. For example, the original space of an image having a spatial resolution (width and height) of 100x100 pixels, 3 channels of red, green, and blue (RGB), and a bit depth of 8 bits is {x|0≤x≤2 8 It can be expressed as a vector space of the set of integers (Z), as in the formula {-1, x∈Z}(100×100×3). Additionally, the original space of a 3D Gaussian splatting with 59 real coefficients is R 59It can be expressed as a space. In the present invention, a latent representation refers to an abstract representation that serves as an element of the latent space and can be returned to the representation of the original space through a series of inverse transformation processes. Specific examples of such latent representations may include principal components, which are the outputs of Principal Component Analysis (PCA), or intermediate outputs of an autoencoder that receives visual information of the original space, converts it into a latent representation, and then inversely transforms it back to the original space. In the present invention, reconstruction refers to returning such latent representation back to the visual information of the original space through an inverse transformation process.

[0146] Consequently, "editing in latent space" in the present invention can collectively refer to all forms of image processing activities that apply manipulation to the latent representation itself to cause new changes or influences to the final restored visual information result of the original space. As an example of such editing in latent space, a diffusion model may be utilized, which performs editing in latent space by converting the visual information of the original space into a latent representation, adding noise to the said latent representation, and then restoring it.

[0147]

[0148] FIG. 4 is a diagram showing the configuration of a visual information encoding device and a decoding device based on latent expression partitioning and recombination according to an embodiment of the present invention.

[0149] Referring to FIG. 4, an image encoding device and an image decoding device according to one embodiment of the present invention may include a common / independent latent space conversion means, a latent space partitioning means, a partition unit latent representation encoding means, a decoding unit determination means, a partition unit latent representation decoding means, a latent space combination means, and a destination space restoration means. Each component shown in FIG. 4 is illustrated independently to indicate different characteristic functions of the device and does not imply that each component consists of separate hardware or a single software component unit.

[0150] A common / independent latent space transformation means can generate a latent representation by transforming input visual information. In this case, the semantic unit visual information may be at least one of visual information having a higher-level semantics and visual information optimized for the purpose of use.

[0151] In the present invention, 'higher-level meaning' may refer to information at a level where visual information is not viewed merely as pixel units or low-dimensional features (e.g., color, brightness, edge, etc.), but can be recognized and analyzed in concrete and semantic units such as objects (e.g., people, things), backgrounds, actions, and postures.

[0152] In addition, in the present invention, the term 'latent space capable of separating common / independent information between semantic units' refers to a latent space capable of expressing common visual information or independent visual information between different semantic units, and may mean a space capable of separating into latent spaces that express each common visual information or independent visual information without duplication. The term 'semantic unit latent representation' may refer to expressing such specific semantic unit visual information on the latent space.

[0153] A common / independent latent space transformation means according to one embodiment of the present invention receives visual information as input and performs spatial transformation through one or more neural network blocks, thereby outputting a latent representation in a latent space capable of separating common information and independent information between semantic units.

[0154] Specifically, the common / independent latent space transformation means may include, as examples of neural network blocks, residual blocks, transformer blocks, etc., and is not limited to a specific neural network structure, but may use ordinary neural network blocks or image processing techniques.

[0155] A latent space partitioning means according to one embodiment of the present invention can partition a generated latent expression to generate a latent expression (or partitioned unit latent expression) partitioned according to common information and independent information between semantic unit visual information.

[0156] A destination space restoration means according to one embodiment of the present invention receives one or more semantic unit latent representations as input and can output one or more restored semantic visual information by performing spatial transformation through one or more neural network blocks. In this case, 'destination space' refers to a space for expressing visual information that the user ultimately intends to restore, and is generally identical to the input visual information space, but may be configured differently from the input visual information space depending on the embodiment.

[0157] Through the conversion and restoration means according to the present invention, it is possible to convert visual information into a latent space in which semantic unit visual information can be effectively separated, thereby drastically minimizing the computational complexity and memory complexity that occur during visual information compression and video editing according to the purpose.

[0158] The partitioned unit latent representation encoding means can generate a bitstream by independently encoding the partitioned latent representation based on identifier mapping information.

[0159] The decoding unit determination means can determine a bitstream region to be decoded among the bitstreams based on the purpose of use and identifier mapping information. Specifically, a bitstream region optimized for at least one of machine vision and human vision can be determined as the bitstream region to be decoded.

[0160] The partitioned unit latent representation decoding means can restore the partitioned latent representation by decoding a determined bitstream region.

[0161] A latent space combination means can recombine semantic unit latent expressions from divided latent expressions.

[0162] The means of restoring the destination space can convert the recombined semantic unit latent representation into the destination space.

[0163]

[0164] FIG. 5 is a diagram illustrating semantic unit visual information according to an embodiment of the present invention.

[0165] Referring to FIG. 5, the semantic unit visual information in the present invention may be at least one of visual information having a higher-level semantic meaning and visual information optimized for the purpose of use.

[0166] Visual information having a higher-level meaning may specifically include object visual information and background visual information.

[0167] First, object visual information can refer to visual information that is semantically and spatially distinct from other objects. For example, object visual information can be represented as a set of pixels in 2D visual information and as a spatial division in 3D visual information. In this context, an 'object' can refer to a target that is recognized or understood by objectifying a specific entity, or to an identifiable target that possesses its own unique attributes among things that exist physically or can be conceived of abstractly. Furthermore, an object may consist of internal components that are related to one another and combined into a single consistent concept or entity. As a specific example, a rabbit and an apple can be defined as separate objects because they are semantically and spatially distinct from each other. Moreover, elements such as rabbit ears and a rabbit tail existing within the object called a rabbit all consistently represent the object called a rabbit; therefore, these can be considered internal components of the object called a rabbit.

[0168] Next, background visual information may refer to the surrounding area excluding the previously defined object within the overall visual information. This background visual information can influence the entire scene and play a role in assisting the recognition of visual information. Additionally, unlike objects, the internal components constituting the background may not be related to one another and may not constitute a specific or consistent concept. As a specific embodiment, in the visual information of FIG. 5, the background refers to the remaining area excluding the rabbit and apple objects; this is an element clearly distinguishable from the objects and can simultaneously function as visual information that influences the user's overall scene recognition.

[0169] Next, visual information optimized for the intended use may include visual information optimized for machine vision and visual information optimized for human vision.

[0170] Visual information optimized for machine vision can refer to visual information optimized for the performance of various machine vision tasks such as object classification, object recognition, object detection, object segmentation, and object tracking, or various image processing tasks such as super-resolution and frame interpolation. For example, as illustrated in Fig. 5, the visual information affecting the performance of object classification tasks may be limited to some visual information representing the rabbit's ears, fur color, hind legs, and forelegs, rather than visual information for the entire rabbit image, and the remaining visual information may be features unnecessary for object classification tasks.

[0171] Visual information optimized for human vision can refer to visual information that makes the restored visual information similar to the original for human perception. For example, as illustrated in Fig. 5, all visual information constituting the rabbit can largely influence how humans perceive the rabbit, so it can be said to be visual information for human vision.

[0172] As described above, the present invention distinguishes between visual information required for machine vision and visual information required for human vision according to different purposes of use, and can process them by separating them into common information and independent information among visual information units of meaning.

[0173]

[0174] FIG. 6 is a diagram showing a multi-layer common / independent latent space conversion means according to an embodiment of the present invention.

[0175] Referring to FIG. 6, the common / independent latent space transformation means and the target space transformation means may use one or more neural network blocks. Specifically, depending on the purpose of use, multiple common / independent latent space transformation means may be used to transform input time information (original space) into a latent representation.

[0176] Input time information can be converted into a first-order latent representation through a first-order common / independent latent space conversion means, and the first-order latent representation can be converted into a second-order latent representation through a second-order common / independent latent space conversion means. Through such a continuous process, it can finally be converted into an n-order latent representation through an n-order common / independent latent space conversion means.

[0177] Through a multi-layered transformation process, common information and independent information between semantic unit visual information can be extracted according to the purpose. For example, a first-order latent representation can extract only common information, and a second-order latent representation can extract only independent information. Or, as another embodiment, the first-order latent representation performs the extraction of independent information for machine vision only, and the second-order latent representation can extract both common information and independent information for human vision.

[0178] Likewise, the destination space transformation means for transforming the recombined semantic unit latent representation into the destination space may also use one or more neural network blocks as described above.

[0179]

[0180] FIG. 7 is a diagram illustrating the process of encoding and decoding a subdivided unit latent representation according to an embodiment of the present invention.

[0181] Referring to FIG. 7, the common / independent latent space transformation means and the target space restoration means of the present invention can be performed based on a Convolutional Neural Network (CNN)-based transformation. A conventional CNN includes a convolutional layer and can extract a feature map by convolutioning a kernel with input time information. The common / independent latent space transformation means or the target space restoration means can perform dimensionality reduction or dimensionality expansion of the input by adjusting the kernel stride or using a pooling layer.

[0182] Looking at an example of a CNN-based latent space transformation process illustrated in FIG. 7, a common / independent latent space transformation means (g a ) can receive visual information (x) of the original space, which is a color image with H × W × 3 dimensions, and perform a convolution operation. Specifically, the visual information (x) can be transformed into a latent representation (y) in a latent space of H / 16 × W / 16 × M dimensions (where the value of M is 192 or 320) by undergoing a convolution operation in which the spatial size is reduced by half four times at each layer. The latent representation (y) in the latent space is given by the following mathematical formula.

[0183] [Mathematical Formula 1]

[0184]

[0185] Looking at an example of a CNN-based target space restoration process illustrated in FIG. 7, the target space restoration means (g s) can receive an input tensor (ŷ) of 16×16×M dimensions (where the value of M is 192 or 320), such as a recombined semantic unit latent representation. The received input tensor undergoes four transposed convolution layer operations to expand its dimensions, and as a result, can be transformed into reconstructed visual information (x̂) having H×W×3 dimensions. The reconstructed visual information (x̂) is given by the following mathematical equation.

[0186] [Mathematical Formula 2]

[0187]

[0188] In addition, the neural network blocks constituting the CNN-based transformation and restoration means may include Generalized Divisive Normalization (GDN) layers and Inverse-GDN (IGDN) layers, which are types of feature map normalization methods. However, the present invention is not limited thereto, and GDN and IGDN may be replaced with other types of normalization methods or non-linear functions.

[0189] In addition, FIG. 6 shows such latent space transformation (g a ) and restoration(g s In addition to the ) process, for efficient compression of the latent representation (y), an additional auxiliary neural network (h) for quantization (Q), arithmetic coding (AE), arithmetic decoding (AD), and estimation of hyperprior information. a , h s It may be a whole neural network-based image encoding and decoding architecture in which ) is organically combined.

[0190]

[0191] FIG. 8 is a drawing for explaining an example of a means for converting a potential space and restoring a target space according to an embodiment of the present invention. FIG. 9 is a drawing for explaining an example of a means for converting a potential space and restoring a target space according to an embodiment of the present invention.

[0192] Referring to FIGS. 8 and 9, the common / independent latent space transformation means and the target space restoration means of the present invention can be performed based on a Multi-Layer Perceptron (MLP)-based transformation. A conventional MLP is configured to include a plurality of hidden layers between an input layer and an output layer, and each layer configured can be a fully-connected layer (FC). The common / independent latent space transformation means or the target space restoration means can perform dimensionality reduction or dimensionality expansion of input time information by adjusting the number of nodes in the hidden layer.

[0193] In particular, for visual information where spatial resolution (horizontal and vertical axes) does not exist, such as Gaussian splatting, an MLP-based transformation method may be more suitable than the aforementioned CNN-based transformation method.

[0194] Referring to FIG. 8, a common / independent latent space conversion means (g) according to one embodiment of the present invention a ) can receive an input Gaussian (x) having N x D dimensions. Here, N represents the number of 3-dimensional Gaussians constituting the input Gaussian (x), and D represents the dimension of the features of each 3-dimensional Gaussian.

[0195] Each 3-dimensional Gaussian constituting the input Gaussian (x) may include at least one of position (μ) information, covariance matrix (Σ) information, opacity (α) information, and color (c) information as a feature, as previously defined. In this case, the position (μ) information may be processed together with the input Gaussian (x) or may be processed independently through a separate encoding means (e.g., G-PCC (Geometry-based Point Cloud Compression), etc.). Accordingly, D may be the sum of all features of the 3-dimensional Gaussian (e.g., 59) or the sum of dimensions excluding some features (e.g., 56 excluding the 3-dimensional position information), and the present invention is not limited to a specific value of D.

[0196] The received input Gaussian (x) can be transformed into a latent representation (y) in latent space by passing through a neural network containing multiple fully-connected layers. In this case, the neural network can be implemented in various forms, such as a Convolutional Neural Network (CNN) and a Transformer, in addition to a Multi-Layer Perceptron (MLP) structure, and the number of layers, the hidden dimension of each layer, and the type of activation function can be designed in various ways depending on the application purpose and data characteristics.

[0197] Subsequently, means of restoring the target space (g s ) can receive an input tensor (ŷ̂), such as a latent representation (y) or a recombined semantic unit latent representation, and restore it with the characteristics of a 3D Gaussian. A target space restoration means (g) according to an embodiment of the present invention s) may be configured to restore all features of a 3D Gaussian at once, or to selectively restore only some features and combine the remaining features with features obtained through a separate path (e.g., a direct quantization path, external encoding / decoding means, etc.). The present invention is a target space restoration means (g s It is not limited to a specific configuration method of ).

[0198] Looking at an example of an MLP-based latent space transformation process illustrated in FIG. 9, a common / independent latent space transformation means (g a ) is an input Gaussian feature tensor (g) with N×56 dimensions. fea ) can be received. Here, N represents the number of 3-dimensional Gaussians, and 56 is the sum of the dimensions of the remaining features excluding the 3-dimensional position (μ) information among the previously defined 59 features; specifically, it may represent the sum of the 7-dimensional covariance matrix (Σ) information, the 1-dimensional opacity (α) information, and the 48-dimensional color (c) information (7 + 1 + 48 = 56). That is, in this embodiment, the position (μ) information of the 3-dimensional Gaussian is the common / independent latent space transformation means (g a It can be processed independently through a separate encoding / decoding means (e.g., G-PCC, etc.) without going through )

[0199] Received input Gaussian feature tensor (g fea ) undergoes dimension transformation through multiple fully connected layers (e.g., FC 1 to FC 5 layers having dimensions of 56 x 256, 256 x 256, etc.) and can consequently be transformed into a latent representation (y) in a latent space having N×256 dimensions.

[0200] y=g a (x)

[0201] Looking at an example of an MLP-based target space restoration process illustrated in FIG. 9, the target space restoration means (g s) receives an N×256 dimensional input tensor (ŷ), such as a recombined semantic unit latent representation, and can selectively restore only the color (c) information among the features of a 3-dimensional Gaussian. That is, in this embodiment, g s is the color tensor(x̂ color It outputs only (N × 48 dimensions), and the geometric tensor (x) corresponding to the covariance matrix (Σ) information and opacity (α) information. geo ) is this g a ,g s Processed through a separate direct quantization path without going through (x̂ geo The received input tensor (ŷ) can be composed of multiple fully connected layers (e.g., multiple FC layers with dimensions of 256 x 256, 256 x 45, etc.).

[0202] x̂ color =g s (ŷ)

[0203] Afterwards, output tensor(x̂ color N×8 dimensional geometric and local information (x̂) obtained through linear operations on ) geo By combining ), a restored Gaussian (x̂) having the same N×56 dimensions as the original space can finally be obtained.

[0204] However, the number of shared fully connected layers and the dimension of each layer, the configuration of the first and second restoration paths, the method of decomposing color (c) information (e.g., whether DC components and higher-order components are separated, setting of SH degree, etc.), and the geometric tensor (x̂ geo The direct quantization method of ) and the path for obtaining position (μ) information are merely specific embodiments of the present invention, and the present invention is not limited thereto.

[0205]

[0206] FIG. 10 is a diagram illustrating a channel-unit latent space partitioning method according to an embodiment of the present invention.

[0207] Referring to FIG. 10, the potential space partitioning means of the present invention is R (d1×d2×d3×...×dn) It can be performed based on a partitioning method that can divide the latent space expressed as such without overlap. The latent space partitioning means can generate latent representations divided by common information and independent information between one or more semantic unit visual information by partitioning the transformed latent representation without overlap.

[0208] In this case, the spatial partitioning method can be applied differently depending on the form of the dimension constituting the transformed latent representation.

[0209] For example, if the dimension constituting the transformed latent representation is a 3D tensor consisting of spatial resolution (horizontal, vertical) and channel axes, the latent space partitioning means may use a channel-wise partitioning method as shown in FIG. 9.

[0210] Specifically, the latent representation transformed from the visual information of the original space is R (C×H×W) Assuming that it has dimensions (where C represents the number of channels, H represents the height, and W represents the width), the latent space partitioning means can partition the latent representation without overlap based on the channel axis (C) while maintaining the size of the spatial resolution axes (H, W).

[0211] Consequently, R (C×H×W) The latent representation of the dimension is R, as illustrated in FIG. 10. (C1×H×W) ,R (C2×H×W) , R (C3×H×W) It can be output by dividing into three partitioned latent representations with dimensions. In this case, since the sum of the number of channels in each partitioned latent representation must be equal to the total number of channels of the original latent representation before partitioning, it is as follows:

[0212] [Mathematical Formula 3]

[0213]

[0214] That is, the latent space is divided such that C1+ C2+ C3= C holds, and through this, semantic unit visual information mixed within the input visual information can be completely separated into a common information area and one or more independent information areas in the latent space without loss or duplication of information.

[0215]

[0216] FIG. 11 is a diagram illustrating a method for dividing potential space into spatial units according to an embodiment of the present invention.

[0217] Referring to FIG. 11, a latent space partitioning means according to one embodiment of the present invention can generate latent representations partitioned by common information and independent information between one or more semantic unit visual information by partitioning the converted latent representation without overlap. In addition to the channel unit partitioning described above in FIG. 10, the latent space partitioning means may use a spatial unit partitioning method based on the spatial resolution axis of the converted latent representation.

[0218] Specifically, the latent representation transformed from the visual information of the original space is R (C×H×W) Assuming that it has dimensions (where C represents the number of channels, H represents the height, and W represents the width), the latent space partitioning means can partition the latent representation without overlap based on the spatial resolution axes (H, W) while maintaining the size of the channel axis (C).

[0219] Consequently, R (C×H×W) The latent representation of the dimension is R, as shown in FIG. 11. (C×H1×W1) ,R (C×H2×W2) , R (C×H3×W3) It can be divided into partitioned latent representations with dimensions and output. In this case, since the sum of the spatial resolutions of each partitioned latent representation must be equal to the total spatial resolution of the original latent representation before partitioning, it satisfies the condition of the following mathematical formula.

[0220] [Mathematical Formula 4]

[0221]

[0222] In addition, according to another embodiment of the present invention, the potential space partitioning means may use a partitioning method in the form of a combination of the channel unit partitioning and space unit partitioning described above.

[0223] Specifically, R (C×H×W) When a dimensional latent representation is partitioned based on both the channel and spatial resolution axes, the partitioned latent representation is R (C1×H1×W1) ,R (C1×H2×W2) , R (C2×H3×W3) , R (C2×H4×W4) It can be output in forms having dimensions such as. When partitioning into such channel-space combination units, the following mathematical condition is satisfied.

[0224] [Mathematical Formula 5]

[0225]

[0226] Through the described spatial unit partitioning and combination-type partitioning, semantic unit visual information mixed within the input visual information can be completely separated into a common information area and one or more independent information areas in the latent space without any loss or duplication of information.

[0227]

[0228] FIG. 12 is a diagram illustrating a channel-unit latent space recombination method according to one embodiment of the present invention.

[0229] Referring to FIG. 12, a latent space combination means according to one embodiment of the present invention receives one or more decoded divided latent representations as input, and R (d1×d2×d3×...×dn) One or more semantic unit latent representations can be output by recombining them in the form of.

[0230] The latent space combination means is that the combined latent space is R (C×H×W)To recombine into a latent representation having dimensions (where C represents the number of channels, H represents the height, and W represents the width), a channel-unit recombination method as shown in FIG. 12 can be used.

[0231] Specifically, the latent space combination means can combine divided latent representations having the same size of spatial resolution axes (H, W) based on the channel axis (C). For example, this R (C1×H×W) ,R (C2×H×W) , R (C3×H×W) Partitioned latent representations with dimensions undergo a channel-level recombination process to finally R (C×H×W) It can be output as a single, complete semantic unit latent representation with dimensions.

[0232] At this time, since the total number of channels of the latent expressions generated through channel-unit recombination must be equal to the sum of the number of channels of each combined divided latent expression, the following mathematical formula condition is satisfied.

[0233] [Mathematical Formula 6]

[0234]

[0235] In other words, the partitioned latent representations are combined along the channel axis so that C1 + C2 + C3 = C holds, and through this, partial information that was independently separated and decoded can be recombined into semantic unit latent representations.

[0236]

[0237] FIG. 13 is a diagram illustrating a spatial unit latent space recombination method according to one embodiment of the present invention.

[0238] Referring to FIG. 13, a latent space combination means according to one embodiment of the present invention receives one or more decoded divided latent representations as input, and R (d1×d2×d3×...×dn) One or more semantic unit latent representations can be output by recombining in the form of. The latent space combination means has a combined latent space R (C×H×W)To recombine into a latent representation having dimensions (where C represents the number of channels, H represents the height, and W represents the width), a spatial unit recombination method as illustrated in FIG. 13 may be used. Specifically, the latent space combination means may combine divided latent representations having the same size of the channel axis (C) based on the spatial resolution axes (horizontal W, vertical H). For example, R (C×H1×W1) ,R (C×H2×W2) , R (C×H3×W3) Partitioned latent representations with dimensions undergo a spatial unit recombination process to finally R (C×H×W) It can be output as a single, complete semantic unit latent representation having dimensions. In this case, since the total height and width of the latent representation generated through spatial unit recombination must be equal to the sum of the heights and widths of each combined divided latent representation, the following mathematical condition is satisfied.

[0239] [Mathematical Formula 7]

[0240]

[0241] That is, the partitioned latent representations are combined along the spatial axis direction such that H1+ H2+ H3= H and W1+ W2+ W3= W hold, and through this, partial information that was independently separated and decoded can be recombined into semantic unit latent representations.

[0242] Additionally, a latent space combination means according to one embodiment of the present invention receives one or more decoded divided latent representations as input, R (d1×d2×d3×...×dn) One or more semantic unit latent representations can be output by recombining in the form of. The latent space combination means has a combined latent space R (C×H×W)To recombine into a latent representation having dimensions (where C represents the number of channels, H represents the height, and W represents the width), a method combining channel-unit and spatial-unit recombination can be used. Specifically, the latent-space combination means can combine latent representations in which both the channel axis (C) and the spatial resolution axes (H, W) are complexly divided based on each axis. For example, R (C1×H1×W1) ,R (C1×H2×W2) , R (C2×H1×W1) ,R (C2×H2×W2) Dimensional partitioned latent representations undergo a complex recombination process at the channel and spatial unit levels to finally R (C×H×W) It can be output as a single, complete semantic unit latent representation with dimensions.

[0243]

[0244] FIG. 14 is a drawing illustrating a latent space recombination method according to one embodiment of the present invention.

[0245] Referring to FIG. 14, a latent space combination means according to one embodiment of the present invention receives one or more decoded divided latent representations as input, and R (d1×d2×d3×...×dn) One or more semantic unit latent representations can be output by recombining in the form of. The latent space combination means has a combined latent space R (C×H×W) To recombine into a latent representation having dimensions (where C is the number of channels, H is the height, and W is the width), a composite recombination method using dummy values ​​can be used.

[0246] In particular, when combining using only partial partitioned latent representations, the latent space combination means can perform spatial combination by padding the excluded space with a constant value such as '0' or omitting the corresponding dimension.

[0247] Specifically, when recombining using only some partitioned latent representations, a dummy value tensor with a constant value such as 0 in the empty area may be combined to match the dimension of the latent representation before partitioning. In this case, since the total number of channels and resolution size of the latent representation generated through composite recombination must be equal to the sum of the dimensions of each combined partitioned latent representation and the dummy tensor, the condition of Equation 8 below is satisfied.

[0248] [Mathematical Formula 8]

[0249]

[0250] In other words, the divided latent representations are combined in the channel and spatial axis directions so that a sum is established for each axis, and through this, partial information that was decoded separately in multiple dimensions can be recombined into complete semantic unit latent representations.

[0251] Through such division and combination means, the present invention can drastically separate semantic unit visual information mixed within input visual information in a latent space and can recombine only the necessary semantic unit latent representations, thereby maximizing compression efficiency and further providing an excellent effect of minimizing image editing complexity.

[0252]

[0253] FIG. 15 is a diagram illustrating a visual information encoding means based on latent expression partitioning and recombination according to an embodiment of the present invention.

[0254] Referring to FIG. 15, a time information encoding and decoding system according to one embodiment of the present invention may be configured to include a partition unit latent representation encoding means, a decoding unit determination means, and a partition unit latent representation decoding means, and may independently encode and decode only the necessary time information according to the purpose of use based on identifier mapping information.

[0255] Specifically, the partition unit latent representation encoding means can output a bitstream partitioned into partition units by independently encoding the latent representation (partition unit latent representation) partitioned by the latent space partitioning means based on identifiers and identifier mapping information. Here, a semantic unit identifier refers to a signal or information mapped one-to-one with a higher-level semantic unit, such as an object or background, and a partition unit identifier refers to a signal or information mapped one-to-one with a partition unit latent representation to distinguish bitstream sections. The identifier is encoded together with the latent representation as supplementary information. The identifier may be included in the bitstream header or transmitted in a separate table format.

[0256] In addition, identifier mapping information is information representing the identifier mapping relationship between semantic units and partition units. Semantic units can be estimated through identifier mapping information. Identifier mapping information can be structured in the form of a hash table based on a hash function that maps semantic unit identifiers (A, B), which are keys, to partition unit identifiers, which are result values ​​(e.g., semantic unit A is 1 and 3, and B is 2 and 3). Furthermore, identifier mapping information is not limited to this hash table form only; depending on the purpose of data processing and retrieval, it can be implemented and extended into various data structure forms such as tree structures, other data structures based on key-value pairs, arrays, or associated lists. This identifier mapping information may be encoded and included in a bitstream, or it may be provided in a form pre-stored in a decoder without encoding.

[0257] Encoding of partition unit latent representations can be performed through various encoding methods, such as conventional neural network-based entropy methods and methods utilizing standard compression encoding techniques.

[0258] In the subsequent decoding process, the decoding unit determination means can determine a bitstream section to be decoded according to one or more intended uses among the encoded bitstreams. The decoding unit determination means can determine one or more semantic units to be decoded according to the intended use based on identifier information and identifier mapping information, and determine the corresponding bitstream section as the bitstream to be decoded. For example, by using identifier mapping information defined in the form of a hash table to determine semantic units according to the intended use, and by outputting segmentation unit identifiers mapped to the semantic unit identifiers to be restored, only the bitstream section mapped to the corresponding segmentation unit identifier can be determined as the decoding target. Through this, partial decoding can be performed, in which only a portion of the entire bitstream from which parts unnecessary for the intended use have been excluded is decoded. Alternatively, the determination of a decoding unit according to the intended use may not be performed. Through this, full decoding can be performed, in which the entire bitstream is decoded.

[0259] Finally, the partitioned unit latent representation decoding means independently decodes the bitstream region determined as the decoding target by the decoding unit determination means for each semantic unit and outputs the partitioned latent representation.

[0260] Decoding of partitioned unit latent representations can be performed through various decoding methods, such as conventional neural network-based entropy methods and methods utilizing standard compression coding techniques.

[0261] Through the above configuration, the present invention can prevent unnecessary overhead caused by encoding and decoding all bitstreams regardless of the purpose of use, as in conventional methods. That is, by encoding the bitstream so that it can be divided into partitioned potential representation units, and selectively decoding only a portion of the bitstream area according to the purpose of use during decoding, the processing complexity can be drastically reduced.

[0262]

[0263] FIG. 16 is a diagram illustrating a bitstream structure and a parsing method configured independently based on a Network Abstraction Layer (NAL) unit according to an embodiment of the present invention.

[0264] Referring to FIG. 16, the division unit latent representation encoding means of the present invention can encode the divided unit latent representation into separate NAL units to divide the entire bitstream into division units. The NAL unit is a basic unit constituting the bitstream and may consist of an NAL header and a Raw Byte Sequence Payload (RBSP).

[0265] Specifically, the RBSP is not only an area where the divided potential representation is encoded, but may also contain actual data such as slices, parameter sets, and metadata, and zero-byte insertion and emulation prevention techniques may be applied for parsing. In addition, a byte stream NAL unit form may be used in which a 'start code' syntax element in the form of a prefix is ​​added to the front of the NAL unit to distinguish each NAL unit, and one or more zero bytes ('0x00') may be inserted at the front and back of the byte stream NAL unit considering the length of the NAL unit.

[0266] Multiple NAL units configured in this manner are combined consecutively to form the entire bitstream, and the type and priority of the corresponding data can be identified through unique header information, such as segmentation unit identifiers included in the header of each NAL unit. As an example of this configuration, in conventional video codecs such as HEVC, an integer number of Coding Tree Units (CTUs) are grouped to define a segment, and encoded slice segments can be processed by encapsulating them into a single NAL unit.

[0267] Consequently, during the decoding process, the parsing means can first decode only the header information of the NAL unit without needing to decode the entire bitstream, and selectively decode only the NAL unit corresponding to the identifier determined as the recovery target by the decoding unit determination means.

[0268]

[0269] FIG. 17 is a diagram illustrating a bitstream structure and parsing method independently encoded based on a unique syntax element (Syntax) according to an embodiment of the present invention.

[0270] Referring to FIG. 17, the division unit latent expression encoding means of the present invention defines a unique syntax element for distinguishing between latent expressions as an identifier and encodes it, thereby dividing the entire bitstream into division units. Specifically, the division unit latent expression encoding means can clearly distinguish each section by adding an identifier syntax element between the divided latent expressions within the RBSP constituting the NAL unit. At this time, the division unit identifier may be configured by adding a specific code before or after the identifier to further facilitate mutual distinction.

[0271] For a bitstream divided into sections by unique identifier syntax elements within the RBSP as described above, the parsing process can be performed by sequentially decoding and searching the RBSP areas of the bitstream. If an identifier appears during the search, the decoding means may decode it only if the identifier corresponds to the identifier determined as a recovery target by the decoding unit determination means. On the other hand, if the discovered identifier is not a recovery target, the operation of skipping the corresponding area of ​​the bitstream without decoding it until the next recognizable identifier appears may be performed.

[0272]

[0273] FIG. 18 is a diagram illustrating bitstream header-based identifier mapping information according to an embodiment of the present invention.

[0274] Referring to FIG. 18, a partition unit latent expression encoding means according to one embodiment of the present invention can encode by defining identifier mapping information within the header of a NAL unit. Specifically, the partition unit latent expression encoding means can perform encoding by including a semantic unit identifier to which the partition unit belongs, along with a specific partition unit identifier, within the header area. Through this, the decoding means can immediately identify the mapping relationship between the semantic unit and the partition unit by parsing the header without separate table information.

[0275]

[0276] FIG. 19 is a diagram illustrating table-based identifier mapping information according to another embodiment of the present invention.

[0277] Referring to FIG. 19, the subdivision unit latent representation encoding means of the present invention can process identifier mapping information representing the mapping relationship between a semantic unit and a subdivision unit in a table form. The identifier mapping information in the table form is additional information required to restore the subdivided latent representation, and may be encoded and included as separate data within a bitstream or stored and maintained within a decoder in the form of a predefined table.

[0278] Specifically, the identifier mapping table can be configured in the form of a hash-based mapping table that links semantic unit identifiers with their corresponding segmentation unit identifiers based on a specific hash value. Furthermore, identifier mapping information is not limited to this hash table format; depending on the purpose of data processing and retrieval, it can be implemented and extended into various data structure forms, such as tree structures, other data structures based on key-value pairs, arrays, or associated lists.

[0279]

[0280] FIG. 20 is a diagram illustrating a neural network-based segmented unit latent representation encoding and decoding means according to an embodiment of the present invention.

[0281] Referring to FIG. 20, the segmented unit latent expression encoding means and the segmented unit latent expression decoding means according to one embodiment of the present invention can process segmented unit latent expressions using a neural network-based entropy encoding and decoding method.

[0282] Specifically, the neural network-based partition unit latent representation encoding and decoding means may consist of quantization, probability distribution estimation, arithmetic encoding, and arithmetic decoding means.

[0283] In the encoding process, the partition unit latent representation encoding means can generate a quantized partitioned latent representation by first quantizing the partitioned latent representation received from the latent space partitioning means. At the same time, the encoding means can estimate the probability distribution of the corresponding data by utilizing a hyperprior-based Gaussian probability model (N(μ,σ)) or a non-parametric factorized prior model. Subsequently, a final compressed bitstream can be generated and output by performing arithmetic encoding on the quantized partitioned latent representation based on the estimated probability distribution information.

[0284] In the corresponding decoding process, the partitioned unit latent representation decoding means can perform arithmetic decoding on the received bitstream based on a probability distribution estimated identically to that of the encoding process. Through this, the bitstream can be accurately restored to its original quantized partitioned latent representation. Through this neural network-based precise probability estimation and arithmetic encoding / decoding process, the present invention can effectively compress the partitioned latent representation according to the characteristics of the data for transmission or storage.

[0285]

[0286] FIG. 21 is a diagram illustrating an algorithm-based partitioned unit latent representation encoding and decoding means according to an embodiment of the present invention.

[0287] Referring to FIG. 21, the partition unit latent expression encoding means and the partition unit latent expression decoding means of the present invention may use an algorithm-based entropy coding method as a means for encoding and decoding a partition unit latent expression. Algorithm-based coding methods may include methodologies such as rANS (Range variant of Asymmetric Numerical System), CABAC (Context-based Adaptive Binary Arithmetic Coding), and CAVLC (Context-based Adaptive Variable Length Coding).

[0288] Specifically, the rANS method can be a type of entropy coding that generates a compressed bitstream based on the probability distribution of each symbol within discrete data. The encoding and decoding processes of the rANS method can be performed through specific mathematical formulas.

[0289] First, in the encoding process, the input symbol can be defined as 's', the probability of the symbol as 'p', and the encoded numeric value for signal transmission as 'x'. The encoding means can produce an encoded numeric value (x) by repeatedly performing operations based on the input symbol (s) and the probability (p) of the corresponding symbol until the end of the data (EOS) is reached. Specifically, looking at the operation algorithm of the encoding process, when the input symbol s is 0, the encoded numeric value x can be produced using the formula x = ceil((x+1) / (1-p)) - 1, and when the input symbol s is 1, the encoded numeric value x can be produced using the formula x = floor(x / p). The numeric value x produced through the operation can be output from the encoder to form a bitstream.

[0290] In the corresponding decoding process, the encoded numeric value received from the encoder can be defined as 'x', the probability of the symbol as 'p', and the decoded symbol as 's'. The decoding means can output the original decoded symbol (s) by repeatedly performing inverse operations based on the information of the encoded numeric value (x) and probability (p) until the end of the data (EOS) is reached. Specifically, regarding the operation algorithm of the decoding process, the decoded symbol s can be calculated first through the formula s = ceil((x+1) p) - ceil(xp). After calculating the symbol s, the numeric value x can be updated to proceed with the continuous decoding process. At this time, if the restored symbol s is 0, the formula x = x - ceil(xp) is applied, and if the restored symbol s is 1, the formula x = ceil(xp) is applied to update the numeric value x, respectively.

[0291] Alternatively, the segmented unit latent representation encoding means and the segmented unit latent representation decoding means according to one embodiment of the present invention may use an encoding method based on a Standard Video Codec as a means for encoding and decoding the segmented unit latent representation. In this case, the Standard Video Codec may refer to a conventional Standard Video Codec such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

[0292] Specifically, a series of processing steps by a standard video codec-based segmented unit latent representation encoding / decoding means may consist of the following sequential steps.

[0293] First, the segmented unit latent representation encoding means can perform processes such as quantization, padding or scaling, or rearrangement on the input segmented unit latent representation to convert the segmented unit latent representation into a format suitable for the encoder of a standard video codec.

[0294] Next, the segmented unit latent representation encoding means can generate a bitstream by compressing the converted latent representation through an encoder of a standard video codec. Correspondingly, the segmented unit latent representation decoding means can decode the received bitstream through a decoder of a standard video codec.

[0295] Next, the partition unit latent representation decoding means can output the final partition unit latent representation by inversely converting the partition unit latent representation that has undergone the decoding process back to its original form prior to format conversion.

[0296] Through an algorithm-based coding method that follows such clear formulas, the present invention can efficiently compress and precisely restore divided latent representations in accordance with the symbol probability distribution characteristics of the corresponding data.

[0297]

[0298] FIG. 22 is a drawing for explaining a means for determining a decoding range for a single purpose according to an embodiment of the present invention.

[0299] Referring to FIG. 22, a decoding unit determination means (or decoding range determination means) according to one embodiment of the present invention can determine a decoding target according to a specific single use purpose by using identifier mapping information that is encoded and included in a bitstream or is pre-stored in a decoder.

[0300] Specifically, the means for determining the decoding unit can determine the segmented unit identifier to be decoded in the following sequential steps to achieve a specific single use purpose, such as machine vision. First, a semantic unit identifier requiring decoding according to the use purpose (e.g., 'A', which is time information required for machine vision) can be determined. Next, identifier mapping information defined in the form of a hash table, etc., can be decoded from a bitstream or loaded from a decoder. Subsequently, by referring to the decoded or loaded identifier mapping information, all segmented unit identifiers (e.g., identifiers '1' and '3') mapped to the previously determined semantic unit identifier ('A') can be derived, and these can be determined as the final decoding targets.

[0301] Furthermore, embodiments of such decoding range determination means according to the present disclosure may be freely combined with various identifier encoding methods and identifier mapping information embodiments, such as bitstream header-based or table-based methods described above.

[0302] Consequently, depending on the decoding range determined in this way, the partition unit latent representation decoding means can decode part or all of the received bitstream. That is, instead of unconditionally decoding the entire bitstream, by selectively partially decoding only specific areas (areas corresponding to partition unit identifiers 1 and 3) that are suitable for the purpose of use, processing efficiency can be increased by omitting the decoding operation of visual information (e.g., information 'B' and partition unit identifier '2' mapped thereto, which is necessary only for human vision) that is unnecessary for achieving a single purpose.

[0303]

[0304] FIG. 23 is a diagram illustrating a means for determining a decoding range according to multiple purposes according to an embodiment of the present invention.

[0305] Referring to FIG. 23, a decoding unit determination means (or decoding range determination means) according to one embodiment of the present invention can determine a decoding target according to multiple purposes by using identifier mapping information that is encoded and included in a bitstream or is pre-stored in a decoder.

[0306] Specifically, the means for determining the decoding unit can determine the segmented unit identifiers to be decoded in the following sequential steps to achieve multiple usage purposes, such as supporting both human vision and machine vision. First, semantic unit identifiers requiring decoding for all usage purposes (e.g., 'A', which is visual information required for machine vision, and 'B', which is visual information required for human vision) can be determined. Next, identifier mapping information defined in the form of a hash table, etc., can be decoded from a bitstream or loaded from a decoder. Subsequently, by referring to the decoded or loaded identifier mapping information, all segmented unit identifiers (e.g., identifiers '1', '2', '3') mapped to all previously determined semantic unit identifiers ('A', 'B') can be derived, and these can be determined as the final decoding targets.

[0307] At this time, the decoding unit determination means can construct the final set of segmented unit identifiers required for restoring each semantic unit potential representation by taking the union of sets of segmented unit identifiers corresponding to each semantic unit identifier.

[0308] Furthermore, embodiments of such decoding range determination means according to the present disclosure may be freely combined with various forms of identifier and identifier mapping information embodiments, such as bitstream header-based or table-based embodiments described above.

[0309] Consequently, depending on the decoding range determined in this way, the partitioned unit latent representation decoding means can decode part or all of the received entire bitstream.

[0310] Additionally, to support cases where only specific information (e.g., objects, etc.) must always be encoded depending on the application field, or where it is necessary to select which information to encode from the encoder stage, the present invention may be configured to preemptively determine the encoding unit at the encoder and selectively encode only the required partial information.

[0311]

[0312] FIG. 24 is a schematic diagram illustrating a latent space learning means based on a semantic unit-contrast loss function between partitioned latent representations according to an embodiment of the present invention.

[0313] Referring to FIG. 24, a latent space learning means according to one embodiment of the present invention can learn to represent common information and independent information between semantic unit visual information without overlap on different divided latent spaces by learning the neural network parameters of a common / independent latent space transformation means and a target space restoration means to minimize the semantic unit-contrast loss function between divided latent representations.

[0314] Specifically, the latent space learning means may be configured to include a common / independent latent space transformation means, a latent space partitioning means, a target space restoration means, a means for calculating a loss function per semantic unit, a means for calculating a semantic unit-contrast loss function between partitioned latent representations, and a neural network parameter update means.

[0315] First, the means for calculating the loss function per semantic unit is one or more partitioned latent representations (y) partitioned through the latent space partitioning means. p  Visual information (x̂) restored from (p=1,…,P) through the destination space restoration means p  (p=1,…, P)) can be received. The means for calculating the loss function per semantic unit takes the restored visual information as input and the semantic unit objective (T kBy calculating the respective loss functions that can be optimized for (k=1,…, K), K×P loss functions for each semantic unit (L Tk (x̂ p )) can be derived and output. In this case, the loss function that can be optimized for the semantic unit purpose can calculate the error with the original visual information, the error with the target label, or cross-entropy, and the error between the original visual information and the restored visual information can utilize MSE, L1, LPIPS, DISTS, MS-SSIM, SSIM, etc.

[0316] Next, the means for calculating the semantic unit-contrast loss function between partitioned latent representations is the output K×P semantic unit-specific loss function (L Tk (x̂ p )) and predefined weight parameters (c k,p , w k ) receiving ) and based on their linear combination, semantic unit-contrast loss function (L between partitioned latent representations c It can calculate and output ). Semantic unit-contrast loss function (L c ) can be calculated by the following mathematical formula 9.

[0317] [Mathematical Formula 9]

[0318]

[0319] Here, c k,p maximizes the corresponding loss function (c k,p Minimize whether to =-1)(c k,p =1) is a directional parameter that determines whether to do it, and w k is the weight of the loss function for each semantic unit (0 <w k It means <1). In particular, the semantic unit-contrast loss function (L c ) is c for at least one partitioned latent representation (p). k,p Set so that the semantic unit k satisfying is unique (i.e., for the remaining k, c k,pBy setting it to -1, separated latent representations can be induced to clearly compare information and learn without duplication.

[0320] A means for calculating a loss function per semantic unit according to one embodiment of the present invention may utilize a loss function for maximizing perceptual quality. The loss function for maximizing perceptual quality may include at least one of a pixel similarity-based loss function and a feature space-based perceptual quality loss function.

[0321] First, pixel similarity-based loss functions are a method traditionally widely used for image restoration; however, when used alone, they can cause blurring or limit the improvement of perceptual sharpness. Examples of pixel similarity-based loss functions include the L1 loss function (L1(x,x̂)) and the MSE loss function (L MSE (x,x̂)), and SSIM (Structural Similarity Index Measure) loss function (L SSIM (x, x̂)) etc., which can be calculated as shown in the following mathematical formulas 10 to 12. Here, x represents the original time information, and x̂ represents the restored time information.

[0322] [Mathematical Formula 10]

[0323]

[0324] [Mathematical Formula 11]

[0325]

[0326] [Mathematical Formula 12]

[0327]

[0328]

[0329] Next, the feature space-based cognitive quality loss function can be calculated using features extracted from the internal layers of a pre-trained deep learning model (e.g., VGG, ResNet, etc.). This reflects the characteristic that human visual perception is more sensitive to high-dimensional features such as edges, textures, and patterns than to differences at the pixel level. In other words, by reducing the difference in the distribution of high-dimensional features between the output image and the ground image, natural and clear restoration results can be induced from a human perspective, which can be calculated as shown in the following mathematical formula.

[0330] [Mathematical Formula 13]

[0331]

[0332] (here, ϕ l represents the feature map extracted from the l-th layer of a pre-trained neural network, and α l represents the loss weight of each layer)

[0333] A means for calculating a loss function per semantic unit according to an embodiment of the present invention may utilize a loss function for maximizing machine vision task performance. The loss function for maximizing machine vision task performance may refer to a loss function designed to maximize the performance of conventional machine vision tasks such as classification, object detection, regression, segmentation, and 2D / 3D pose estimation. To calculate such a loss function, a neural network model f optimized for a specific machine vision task is used. task It can be utilized. The specific method for calculating the loss function for each task is as follows.

[0334] Cross Entropy Loss (L) is used as the loss function to maximize classification performance. CE) can be used. This is used to classify objects within a video or image into a specific class c, and the actual label y c and model f task (x̂) predicted probability distribution (ŷ c It measures the difference between ) from the perspective of information theory and can be calculated as shown in the following mathematical formula.

[0335] [Mathematical Formula 14]

[0336]

[0337] (Here, C represents the total number of classes, and y_c is the actual label (an indicator corresponding to class c in one-hot encoding, typically 0 or 1), ŷ c represents the probability of class c predicted by the machine vision task model based on reconstructed visual information.

[0338] Bounding box regression loss function (L bbox ) can be used. This is a loss function for predicting the object's location (bounding box), and the predicted bounding box (B̂ = (x min , y min , x max , y max To reduce the coordinate difference between )) and the correct box (B), a conventional Smooth L1 loss function can be applied and calculated as shown in the following mathematical formula.

[0339] [Mathematical Formula 15]

[0340]

[0341] Third, pixel-wise cross-entropy (L) is used as the loss function to maximize segmentation performance. seg-ce ) and die loss (Dice Loss, L dice...can be used. In the task of classifying each pixel in an image into a specific class including the background, pixel-level cross-entropy calculates the error on a pixel-level basis, and dice loss can be used to help extract accurate object regions in situations where there is severe imbalance between classes (e.g., medical image segmentation where the background occupies more than 90%). These can be calculated as shown in Equation 16 and Equation 17, respectively.

[0342] [Mathematical Formula 16]

[0343]

[0344] (Here, N represents the total number of pixels, C represents the number of segmentation mask classes, and y i,c is the one-hot encoded value for the actual class of the i-th pixel, ŷ i,c represents the class probability of the i-th pixel predicted by the model)

[0345] [Mathematical Formula 17]

[0346]

[0347] (here, p i is the i-th pixel value of the segmentation mask predicted by the model (a continuous value between 0 and 1), g i ε represents the i-th pixel value of the actual (ground truth) mask (typically 0 or 1), and ϵ represents a numerical stabilization term to prevent division by zero.

[0348] Joint coordinate regression loss function (L) as the loss function to maximize 2D / 3D pose estimation performance pose ) or heatmap-based loss function (L heatmap ) can be used. The joint coordinate regression loss function is the correct answer P of the human joint position (keypoint) in the visual information. j and predicted value P̂ jTo precisely estimate joint-specific coordinates using the error between them, the error is calculated using the L1 or Smooth L1 method, and the heatmap-based loss function provides an advantageous effect for stable learning and handling partial occlusion by predicting and comparing the vicinity of the joint position with a probability distribution (heatmap) instead of coordinate regression. These can be calculated as shown in Equation 18 and Equation 19, respectively.

[0349] [Mathematical Formula 18]

[0350]

[0351] (Here, J is the number of joints (key points), P j is the actual joint coordinate, P̂ j represents predicted joint coordinates)

[0352] [Mathematical Formula 19]

[0353]

[0354] (Here, Ĥ j is the 2D heatmap for joint j predicted by the model, H j ) refers to the ground truth heatmap mapped in a Gaussian form of the actual joint positions)

[0355] The loss function for multi-object tracking is the object detection loss (L det-total ), Triplet or Contrastive loss (L triplet ), and loss of identity consistency (L assoc ) can be combined and used.

[0356] The object detection loss includes standard losses (multi-class classification and bounding box regression losses) used in detection models such as Faster R-CNN and YOLO, and the triplet loss in Equation 21 reduces misassociation by inducing identical objects (IDs) to be close and different objects to be far apart in the embedding space. Additionally, the ID consistency loss induces temporal consistency by reducing the embedding differences of identical objects across consecutive frames. The composite loss function (L) of the multi-object tracking model MOT-total ) can be calculated through mathematical formulas 20 to 23.

[0357] [Mathematical Formula 20]

[0358]

[0359] (where λ is the weight of each loss term)

[0360] [Mathematical Formula 21]

[0361]

[0362] (Here, f(∙) represents the embedding extraction neural network, (a,p,n) represent the Anchor, Positive (same ID), and Negative (different ID) in order from left to right, and α represents the margin)

[0363] [Mathematical Formula 22]

[0364]

[0365] [Mathematical Formula 23]

[0366]

[0367] (Here, L ID represents losses such as Re-ID classification and triplets, and L assoc represents auxiliary loss to improve the quality of ID association between consecutive frames. The optional λ det ,λ id ,λ assoc Weights can be adjusted according to task importance or data characteristics.

[0368]

[0369] Finally, the neural network parameter update means comprises the neural network parameters (θ) of the common / independent latent space transformation means, the neural network parameters (ϕ) of the target space restoration means, and the calculated semantic unit-contrast loss function (L c Receives ) as input, and the corresponding loss function (L c Gradient (gradient, ∇) for ) θ L c ,∇ ϕ L c ) can be calculated. Subsequently, the neural network parameter update means updates the parameters (θ,ϕ) using gradient descent, etc., and finally the updated parameter (θ updated , ϕ updated Can output ).

[0370] A neural network parameter update means according to one embodiment of the present invention can update neural network parameters using a backward propagation algorithm and an update algorithm used in conventional deep learning model training. The specific parameter update process can be performed in the following sequential steps, assuming that initial weights or current weights w exist and a loss function L for input data has been calculated.

[0371] First, through the backpropagation algorithm, the gradients (Gradient, ∇ for each layer and parameter w L) can be calculated. Specifically, the gradient is obtained by differentiating the loss function with respect to the model parameters, and the chain rule can be applied to each layer using the automatic differentiation technique. Additionally, by performing layer-by-layer backpropagation to propagate the contribution of each parameter to the final loss stepwise in the reverse direction (from the output layer to the input layer), the gradients for the weights and biases of all layers can be numerically derived.

[0372] Next, the learning rate (η) to be used for updating parameters can be determined. The learning rate can be a fixed value or can be dynamically adjusted during the learning process (learning rate scheduling).

[0373] Next, the parameters can be finally updated by applying an update algorithm based on the calculated gradient and learning rate. Any of the conventional update algorithms can be used as the update algorithm, and in addition to the update algorithm specifically presented in this specification, any conventional deep learning model parameter update algorithm can be similarly applied.

[0374] For example, when applying Stochastic Gradient Descent (SGD), w←w-η·∇ w The parameters can be updated through the formula for L. When applying the momentum technique, v←αv-η·∇ w The parameters can be updated through the formulas L and w←w+v, where v represents the momentum term and α represents the momentum coefficient. Furthermore, when applying the Adam optimization technique, m t ←β1m t-1 +(1-β1)∇ w L and v t ←β2v t-1 +(1-β2) (∇ w L) 2 The moment estimate can be calculated first using the formula. Afterwards and Finally, after undergoing the operation The parameter can be updated through the formula. Here, m t ,v t represents the estimated values ​​of the first and second moments, respectively, β1 and β2 represent the moment coefficients, and ϵ represents the numerical stabilization term.

[0375] Through a learning method using a semantic unit-contrast loss function between such partitioned latent representations, the present invention can overcome the limitations of existing latent space learning methods in which semantic unit visual information is not properly separated by utilizing contrast learning between positive pairs and negative pairs. That is, the present invention can dramatically maximize the universality and usability of the latent space by inducing that common information or independent information between semantic unit visual information is separated and represented without duplication on different partitioned latent spaces during latent space partitioning.

[0376] FIG. 25 is a diagram illustrating a specific embodiment for calculating a semantic unit-contrast loss function between partitioned latent representations according to one embodiment of the present invention.

[0377] Referring to FIG. 25, a means for calculating a semantic unit-contrast loss function between partitioned latent representations according to one embodiment of the present invention comprises a semantic unit considered in learning ( The number of ) is 2( ) and partitioned latent representation( The number of ) is 3( Assuming a situation where ) semantic unit-contrast loss function( ) can be produced.

[0378] Specifically, in the embodiment, the two semantic units are each a first semantic unit aimed at maximizing cognitive quality ( ) and a second semantic unit aimed at maximizing object classification performance ( It can be assumed that ). Correspondingly, each semantic unit loss function calculated by the semantic unit loss function calculation means is the first semantic unit ( MSE (Mean Squared Error) loss function for ) ) and second semantic unit( Cross Entropy loss function for ) It can be.

[0379] First partitioned potential representation ( ) is the first semantic unit ( It is trained to include only independent information about ), and the third partitioned latent representation ( ) is the second semantic unit ( It is trained to include only independent information about ), and the second partitioned latent representation ( ) is the first semantic unit ( ) and second semantic unit( It can be learned to include common information about everyone.

[0380] To induce this learning direction, directional parameters for each partitioned latent representation ( ) and weight parameters( ) can be specifically defined and applied as follows. First, the first partitioned potential representation ( For ), the loss function of the first semantic unit ( Minimize ) and the loss function of the second semantic unit ( In order to induce maximizing ) It can be set as. Second, the second partitioned potential representation ( For ), the loss function ( simultaneously for two semantic units In order to induce ) to be minimized It can be set as. Third, the third partitioned potential representation ( For ), the loss function of the first semantic unit ( Maximize ) and the loss function of the second semantic unit ( In order to induce minimization of ) It can be set as such. Furthermore, the weights applied to the loss function for each semantic unit are, for example, It can be set to.

[0381] Based on such specific parameter assumptions, the means for calculating the semantic unit-contrast loss function between partitioned latent representations is the final semantic unit-contrast loss function between partitioned latent representations ( ) can be calculated and produced through a linear combination as shown in the mathematical formula below.

[0382] [Mathematical Formula 24]

[0383]

[0384]

[0385] The latent space learning means of the present invention thus has a directional parameter ( A precisely designed loss function () having at least one unique minimization condition and the rest maximization conditions By updating neural network parameters to minimize ), learning can be performed efficiently so that common information and independent information between multiple semantic units are thoroughly separated and represented without any overlap in different partitioned latent spaces.

[0386]

[0387] FIG. 26 is a drawing for explaining a video editing means based on a segmented unit latent representation and the concept thereof according to an embodiment of the present invention.

[0388] Referring to FIG. 26, image editing can be performed through a segmented unit latent expression editing means according to one embodiment of the present invention. In the present invention, 'image editing' may mean a process of creating new visual information by adding or deleting part or all of the visual information and replacing it with other visual information.

[0389] Specifically, the partition unit latent expression editing means can receive the original 'partition unit latent expression' and the 'partition unit latent expression to be used for editing' as input. The editing means can perform simple operations such as addition, deletion, and replacement on the received latent expressions, and can apply these operations to finally generate and output an 'edited partition unit latent expression'.

[0390] The video editing process may include the following additional procedures and information as necessary. First, the latent representation of the segmentation unit to be used for editing can be obtained by passing the original visual information to be edited through the components of the aforementioned decoding / encoding unit (e.g., a common / independent latent space transformation means and a latent space segmentation means). Second, the editing means may receive and utilize auxiliary information to specifically designate the editing method (addition, deletion, replacement, etc.) and the semantic unit to be targeted. For example, the auxiliary information may include information regarding the location to be added if a specific object is to be added, information regarding the object to be deleted if a specific object is to be deleted, and information capable of identifying the semantic unit to be replaced if a replacement with another object is to be included.

[0391] The edited segmented unit latent representation generated through such an editing process can be restored back into visual information of the target space through a target space restoration means depending on the application purpose, or encoded back into a bitstream form through a segmented unit latent representation encoding means and stored in a storage device or transmitted through a network.

[0392] Consequently, the image editing method based on a segmented unit latent representation according to the present invention can directly perform image editing using only simple operations (addition, deletion, replacement) at the latent representation level, without undergoing a decoding process into the original visual information space. Through this, the present invention can drastically reduce the computational complexity required for image editing compared to conventional technology, and the editing method itself can be configured in a relatively simple and intuitive manner.

[0393]

[0394] FIG. 27 is a diagram illustrating an image editing structure that restores a target space by replacing a specific object included in a bitstream with another object according to an embodiment of the present invention.

[0395] Referring to FIG. 27, the present embodiment may assume a situation in which a specific object included in an existing bitstream is replaced with another object, and the editing result is to be restored as visual information in the target space. Specifically, this type of video editing process can be performed through the following procedure.

[0396] First, the partition unit latent representation decoding means can decode the partition unit latent representation from the original bitstream and output it. Second, the common / independent latent space conversion means can receive new time information to be used for replacement editing and convert it into a latent representation. Third, the partition unit latent representation editing means can receive auxiliary information indicating an editing method that directs the replacement of an object along with the converted latent representation as input, and can replace semantic units related to a specific object within the decoded existing partition unit latent representation with the latent representation of a new object. Fourth, the destination space restoration means can receive the edited partition unit latent representation finally generated through the replacement process, convert it into time information of the destination space, and output it.

[0397] This object-substitution-based video editing method is characterized by the fact that editing is possible solely through object-unit substitution in latent space, without the need to perform complex operations such as re-encoding the entire bitstream to change specific objects within the video. Therefore, this method can provide a highly advantageous effect for dynamically generating and efficiently providing user-customized visual information on the receiver side (decoder side).

[0398]

[0399] FIG. 28 is a diagram illustrating an image editing structure that saves an edited bitstream by deleting a specific object included in the bitstream according to another embodiment of the present invention.

[0400] Referring to FIG. 28, the present embodiment may assume a situation in which a specific object is deleted from an existing bitstream, and the editing result is encoded into a new bitstream and saved. Specifically, this type of video editing process can be performed through the following procedure.

[0401] First, the partition unit latent representation decoding means can decode the partition unit latent representation from the existing original bitstream and output it. Second, the partition unit latent representation editing means can receive and utilize auxiliary information indicating an editing method that directs the deletion of an object, and remove the semantic unit corresponding to the object from the decoded partition unit latent representation. Third, the partition unit latent representation encoding means can receive the finally generated edited partition unit latent representation after the removal process, and encode it back into a bitstream to output a new edited bitstream.

[0402] Through this object deletion-based video editing method, the present invention can intuitively delete only the information of unwanted objects from an existing bitstream and immediately recompress and save it in bitstream form without an unnecessary restoration process, thereby reducing data capacity and improving the efficiency of video management.

[0403]

[0404] FIG. 29 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[0405] Referring to FIG. 29, a bitstream region to be decoded among the bitstreams can be determined based on identifier mapping information (S2810).

[0406] In one embodiment, the step of determining a bitstream region to be decoded may determine a bitstream region optimized for at least one of machine vision and human vision as the bitstream region to be decoded.

[0407] In another embodiment, the identifier mapping information may be at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

[0408] Next, the determined bitstream region can be decoded to restore the divided latent representation (S2820).

[0409] In another embodiment, the partitioned latent representation can be restored based on at least one of a neural network-based partitioned latent representation decoding method, an algorithm-based partitioned latent representation decoding method, and a standard codec-based partitioned latent representation decoding method.

[0410] Next, semantic unit latent expressions can be recombined from the divided latent expressions (S2830).

[0411] In another embodiment, the step of recombining semantic unit latent expressions may be performed based on at least one combination means among channel unit recombination and spatial unit recombination.

[0412] Next, the recombined semantic unit latent representation can be converted into a target space (S2840).

[0413] In another embodiment, the step of converting the recombined semantic unit latent representation into the target space may be performed based on at least one of a CNN-based transformation and an MLP-based transformation.

[0414] In another embodiment, the step of decoding a partitioned latent representation based on at least one of a NAL unit-based independent encoding method and a unique syntax-based independent encoding method may be further included.

[0415] FIG. 30 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

[0416] Referring to FIG. 30, input time information can be converted to generate a latent representation (S2910).

[0417] In one embodiment, the step of generating a latent representation by converting input time information can be performed based on at least one method among CNN-based conversion and MLP-based conversion.

[0418] Next, the generated latent expression can be divided to generate latent expressions divided by common information and independent information between semantic unit visual information (S2920).

[0419] In another embodiment, the step of dividing the generated latent representation to generate latent representations divided by common information and independent information between semantic unit visual information may be performed based on at least one method among channel unit division and spatial unit division.

[0420] In another embodiment, the semantic unit visual information may be at least one of visual information having a higher-level semantics and visual information optimized for the purpose of use.

[0421] Next, the divided potential representation can be independently encoded based on identifier mapping information to generate a bitstream (S2930).

[0422] In another embodiment, the step of encoding a divided latent representation to generate a bitstream can generate a bitstream composed of independent regions based on at least one of a NAL unit-based independent encoding method and a unique syntax-based independent encoding method.

[0423] In another embodiment, the identifier mapping information may be at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

[0424] In another embodiment, the step of encoding the segmented latent representation may be performed based on at least one method among neural network-based segmented latent representation encoding, algorithm-based segmented latent representation encoding, and standard codec-based segmented latent representation encoding.

[0425]

[0426] FIG. 31 is a flowchart illustrating a latent space learning method according to one embodiment of the present invention.

[0427] Referring to FIG. 31, a latent representation can be generated by converting input time information (S3010).

[0428] In one embodiment, the step of generating a latent representation by converting input time information may be performed based on at least one conversion method among CNN-based conversion and MLP-based conversion.

[0429] Next, the transformed latent representation can be divided to generate latent representations divided by common information and independent information between semantic unit visual information (S3020).

[0430] Next, the divided potential representation can be converted into a target space (S3030).

[0431] In another embodiment, the step of converting the partitioned latent representation into the target space may be performed based on at least one of a CNN-based conversion and an MLP-based conversion.

[0432] Next, a loss function for each semantic unit of the transformed visual information can be calculated (S3040).

[0433] In another embodiment, the step of calculating a loss function per semantic unit for the converted visual information may be performed based on at least one method among a loss function for maximizing cognitive quality and a loss function for maximizing machine vision task performance.

[0434] Next, a semantic unit-contrast loss function can be calculated based on a linear combination of the calculated semantic unit-specific loss function and predefined parameters (S3050).

[0435] Next, neural network parameters can be updated based on the calculated semantic unit-contrast loss function (S3060).

[0436]

[0437] FIG. 32 is a diagram illustrating an exemplary content streaming system to which an embodiment according to the present invention can be applied.

[0438] Referring to FIG. 32, a content streaming system to which an embodiment of the present invention is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0439] The encoding server described above compresses content input from multimedia input devices, such as smartphones, cameras, and CCTVs, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and CCTVs, generate the bitstream directly, the encoding server may be omitted.

[0440] The bitstream above may be generated by a video encoding method and / or video encoding device to which an embodiment of the present invention is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0441] The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can perform the role of controlling commands and responses between each device within the content streaming system.

[0442] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.

[0443] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0444] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0445]

[0446] The above embodiments may be performed in the same or a corresponding way in the encoding device and the decoding device. Additionally, an image may be encoded / decoded using at least one of the above embodiments or a combination of at least one.

[0447] The order in which the above embodiments are applied may differ between the encoding device and the decoder. Alternatively, the order in which the above embodiments are applied may be the same between the encoding device and the decoder.

[0448] The above embodiments may be performed for each of the luminance and chrominance signals. Alternatively, the above embodiments for the luminance and chrominance signals may be performed in the same way.

[0449] In the above embodiments, methods are described based on flowcharts as a series of steps or units; however, the present invention is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, that other steps may be included, or that one or more steps of the flowcharts may be omitted without affecting the scope of the present invention.

[0450] The above embodiments may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0451] The bitstream generated by the encoding method according to the above embodiment may be stored in a non-transient computer-readable recording medium. Additionally, the bitstream stored in the non-transient computer-readable recording medium may be decoded by the decoding method according to the above embodiment.

[0452] Herein, examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0453] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.

[0454] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims set forth below, as well as the claims described below, shall be considered to fall within the scope of the concept of the present invention.

[0455] The present invention can be used in a device for encoding / decoding images and a device for learning latent spaces.

Claims

1. A step of determining a bitstream region among the bitstreams to be decoded based on identifier mapping information; A step of decoding the determined bitstream region above to restore the divided latent representation; A step of recombining semantic unit latent expressions from the above-described divided latent expressions; and An image decoding method characterized by including a step of converting the above-mentioned recombined semantic unit latent representation into a target space.

2. In Paragraph 1, The step of determining the bitstream region to be decoded above is, A video decoding method characterized by determining a bitstream region optimized for at least one of machine vision and human vision as the bitstream region to be decoded.

3. In Paragraph 1, A video decoding method characterized in that the identifier mapping information is at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

4. In Paragraph 1, An image decoding method characterized by further including the step of decoding the segmented latent representation based on at least one of a NAL unit-based independent encoding method and a unique syntax-based independent encoding method.

5. In Paragraph 1, An image decoding method characterized in that the above-mentioned segmented latent representation is restored based on at least one of a neural network-based segmented unit latent representation decoding method, an algorithm-based segmented unit latent representation decoding method, and a standard codec-based segmented unit latent representation decoding method.

6. In Paragraph 1, The step of recombining the above semantic unit latent expressions is, A video decoding method characterized by being performed based on at least one combination means among channel-unit recombination and spatial-unit recombination.

7. In Paragraph 1, The step of converting the above-mentioned recombined semantic unit latent representation into a target space is, An image decoding method characterized by being performed based on at least one transformation method among CNN-based transformation and MLP-based transformation.

8. A step of converting input visual information to generate a latent representation; A step of dividing the above-mentioned latent expression to generate latent expressions divided by common information and independent information between semantic unit visual information; and A video encoding method characterized by including the step of independently encoding the above-described divided potential representations based on identifier mapping information to generate a bitstream.

9. In Paragraph 8, A video encoding method characterized in that the above semantic unit visual information is at least one of visual information having a higher-level semantic meaning and visual information optimized for the purpose of use.

10. In Paragraph 8, The step of converting the above input time information to generate a latent representation is, An image encoding method characterized by being performed based on at least one of a CNN-based transformation and an MLP-based transformation.

11. In Paragraph 8, The step of dividing the above-mentioned latent representation to generate latent representations divided by common information and independent information between semantic unit visual information is: A video encoding method characterized by being performed based on at least one of channel-unit partitioning and spatial-unit partitioning.

12. In Paragraph 8, The step of encoding the above-mentioned divided latent representation to generate a bitstream is: A video encoding method characterized by generating a bitstream composed of independent regions based on at least one of a NAL unit-based independent encoding method and a unique syntax-based independent encoding method.

13. In Paragraph 8, A video encoding method characterized in that the identifier mapping information is at least one of bitstream header-based identifier mapping information and table-based identifier mapping information.

14. In Paragraph 8, The step of encoding the above-mentioned divided latent representation is, An image encoding method characterized by being performed based on at least one of neural network-based segmented latent representation encoding, algorithm-based segmented latent representation encoding, and standard codec-based segmented latent representation encoding.

15. A step of converting input visual information to generate a latent representation; A step of dividing the above-mentioned transformed latent representation to generate latent representations divided by common information and independent information between semantic unit visual information; A step of converting the above-mentioned divided potential representation into a target space; A step of calculating a loss function for each semantic unit for the above-mentioned converted visual information; A step of calculating a semantic unit-contrast loss function based on a linear combination of the above-calculated semantic unit-specific loss function and predefined parameters; and A latent space learning method characterized by including a step of updating neural network parameters based on the above-calculated semantic unit-contrast loss function.

16. In Paragraph 15, The step of converting the above input time information to generate a latent representation is, A latent space learning method characterized by being performed based on at least one transformation method among CNN-based transformation and MLP-based transformation.

17. In Paragraph 15, The step of converting the above-mentioned divided potential representation into a target space is, A latent space learning method characterized by being performed based on at least one transformation method among CNN-based transformation and MLP-based transformation.

18. In Paragraph 15, The step of calculating a loss function for each semantic unit for the above-mentioned transformed visual information is, A latent space learning method characterized by being performed based on at least one of a loss function for maximizing cognitive quality and a loss function for maximizing machine vision task performance.