Video encoding device, video decoding device, video encoding method, video decoding method, and image transmission system

Abstract

A video encoding device (11) comprises: a feature map conversion unit (112) that selects and packs one feature map or a plurality of feature maps, using a plurality of feature quantities extracted from an image (T); and an encoding unit (113) that performs encoding using the selected one or plurality of feature maps.

Description

Moving image encoding device, moving image decoding device, moving image encoding method, moving image decoding method, and image transmission system

[0001] The present disclosure relates to a moving image encoding device, a moving image decoding device, a moving image encoding method, a moving image decoding method, and an image transmission system

[0002] A moving image encoding device is a device that encodes a moving image, and the encoded moving image can be efficiently transmitted or recorded. A moving image decoding device generates a decoded image by decoding encoded data encoded by a moving image encoding device. Examples of moving image encoding methods include H.266 / VVC (Versatile Video Coding) or H.265 / HEVC (High Efficiency Video Coding).

[0003] Examples of machine-based analysis processing of moving images include object detection, object segmentation, or object tracking. As an encoding method suitable for such analysis processing, for example, Patent Document 1 describes an encoding method that encodes a feature map derived from a moving image by deep learning or the like for machine recognition. The encoding method described in Patent Document 1 divides a feature map extracted from a moving image by a neural network into a plurality of subchannels and packs each channel. The subchannel may be a frame of the moving image, or a plurality of subchannels may be set in one frame.

[0004] When the feature map has a spatial height of h, a horizontal width of w, and a channel direction of c, it can be represented as a tensor of (h × w × c). In conventional packing, the tensor is rearranged and converted into a matrix of (ah × bw). Note that a and b are integer values. By converting the tensor into the form of a two-dimensional matrix in this way, it becomes a data format that can be handled in moving image encoding such as HEVC and VVC, which is adopted as Feature Inner Coding.

[0005] In packing, the feature of the 0th channel is placed in the upper left corner of the picture, and then the feature of the subsequent channels is arranged horizontally in order. The width of the picture is predetermined, and once this width limit is reached, the feature moves to the next row and is arranged from left to right. In Feature Encoding, a submodule of Feature Internal Coding, the luminance component of the image is used as an element of the above two-dimensional matrix, and is encoded with a Y:U:V = 4:0:0 setting that does not utilize the color channels.

[0006] International Publication No. 2023 / 112879

[0007] Conventional packing methods have a challenge in that there is room for improvement in compression efficiency in order to efficiently encode and decode feature maps. For example, after processing feature maps using activation functions such as ReLU (Rectified Linear Unit), most of the information consists of regions with little spatial variation. Features in regions of the feature map with little spatial variation have room for improvement in compression ratio. Also, the color components U and V have a higher compression ratio than the luminance component Y, but these channels are not being effectively utilized.

[0008] This disclosure aims to solve the above-mentioned problems and to provide a video encoding device that can efficiently encode feature maps and a video decoding device that can efficiently decode encoded feature maps.

[0009] The video encoding device according to this disclosure comprises a feature map conversion unit that selects and packs either a single feature map or multiple feature maps using a plurality of feature quantities extracted from an image, and an encoding unit that encodes using the selected single or multiple feature maps.

[0010] The video decoding device according to this disclosure comprises a decoding unit that decodes information on a sequence or picture basis, where the number of feature maps can be selected, using a plurality of feature quantities extracted from an image, and a feature map inverse conversion unit that inversely converts one or more packed feature maps back into a plurality of feature quantities extracted from an image based on the decoded information.

[0011] According to this disclosure, multiple feature quantities extracted from an image are used to select and pack either a single feature map or multiple feature maps, and the selected single or multiple feature maps are used for encoding. This allows the video encoding device according to this disclosure to efficiently encode feature maps. Furthermore, according to this disclosure, multiple feature quantities extracted from an image are used to decode information where the number of feature maps can be selected, on a sequence or picture basis, and based on the decoded information, the packed single or multiple feature maps are inversely transformed back into the multiple feature quantities extracted from the image. This allows the video decoding device according to this disclosure to efficiently decode the encoded feature maps.

[0012] This is a block diagram showing an example configuration of an image transmission system according to Embodiment 1. This is a block diagram showing an example configuration of a video encoding device according to Embodiment 1. This is a block diagram showing an example configuration of a video decoding device according to Embodiment 1. This is a flowchart showing a video encoding method according to Embodiment 1. This is a schematic diagram showing an overview of feature packing for each channel. This is a flowchart showing a video decoding method according to Embodiment 1. This is a schematic diagram showing an overview of the first image transmission procedure in Embodiment 1. This is a schematic diagram showing an overview of the second image transmission procedure in Embodiment 1. This is a diagram showing the hierarchical structure of data in an encoded stream. This is a diagram showing an example of the syntax of sequence-level metadata in the hierarchical structure of data in an encoded stream. This is a diagram showing a first example of the syntax of picture-level metadata in the hierarchical structure of data in an encoded stream. This is a flowchart showing the procedure for feature decoding using the metadata in Figure 11. This is a diagram showing a second example of the syntax of picture-level metadata in the hierarchical structure of data in an encoded stream. This is a flowchart showing the procedure for feature decoding using the metadata in Figure 13. This is a diagram showing a third example of the syntax of picture-level metadata in the hierarchical structure of data in an encoded stream. This is a flowchart showing the procedure for feature decoding using the metadata in Figure 15. Figure 22A and Figure 22B are block diagrams showing the hardware configuration for realizing the functions of the video encoding device and video decoding device according to Embodiment 1.

[0013] Embodiment 1. (Overview of the Image Transmission System) Figure 1 is a block diagram showing an example configuration of the image transmission system 1 according to Embodiment 1. In Figure 1, the image transmission system 1 receives an image T to be encoded as input, generates an encoded stream Te and an encoded stream Fe, and outputs one or more images Td obtained by decoding the encoded stream Te, and one or more feature maps Fd obtained by decoding the encoded stream Fe. The image transmission system 1 is configured to include a video encoding device 11 and a video decoding device 13 that is communicated via a network 12.

[0014] The video encoding device 11 is a device that, upon receiving an image T as input, encodes the input image T to generate an encoded stream Te and an encoded stream Fe. The encoded stream Te is a bitstream in which the image T has been encoded, and one or more images Td are decoded from the encoded stream Te. The encoded stream Fe is a bitstream in which a feature map generated using features extracted from the image T has been encoded, and one or more feature maps Fd are decoded from the encoded stream Fe.

[0015] Network 12 is a communication network for transmitting encoded streams Te and Fe, generated by the video encoding device 11, to the video decoding device 13. Network 12 is a wireless network, a wired network, or a network that combines both wireless and wired connections. Network 12 may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these.

[0016] Network 12 is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, Network 12 may be replaced by a storage medium on which encoded streams are written from the video encoding device 11 and read from the video decoding device 13. Examples of storage media include DVDs (Digital Versatile Disc™), Blu-ray Discs (BD™), or non-volatile memory.

[0017] The video decoding device 13 decodes the encoded streams Te and Fe transmitted through the network 12, decodes one or more images Td from encoded stream Te, and decodes one or more feature maps Fd from encoded stream Fe. The one or more images Td decoded by the video decoding device 13 are output to the display device 14. The one or more feature maps Fd decoded by the video decoding device 13 are output to the analysis device 15.

[0018] The display device 14 is a device that displays all or part of one or more images Td generated by the video decoding device 13. The display device 14 includes, for example, a liquid crystal display, an organic EL (Electroluminescence) display, and other display devices. The form of the display device 14 can be a stationary, mobile, HMD (Head Mount Display), or projector. Furthermore, if the video decoding device 13 has high processing power, it displays high-quality images Td, and if it has lower processing power, it displays images Td that do not require high processing power or display power.

[0019] The analysis device 15 is a device that performs analysis processing, i.e., predetermined tasks, such as object detection, object segmentation, and object tracking, using one or more feature maps Fd generated by the video decoding device 13. Here, "task" refers to a machine task performed by a machine, not a human. All or part of the analysis results may be displayed by the display device 14. For example, the analysis device 15 may use the feature maps Fd to output a list consisting of object IDs indicating the position, size, and type of an object, or a confidence score.

[0020] Applications of the analysis device 15 include identification or detection of people, vehicles, parts, etc., captured by multiple sensors, semantic segmentation, panoptic segmentation, object tracking, anomaly detection of objects, or distance measurement to objects.

[0021] In person identification using surveillance cameras, the surveillance camera side extracts feature quantities from images captured by multiple surveillance cameras, and the server side identifies the person by comparing these feature quantities. In this case, for example, the surveillance camera side is a video encoding device 11, and the server side is a video decoding device 13.

[0022] In Figure 1, the display device 14 and analysis device 15 are shown as separate devices from the video decoding device 13, but the configuration is not limited to this. For example, the video decoding device 13 may include at least one of the following: the display device 14 as a display unit, and the analysis device 15 as a task processing unit that performs tasks using the feature map Fd.

[0023] (Configuration of the video encoding device) Figure 2 is a block diagram showing an example configuration of the video encoding device 11 according to Embodiment 1. In Figure 2, the video encoding device 11 is configured to include a feature extraction unit 111, a feature map conversion unit 112, an encoding unit 113, and an encoding unit 114. The feature extraction unit 111 takes an image T as input and extracts features from the input image T. For example, the feature extraction unit 111 may use the output of a neural network used to detect objects from an image T, such as Faster / Mask R-CNN (Region-based Convolutional Neural Network), 101-FPN (Feature Pyramid Network), or a JDE (Joint Detection and Embedded) network, as features.

[0024] The feature map conversion unit 112 uses multiple features extracted from image T by the feature extraction unit 111 to select and pack either a single feature map or multiple feature maps. For example, after compressing the features extracted from image T, the feature map conversion unit 112 selects whether to use one channel or three channels (Y, U, and V) based on metadata which is information that allows selection of the number of feature maps, and packs (distributes) the compressed features into the image of the selected channel. Then, the feature map conversion unit 112 quantizes the packed features into integer values ​​and converts the range of the features from 0 to 255 to generate a quantized feature map. In this way, by selecting whether to use one channel or three channels, it is possible to select and pack either a single feature map or multiple feature maps. This allows for efficient encoding of feature maps. For example, it is possible to improve the compression ratio even for features in regions with little spatial variation in the feature map, and the U and V subchannels, which are color components with high compression ratios, can be effectively utilized.

[0025] The encoding unit 113 encodes using one or more feature maps selected by the feature map conversion unit 112. For example, the encoding unit 113 encodes one or more feature maps output from the feature map conversion unit 112 and outputs them as an encoded stream Fe. The encoded stream Fe output from the encoding unit 113 is transmitted to the video decoding device 13 via the network 12. As an encoding method, Versatile Video Coding (VVC), Advanced Video Coding (AVC), or HEVC can be used.

[0026] The encoding unit 114 is a video encoding unit that performs video encoding of image T. For example, the encoding unit 114 encodes image T to generate an encoded stream Te and outputs the encoded stream Te. The encoding unit 114 uses VVC / H.266 or HEVC / H.265 for encoding. The encoded stream Te output from the encoding unit 114 is transmitted to the video decoding device 13 via the network 12. Furthermore, the encoding unit 114 also transmits metadata to the video decoding device 13 via the network 12.

[0027] Memory 115 temporarily stores feature maps. For example, the encoding unit 113 reads feature quantities from memory 115, either from the feature map of the previous frame or from the feature quantities located to the left (distributed) between multiple feature maps arranged in a left-right direction, and generates an encoded stream Fe using the read feature quantities.

[0028] In Figure 2, a video encoding device 11 is shown equipped with a feature extraction unit 111, an encoding unit 114, and a memory 115. However, this configuration is not essential. For example, if a feature extraction unit provided by an external device (e.g., another video encoding device) separate from the video encoding device 11 extracts features from an image T and stores them in a storage device separate from the video encoding device 11, the video encoding device 11 does not need to have a feature extraction unit 111. In this case, the feature map conversion unit 112 of the video encoding device 11 can acquire the features from the storage device via a wireless or wired communication network. Similarly, the encoding unit 114 may be provided by an external device separate from the video encoding device 11. Furthermore, if the feature map conversion unit 112 does not process data stored in the memory 115, the video encoding device 11 does not need to have a memory 115. Alternatively, the memory 115 may be provided outside the video encoding device 11, and the feature map conversion unit 112 and the encoding unit 113 may function as an interface for inputting and outputting data to and from the external memory. For example, as will be described later, when the encoding unit 113 encodes the difference between the feature quantities of the feature map of the previous frame and the current frame, the feature map of the previous frame may be obtained from a memory 115 located outside the video encoding device 11.

[0029] (Configuration of the video decoding device) Figure 3 is a block diagram showing an example configuration of the video decoding device 13 according to Embodiment 1. In Figure 3, the video decoding device 13 is configured to include a decoding unit 131, a feature map inverse conversion unit 132, a decoding unit 133, and a memory 134. The decoding unit 131 decodes the encoded stream Fe encoded by the video encoding device 11 and outputs an image (packed subchannel) in which one or more packed feature maps are arranged to the feature map inverse conversion unit 132. The encoded stream Fe is encoded in VVC / H.266 or HEVC / H.265, etc. The decoding unit 131 also decodes metadata that allows the number of feature maps to be selected, in sequence or picture units.

[0030] The feature map inverse transform unit 132 inversely transforms one or more packed feature maps into multiple feature quantities extracted from the image, based on the decoded metadata. For example, the feature map inverse transform unit 132 inversely quantizes the feature quantities contained in the decoded feature map, unpacks the inversely quantized feature quantities, and restores the unpacked feature quantities. Here, the multiple feature quantities output from the feature map inverse transform unit 132 are the feature map Fd.

[0031] Inverse quantization is the process of transforming a feature whose encoded range is 0 to 255 into a range from minimum to maximum using the parameters required for inverse quantization. Unpacking is the process of returning the feature stored in the three channels Y, U, and V back to the dimensions of the feature before encoding. Feature reconstruction transforms a single feature with different vertical and horizontal dimensions into multiple features with different vertical and horizontal dimensions. Alternatively, the original feature may be reconstructed based on basis vectors or weights obtained by Principal Component Analysis (PCA).

[0032] The decoding unit 133 is a video decoding unit that decodes video containing image T. For example, the decoding unit 133 decodes the encoded stream Te in which image T is encoded and outputs the decoded image Td. The decoding unit 133 decodes the encoded stream Te which is encoded using VVC / H.266 or HEVC / H.265, etc.

[0033] Memory 134 temporarily stores the feature map. For example, the feature map inverse transformation unit 132 reads the feature map of the previous frame, which has been decoded by the decoding unit 131, from memory 134, and inversely transforms the read feature map into multiple feature quantities extracted from the image.

[0034] The analysis device 15 performs task processing as described later using Figure 6. Task processing includes processes such as object detection or semantic segmentation, for example, object detection, object segmentation, object tracking, action recognition, pose estimation, or anomaly detection.

[0035] Object detection is the task of detecting specific objects (such as people or cars) within an image. This allows for the identification of important elements within the image, which can then be used for other processing. Object segmentation is the task of dividing objects within an image into pixel units. This allows for a more accurate understanding of the shape or location of objects.

[0036] Object tracking is the task of tracking a specific object within a video. This allows for the analysis of object movement and the prediction of actions. Action recognition is the task of recognizing actions (e.g., walking, running, jumping, etc.) occurring within an image or video. This allows for the understanding of actions such as those of people and the ability to take appropriate action.

[0037] Pose estimation is the task of estimating a person's posture or pose. This allows for the analysis of a person's movement or posture and can be used in fields such as sports or medicine. Anomaly detection is the task of detecting or determining whether an object in an image is normal or abnormal. This allows for determining whether a manufacturing device or robot is functioning correctly, or whether a component of a product is malfunctioning.

[0038] Although Figure 3 shows a video decoding device 13 equipped with a decoding unit 133 and a memory 134, this is not a mandatory configuration. For example, the decoding unit 133 may be provided by an external device (e.g., a video decoding device) separate from the video decoding device 13. Also, if the feature map inverse conversion unit 132 does not process data stored in the memory 134, the video decoding device 13 does not need to have a memory 134. Furthermore, the memory 134 may be provided outside the video encoding device 11, and the decoding unit 131 and the feature map inverse conversion unit 132 may function as an interface for inputting and outputting data to and from the external memory.

[0039] (Video Encoding Method) Next, a video encoding method according to Embodiment 1 will be described. Figure 4 is a flowchart showing the video encoding method according to Embodiment 1, illustrating a series of operations by the video encoding device 11. The feature extraction unit 111 receives an image T as input and extracts features from the input image T (step ST1). For example, the feature extraction unit 111 extracts features from the image T using a neural network. Examples of neural networks include Faster / Mask R-CNN, 101-FPN, or JDE network. For example, the JDE network is a neural network that can track multiple objects simultaneously and perform detection and feature extraction simultaneously (1-shot). The features extracted by the feature extraction unit 111 are output to the feature map conversion unit 112.

[0040] The feature map conversion unit 112 compresses the feature quantities using PCA or the like (step ST2). For example, the feature map conversion unit 112 converts a feature quantity with a single length and width into a single feature quantity by integrating feature quantities with different length and width sizes obtained from multiple layers. Alternatively, the feature map conversion unit 112 may compress the feature quantities into low-dimensional information based on the information obtained by PCA.

[0041] Next, the feature map conversion unit 112 performs packing of the compressed feature quantities (step ST3). For example, the feature map conversion unit 112 rearranges multi-dimensional (e.g., tens to hundreds or thousands) feature quantities to store them in three channels: Y, U, and V. This process is called packing, and there are static and dynamic methods. The static method rearranges the image features into the Y, U, and V channels using pre-stored metadata. The dynamic method sequentially changes the distribution to the Y, U, and V channels. The distribution method is transmitted to the video decoding device 13 as metadata.

[0042] FIG. 5 is a schematic diagram showing an overview of packing feature amounts for each channel. The left diagram in FIG. 5 shows a plurality of feature channels each composed of a plurality of image feature amounts. The feature channels are spatial channels with a vertical size of h, a horizontal size of w, and a channel number of c. The right diagram in FIG. 5 shows a planar channel in which a plurality of feature amounts constituting the feature channels are arranged. This planar channel has a vertical size of Ph and a horizontal size of Pw.

[0043] For example, the feature map conversion unit 112 arranges a plurality of feature amounts constituting the feature channels in the raster scan order indicated by the arrow in FIG. 5 so that the feature channels shown on the left side of FIG. 5 become the planar channel shown on the right side of FIG. 5, thereby packing the feature map as one channel or packing it into three channels of Y, U, and V. Here, whether to pack into one channel or into three channels of Y, U, and V is selected based on meta information. The meta information is information in which the number of feature maps can be selected. By enabling selection between one channel and three channels by the meta information, the encoding efficiency can be increased. Note that the channels of Y, U, and V may be three channels of Y, Cb, and Cr. Y is a luminance component, Cb is a blue difference component, and Cr is a red difference component.

[0044] Next, the feature map conversion unit 112 quantizes the packed feature amounts with integer values and converts the range of the feature amounts to 0 to 255, thereby generating a quantized feature map (step ST4). Note that the feature map conversion unit 112 may quantize the feature amounts at equal intervals or at non-equal intervals.

[0045] The encoding unit 113 encodes one or more feature maps generated by the feature map conversion unit 112 (step ST5). The feature map is encoded by VVC / H.266 or HEVC / H.265 or the like and is output as an encoded stream Fe to the network 12. Thereby, the series of processes shown in FIG. 4 is completed.

[0046] (Moving Image Decoding Method) Next, the moving image decoding method according to Embodiment 1 will be described. FIG. 6 is a flowchart showing the moving image decoding method according to Embodiment 1, showing a series of operations by the moving image decoding apparatus 13 and the operations of the analysis apparatus 15 associated therewith. That is, the processes from step ST1A to step ST4A are the moving image decoding method according to Embodiment 1. The moving image decoding apparatus 13 acquires an encoded stream Fe composed of feature amounts encoded by the moving image encoding apparatus 11 via the network 12. When the decoding unit 131 inputs the encoded stream Fe, it decodes the data encoded as the encoded stream Fe (step ST1A). The data decoded by the decoding unit 131 is output to the feature map inverse conversion unit 132.

[0047] The feature map inverse conversion unit 132 inverse quantizes the feature amounts decoded by the decoding unit 131 (step ST2A). For example, the feature map inverse conversion unit 132 converts the feature amounts quantized in the range from 0 to 255 to the range from the minimum value to the maximum value when encoding.

[0048] Subsequently, the feature map inverse conversion unit 132 unpacks the inverse quantized feature amounts (step ST3A). For example, unpacking is a process of returning the feature amounts packed in one channel or the feature amounts packed in three channels of Y, U, and V to the dimensions of the feature amounts when encoded. The feature map inverse conversion unit 132 performs unpacking based on the meta information transmitted from the moving image encoding apparatus 11.

[0049] Next, the feature map inverse conversion unit 132 restores the unpacked feature amounts (step ST4A). For example, the feature map inverse conversion unit 132 converts the feature amounts with a single vertical and horizontal size to a plurality of feature amounts with different vertical and horizontal sizes when encoding. Also, the feature map inverse conversion unit 132 may restore the original feature amounts based on the basis vectors or weights obtained by PCA. The feature amounts restored by the feature map inverse conversion unit 132 are output to the analysis apparatus 15.

[0050] The analysis device 15 performs task processing using the reconstructed features (step ST5A). Task processing includes, for example, object detection or semantic segmentation. This completes the process shown in Figure 6. Since the analysis device 15 can perform task processing using the efficiently decoded feature map, the performance of the analysis device 15 can be improved.

[0051] Next, the procedure for image transmission using the image transmission system 1, which includes a video encoding device 11 and a video decoding device 13, will be described. Figure 7 is a schematic diagram showing an overview of the first image transmission procedure in Embodiment 1, and illustrates the case where feature quantities are distributed using a static method in feature map packing.

[0052] (Encoder operation) The feature map conversion unit 112 performs feature compression. For example, the feature map conversion unit 112 integrates feature quantities with different vertical and horizontal sizes obtained from multiple layers and converts them into a single-size feature quantity with c channels, h vertical size, and w horizontal size.

[0053] The feature map conversion unit 112 performs packing of the feature map. Here, the feature map conversion unit 112 distributes the dimensionality-reduced feature quantities to each channel of the feature map based on the metadata stored in the database. As a result, for example, multiple feature quantities are distributed to each of multiple channels, where the number of channels is c, the vertical size is a'h, and the horizontal size is b'w.

[0054] Furthermore, the above metadata is transmitted from the video encoding device 11 to the video decoding device 13. For example, during the initial communication between the video encoding device 11 and the video decoding device 13 (for example, after the handshake), the metadata is transmitted from the video encoding device 11 to the video decoding device 13. As a method for transmitting the metadata, for example, predetermined areas of the U channel (or Cb channel) and V channel (or Cr channel) are reserved, and the metadata is stored in the reserved areas and transmitted. This allows the metadata to be transmitted without increasing the total amount of information transmitted. The metadata may also include flag information indicating whether or not to use the features of the previous frame, and information indicating the quantization mode (scalar quantization, differential quantization, vector quantization, etc.). The metadata can be determined by supervised or unsupervised methods as follows.

[0055] (Supervised) If the downstream task, i.e., the device that utilizes the downstream (e.g., an object detection device, an image recognition device, etc.), is known, the packing is determined so as to maximize the performance indicators of the device, and metadata for the determined packing is determined. Here, the performance indicator is the training data for determining the packing, and examples of evaluation indicators include F1 score, precision, recall, AUROC, or mIoU. For example, the packing is determined so as to maximize the performance indicator of the task. Alternatively, the user (developer) may appropriately determine the packing using an input unit (not shown) provided by the video encoding device 11.

[0056] If the downstream task is unknown or the environment in which image T is acquired changes, training data may be obtained as follows. Here, a change in the environment in which image T is acquired refers to, for example, changes in weather, time of day, brightness, changes due to switching the camera used to acquire image T, changes in focal position due to zooming in or out, etc. For example, before the video encoding device 11 transmits the encoded stream Te and encoded stream Fe to the video decoding device 13, it decodes the encoded stream Fe using a video decoding device with the same functionality as the video decoding device 13. Then, it compares the feature quantities before encoding with the feature quantities that have been decoded, and obtains the change in the two feature quantities (for example, error or distance) as training data. Metadata is set so that this change is small. The change in the two feature quantities can be evaluated using mean squared error (MSE) or cosine similarity, etc. By setting the metadata so that the change in the two feature quantities is small, encoding distortion associated with transmission can be reduced, making it possible to maintain high performance for various tasks.

[0057] (Unsupervised) PCA is performed on multiple feature maps composed of features extracted from multiple images T acquired before determining the metadata. By evaluating the eigenvectors corresponding to the high eigenvalues ​​obtained by PCA, channels of feature maps with high variance, i.e., high importance feature maps, are identified, and information indicating channels with high variance and information indicating channels with low variance (feature maps with low importance) are used as metadata. For example, based on the metadata, the feature map conversion unit 112 stores the features of channels with high variance in the Y channel and the features of channels with low variance in the U channel or V channel. This allows metadata to be determined even without training data.

[0058] In addition to PCA, other examples of dimensionality reduction methods include SVD (Singular Value Decomposition), LDA (Latent Dirichlet Allocation), t-SNE (t-Distributed Stochastic Neighbor Embedding), or UMAP (Uniform Manifold Application and Projection). By distributing the features to each channel of the feature map in order to improve task performance or according to the importance of the feature map, it is possible to efficiently encode and decode the feature map. In other words, by determining metadata using the supervised or unsupervised methods described above, it is possible to efficiently encode and decode the feature map.

[0059] Quantization is the process of converting the range of packed features from 0 to 255. Feature internal coding is the process of encoding the quantized features using VVC / H.266 or HEVC / H.265, etc. The feature map encoded by the video encoding device 11 is transmitted to the video decoding device 13 as an encoded stream Fe.

[0060] (Decoder operation) Feature decoding is the process of decoding the data encoded as an encoded stream Fe. This process decodes the features. Inverse quantization is the process of converting features that have been quantized to a range from 0 to 255, for example, to a range from the minimum value to the maximum value when encoded.

[0061] Unpacking is the process of returning a feature packed into a single channel, or a feature packed into three channels (Y, U, and V), back to the dimensions of the encoded feature. In the video decoding device 13, the feature map inverse transform unit 132 performs unpacking of the inversely quantized feature based on metadata indicating the packing performed by the video encoding device 11.

[0062] Feature reconstruction is the process of recovering unpacked features. For example, the feature map inverse transform unit 132 transforms a feature with a single vertical and horizontal size into multiple features with different vertical and horizontal sizes when encoded.

[0063] Figure 8 is a schematic diagram showing an overview of the second image transmission procedure in Embodiment 1, illustrating the case where feature quantities are distributed dynamically in feature map packing. (Encoder operation) Feature compression, quantization, and internal feature coding are the same as in Figure 7. Eigenvectors corresponding to high eigenvalues ​​obtained by dimensionality reduction such as PCA differ over time. Therefore, the feature map conversion unit 112 acquires channels of feature maps with large variance over time, i.e., sequentially, distributes and stores these channels in the Y channel, and distributes and stores channels with small variance in the U channel or V channel. This process is packing that distributes feature quantities dynamically. This process is a sequential process that performs the unsupervised method described above over time, and is also a process that dynamically sets (updates) metadata. Furthermore, it is possible to use the supervised method described above as a packing method that distributes feature quantities dynamically. Specifically, metadata can be dynamically set (updated) over time using the training data obtained over time. By performing packing that dynamically distributes features, the packing of the feature map can follow changes in the environment in which the image T is acquired, making it possible to efficiently encode and decode the feature map. Each time the distribution is changed, metadata indicating the changed distribution is transmitted from the video encoding device 11 to the video decoding device 13.

[0064] (Decoder operation) Feature decoding, inverse quantization, and feature reconstruction are the same as in Figure 7. The feature map inverse transform unit 132 unpacks the inversely quantized features based on the metadata transmitted from the video encoding device 11 each time the distribution is changed.

[0065] (Structure of Encoded Stream Te and Encoded Stream Fe) The data structure of Encoded Stream Te and Encoded Stream Fe, which are generated by the video encoding device 11 and decoded by the video decoding device 13, will be described below. Figure 9 shows the hierarchical structure of data in Encoded Stream Te and Encoded Stream Fe. In Figure 9, Encoded Stream Te and Encoded Stream Fe include, exemplarily, a sequence and a plurality of pictures that constitute the sequence. Encoded video sequence (1) defines sequence SEQ. Encoded picture (2) defines picture PICT. Encoded slice (3) defines slice S, where S = 0, 1, ..., NS-1. Encoded slice data (4) defines slice data. Encoded tree unit (5) is a tree unit included in Encoded slice data (4). This Encoded tree unit (5) includes the encoding unit shown in Figure 9.

[0066] The encoded video sequence (1) defines a set of data that the video decoding device 13 references in order to decode the sequence SEQ to be processed. The sequence SEQ of the encoded video sequence (1) includes, as shown in Figure 9, a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0067] The Video Parameter Set (VPS) defines a set of encoding parameters common to multiple video layers in a video composed of multiple layers, as well as a set of encoding parameters associated with the multiple layers included in the video and each individual layer.

[0068] The sequence parameter set (SPS) defines a set of encoding parameters that the video decoding device 13 references to decode the target sequence. For example, the width or height of the picture is defined. Multiple SPSs may exist. In this case, one of the SPSs is selected from the PPS.

[0069] The picture parameter set (PPS) defines a set of encoding parameters that the video decoding device 13 references to decode each picture in the target sequence. In this case, one of several PPS is selected from each picture in the target sequence.

[0070] The encoded picture (2) defines a set of data that the video decoding device 13 refers to in order to decode the picture PICT to be processed. As shown in Figure 9, the picture PICT includes slices 0 to NS-1. NS is the total number of slices included in the picture PICT.

[0071] The encoded slice (3) defines a set of data that the video decoding device 13 refers to in order to decode the slice S to be processed. As shown in Figure 9, the encoded slice (3) includes a slice header and slice data.

[0072] The slice header contains a set of encoding parameters that the video decoding device 13 references to determine the decoding method for the target slice. The slice header may also include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0073] The encoded slice data (4) defines a set of data that the video decoding device 13 refers to in order to decode the slice data to be processed. The encoded slice data (4) includes CTUs, as shown in Figure 9. A CTU is a block of a fixed size (e.g., 64 x 64) that constitutes a slice, and is sometimes called a Largest Coding Unit (LCU).

[0074] A picture may be further divided into rectangular subpictures. For example, it may be divided into subpictures that are four horizontally and four vertically. The size of the subpictures may be a multiple of the CTU. A subpicture is defined as a set of tiles that are consecutive in integer numbers horizontally and vertically. The slice header may include sh_subpic_id, which indicates the ID of the subpicture.

[0075] There are two types of prediction (hereinafter referred to as prediction modes): intra-prediction and inter-prediction. Intra-prediction is prediction processing within the same picture, while inter-prediction is prediction processing performed between different pictures (for example, between display times, between layer images). In Figure 9, CN indicates a branching point for splitting from CTU to CU (Coding Unit), and is an intermediate block defined by processing defined by the coding tree syntax in H.265 or H.266.

[0076] Next, the syntax of the metadata will be explained. In the video encoding device 11, the encoding unit 113 encodes the metadata on a sequence or picture basis. In the video decoding device 13, the decoding unit 131 decodes the metadata on a sequence or picture basis. Figure 10 shows an example of the syntax of the sequence-level metadata FSPS in the hierarchical data structure of the encoded stream Te and encoded stream Fe. FSPS (Feature Sequence Parameter Set) is a table that shows sequence-level metadata. This table contains the following information: "fsps_multi_channel_packing_flag" is a flag that indicates that packing was performed using multiple channels. Also, "fsps_channel_packing_refresh_period" is set to indicate the expiration time of the packing. If “fsps_channel_packing_refresh_period” is 0, periodic packing changes are not performed. If it is 0 or greater, periodic packing changes are performed.

[0077] "fsps_feat_seq_parameter_set_id" sets the FSPS identifier. "num_ori_feat_layers" sets the number of layers from which to acquire features. "ori_feat_wid" sets the width of the features for each layer. "ori_feat_hei" sets the height of the features for each layer. "num_ori_feat_chan" sets the number of channels for the features for each layer.

[0078] The syntax shown in Figure 10 defines the number of features and the size of each feature for each Y, U, and V layer. The following two parameters define the width and height for storing the combined feature of the Y, U, and V channels: "fused_feat_width" sets the width of the combined feature for all layers. "fused_feat_hei" sets the height of the combined feature for all layers.

[0079] "inner_decoding_bypass_flag" is a flag that sets whether or not the internal decoder (decoding unit 131) is processing. "feat_inner_decoder_info" is set to the information of the internal decoder. The information of the internal decoder is set to the point where the internal decoder is not processing, that is, "inner_decoding_bypass_flag" is set to "false". "fsps_multi_channel_packing_flag" is a multi-channel packing flag.

[0080] The "fsps_multi_channel_packing_flag" flag controls whether multiple channels are packed simultaneously. For example, if this flag is set, the feature map is packed into multiple subchannels (e.g., three subchannels: Y, U, and V) each consisting of multiple components. If this flag is not set when encoding subchannels, multiple channels are packed simultaneously. This allows the feature map to be composed of a single channel, like a normal feature map.

[0081] The above flag allows for the elimination of extraneous higher-level syntax if the arranged features fit perfectly within the rectangular screen of a single channel, depending on the number of features. On the other hand, if the features do not fit within the rectangular screen of a single channel, efficient encoding is possible even with higher-level syntax by, for example, arranging them in three subchannels: Y, U, and V. This flag also enables decoding using metadata, as explained in Figure 7.

[0082] Figure 10 shows sequence-level flags, but the flags may also be defined as picture (frame-level) parameters, on a per-picture basis. This allows the video encoding device 11 to efficiently encode feature maps. Previously, the above flags indicated whether to pack into one frame or three frames. However, the syntax may be set to allow specifying the number of rectangular frames (number of channels) so that the feature map is filled as tightly as possible on the rectangular frames of the channels.

[0083] The encoding unit 113 encodes an image created by transforming multiple feature quantities extracted from image T using the feature map conversion unit 112. The decoding unit 131 decodes information on the channel packing update frequency, which indicates which channel each feature map is packed into. Channel packing determines the number of feature maps packed into a channel, and when the channel packing is updated, the number of feature maps packed into a channel changes. The channel packing update frequency information is, for example, “fsps_channel_packing_refresh_period”. This parameter sets the channel packing refresh period and is used to control the channel packing update frequency. For example, when packing feature maps into multiple subchannels and encoding the subchannels, the value set in this parameter determines how often the packing is updated. Alternatively, instead of using “fsps_channel_packing_refresh_period”, the system may be configured to determine whether or not to update using “multi_color_channel_update_flag”, as described later. By doing so, the amount of code that would normally be used by “fsps_channel_packing_refresh_period” can be reduced in use cases where “fsps_channel_packing_refresh_period” is not needed.

[0084] The "dequant_bypass_flag" is a dequantization bypass flag. This flag is used to bypass quantization under certain conditions when decoding quantized data. For example, if this flag is set when decoding a quantized feature map or data, the quantization process is skipped, and the original data can be used directly. This makes it possible to reduce computational costs while maintaining data accuracy.

[0085] The “unpacking_bypass_flag” is an unpacking bypass flag. This flag is used to bypass packing under certain conditions when decoding feature maps. For example, if this flag is set, the packing process is skipped, and the original data can be used directly. This makes it possible to reduce computational costs while maintaining data accuracy.

[0086] The “feat_restoration_bypass_flag” is a feature restoration bypass flag. When packing a feature map into multiple subchannels and decoding those subchannels, if this flag is set, the restoration process is skipped, and the original data can be used directly. This makes it possible to reduce computational costs while maintaining data accuracy.

[0087] The “feat_restoration_info” field is used to set feature reconstruction information. This feature reconstruction information indicates that “feat_restoration_bypass_flag” is “false”. When packing the feature map into multiple subchannels and decoding those subchannels, information about the reconstruction process is provided. This makes it possible to reduce computational costs while maintaining data accuracy.

[0088] “picture_number” is a parameter that indicates the number of each frame (picture). picture_number may be configured to explicitly set the picture number as syntax, or it may be configured to set a value that the computer internally counts and increments.

[0089] (First Example) Figure 11 shows the first example of the syntax for picture-level metadata in the hierarchical structure of data in encoded stream Te and encoded stream Fe. FSPS is a table showing frame-level metadata, as shown in Figure 11. This table contains the following information: “fpps_feat_pic_parameter_set_id” is set to the identifier for the picture-level FPPS. “fpps_feat_seq_parameter_set_id” is set to the identifier for the sequence-level FSPS. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is set to “false”. The "packed_feat_val_min" flag is set to the minimum value of the packed features when the "dequant_bypass_flag" is set to "false". The "multi_color_channel_update_flag" is the update flag for the multicolor channel. This flag enables decoding using the metadata explained in Figure 8. The same applies to Figures 12 and 13, which will be described later. The "luminance or chromaticity" flag is set to "true" when the "multi_color_channel_update_flag" is "true".

[0090] (Encoder operation) In the video encoding device 11, the feature map conversion unit 112 automatically distributes feature quantities to the Y channel, U channel, and V channel based on the "luma_or_chroma" flag. For example, if this flag is "true", the feature map conversion unit 112 stores and packs the feature quantities in order from the top left of the screen for the Y channel, U channel, and V channel. On the other hand, if this flag is "false", the feature map conversion unit 112 stores feature quantities alternately for the U channel and V channel in that order. In this case, the feature quantities are stored in order from the top left, similar to when packing for the Y channel. In this way, the feature map conversion unit 112 distributes feature quantities to the Y channel, U channel, and V channel using a predetermined order and flag.

[0091] The information regarding the "luma_or_chroma" flag, which is "true," is metadata that selects whether to store the feature at the beginning of the first Y channel or at the beginning of the second and subsequent U and V channels. The encoding unit 113 encodes this information to select whether to start at the beginning of the first or subsequent channels and transmits it to the video decoding device 13. Similarly, the information regarding the "luma_or_chroma" flag, which is "false," is metadata that selects whether to store the feature at the beginning of the second U channel or at the beginning of the third and subsequent V channels. The encoding unit 113 encodes this information to select whether to start at the beginning of the second or third channel and transmits it to the video decoding device 13.

[0092] The metadata FSPS in Figure 11 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0093] (Decoder operation) Figure 12 is a flowchart showing the procedure for feature decoding using the metadata in Figure 11, and shows the flow of operation by the feature map inverse transform unit 132. When the multi-channel packing flag "fsps_multi_channel_packing_flag" is set, a determination is made as to whether the picture is a picture that updates the packing process. One way is to set the picture as an update picture (update_flag = true) if it is within the set period based on "fsps_channel_packing_refresh_period" and "picture_number". Another way is to set the picture as an update picture (update_flag = true) if the multi-color channel update flag ("multi_color_channel_update_flag") is true. Other pictures will not be updated (update_flag=false).

[0094] If the picture is one that updates the packing process (update_flag = True), and the parameter i indicating which channel it is is smaller than the value of "num_fused_feat_channels" (step ST1B), the feature map inverse transform unit 132 determines whether the "luma_or_chroma" flag is "true" or "false" for each channel (step ST2B).

[0095] If the "luma_or_chroma" flag is "true", the feature map inverse transform unit 132 acquires an image based on its position from the Y channel (step ST3B). Then, the feature map inverse transform unit 132 updates the position of the Y channel (step ST4B) and adds the image to the feature map (step ST5B). If the "luma_or_chroma" flag is "false", the feature map inverse transform unit 132 determines whether the position of the U channel and the position of the V channel are equal (step ST6B). If the position of the U channel and the position of the V channel are equal (step ST6B; true), the feature map inverse transform unit 132 acquires an image based on its position from the U channel (step ST7B). Then, the feature map inverse transform unit 132 updates the position of the U channel (step ST8B) and proceeds to step ST5B to add the image to the feature map.

[0096] On the other hand, if the positions of the U channel and the V channel are not equal (step ST6B; false), the feature map inverse transform unit 132 acquires an image based on the position from the V channel (step ST9B). Then, the feature map inverse transform unit 132 updates the position of the V channel (step ST10B) and proceeds to step ST5B to add the image to the feature map.

[0097] Furthermore, there is a multicolor channel update flag ("multi_color_channel_update_flag"), and if parameter i is greater than or equal to the value of "num_fused_feat_channels", the feature map inverse transform unit 132, if the positions of the U channel and the V channel are equal, acquires an image based on the position from the U channel, updates the position of the U channel, and adds the image to the feature map. On the other hand, if the positions of the U channel and the V channel are not equal, the feature map inverse transform unit 132 acquires an image based on the position from the V channel, updates the position of the V channel, and adds the image to the feature map. The information regarding the "luma_or_chroma" flag, which is "true", is metadata that selects whether to store the features at the start of the first Y channel or at the second and subsequent U and V channels. The decoding unit 131 decodes this information that selects whether to store at the start of the first channel or at the second and subsequent channels. Similarly, the information regarding the "luma_or_chroma" flag, which is "false," is metadata that selects whether to store the features at the start of the second U channel or at the start of the third or subsequent V channels. The decoding unit 131 decodes this information that selects whether to start at the second or third channel.

[0098] (Second Example) Figure 13 shows a second example of the syntax for picture-level metadata in the hierarchical data structure of encoded streams Te and Fe. The main points to be explained are the differences from the syntax in Figure 11.

[0099] (Encoder operation) The feature map conversion unit 112 automatically packs the features based on the "channel_y" flag and the "channel_u" flag shown in Figure 13. If the "channel_y" flag is "true", the feature map conversion unit 112 packs the features sequentially from the top left of the Y channel screen. If the "channel_u" flag is "true", the feature map conversion unit 112 packs the features sequentially from the top left of the U channel screen. If both the "channel_y" flag and the "channel_u" flag are "false", the feature map conversion unit 112 packs the features sequentially from the top left of the V channel screen.

[0100] The information related to the “channel_y” flag is metadata that selects whether to store the feature at the start of the first Y channel or at the start of the second and subsequent U and V channels. The encoding unit 113 encodes this information to select whether to start at the first channel or at the start of the second and subsequent channels and transmits it to the video decoding device 13. Similarly, the information related to the “channel_u” flag is metadata that selects whether to store the feature at the start of the second U channel or at the start of the third and subsequent V channels. The encoding unit 113 encodes this information to select whether to start at the second channel or at the start of the third channel and transmits it to the video decoding device 13.

[0101] Furthermore, the metadata FSPS in Figure 13 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0102] (Decoder operation) Figure 14 is a flowchart showing the procedure for feature decoding using the metadata in Figure 13, and shows the flow of operation by the feature map inverse transform unit 132. When the multi-channel packing flag "fsps_multi_channel_packing_flag" is set, a determination is made as to whether the picture is a picture that updates the packing process. One way is to set the picture as an update picture (update_flag = true) if it is within the set period based on "fsps_channel_packing_refresh_period" and "picture_number". Another way is to set the picture as an update picture (update_flag = true) if the multi-color channel update flag ("multi_color_channel_update_flag") is true. Other pictures will not be updated (update_flag=false).

[0103] If the picture is one that updates the packing process (update_flag = true), and the parameter i indicating which channel it is is smaller than the value of "num_fused_feat_channels" (step ST1C), the feature map inverse transform unit 132 determines whether the flags "channel_y" and "channel_u" are "true" or "false" for each channel (step ST2C). If the "channel_y" flag exists (step ST2C; true), the feature map inverse transform unit 132 acquires the image (image features) based on the position from the Y channel (step ST3C), updates the position of the Y channel (step ST4C), and adds the image to the feature map (step ST5C).

[0104] If the “channel_y” flag is not present (step ST2C; false), the feature map inverse transform unit 132 determines whether or not the “channel_u” flag is present in the metadata (step ST6C). If the “channel_u” flag is present (step ST6C; true), the feature map inverse transform unit 132 acquires the image based on the position from the U channel (step ST7C), updates the position of the U channel (step ST8C), and proceeds to step ST5C to add the image to the feature map.

[0105] If the “channel_u” flag is not present (step ST6C; false), the feature map inverse transform unit 132 acquires an image based on its position from the V channel (step ST9C), updates the position of the V channel (step ST10C), and proceeds to step ST5C to add the image to the feature map.

[0106] Furthermore, if there is a multicolor channel update flag ("multi_color_channel_update_flag") and parameter i is greater than or equal to the value of "num_fused_feat_channels", the feature map inverse transform unit 132, if the "channel_v" flag is present, acquires an image based on its position from the V channel, updates the position of the V channel, and adds the image to the feature map.

[0107] The information related to the “channel_y” flag is metadata that selects whether to store the feature at the beginning of the first channel (Y channel) or at the beginning of the second and subsequent channels (U and V channels). The decoding unit 131 decodes this information that selects whether to store at the beginning of the first channel or at the beginning of the second and subsequent channels. Similarly, the information related to the “channel_u” flag or “channel_v” flag is metadata that selects whether to store the feature at the beginning of the second channel (U channel) or at the beginning of the third and subsequent channels (V channel). The decoding unit 131 decodes this information that selects whether to store at the beginning of the second channel or at the beginning of the third channel.

[0108] (Third Example) Figure 15 shows a third example of the syntax of picture-level metadata in the hierarchical data structure of encoded stream Te and encoded stream Fe. The feature map conversion unit 112 packs the feature quantities based on pos = (u, v, c), which specifies the storage location of the feature map. The storage location is transmitted to the decoder side using metadata FPPS.

[0109] The metadata FSPS in Figure 15 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0110] The information regarding the pos(i) flag is metadata that indicates which feature map it is. The encoding unit 113 encodes this information indicating which feature map it is and transmits it to the video decoding device 13.

[0111] (Encoder operation) Figure 16 is a flowchart showing the procedure for feature decoding using the metadata in Figure 15, and shows the flow of operation by the feature map inverse transform unit 132. When the multi-channel packing flag "fsps_multi_channel_packing_flag" is set, the feature map inverse transform unit 132 checks whether or not there is a multi-color channel update flag ("multi_color_channel_update_flag").

[0112] If there is a multicolor channel update flag ("multi_color_channel_update_flag") and the parameter i is smaller than the value of "num_fused_feat_channels" (step ST1D), the feature map inverse transform unit 132 acquires an image based on pos(i) included in the metadata (step ST2D). For example, if the parameter i is the Y channel, the feature map inverse transform unit 132 acquires an image based on the position from the Y channel, updates the position of the Y channel, and adds the image to the feature map (step ST3D). Also, if the parameter i is the U channel, the feature map inverse transform unit 132 acquires an image based on the position from the U channel, updates the position of the U channel, and adds the image to the feature map. Furthermore, if the parameter i is the V channel, the feature map inverse transform unit 132 acquires an image based on the position from the V channel, updates the position of the V channel, and adds the image to the feature map.

[0113] The information regarding the pos(i) flag is metadata that indicates which feature map it is. The decoding unit 131 decodes this information indicating which feature map it is.

[0114] (Fourth Example) Figure 17 shows a fourth example of the syntax of sequence-level metadata in the hierarchical data structure of encoded streams Te and Fe. This is an example of storing the syntax that was stored in FPPS in Figure 11 in FSPS for processing.

[0115] The metadata FSPS in Figure 17 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0116] By storing the syntax previously stored in the FPPS in Figure 11 in the FSPS and sending it, it becomes unnecessary to send metadata for each frame. Also, "luma_or_chroma" is sent only if "multi_color_channel_update_flag" is "true". This mechanism of sending only when necessary reduces the amount of data transmitted. If "multi_color_channel_update_flag" is "false", the "luma_or_chroma" used in the previous sequence is used.

[0117] Note that “multi_color_channel_update_flag” may be held by both FSPS and FPPS, one each. In this case, even if the “multi_color_channel_update_flag” in FSPS is “false”, if the “multi_color_channel_update_flag” in FPPS is “true”, it will be decoded based on the “luma_or_chroma” in FPPS.

[0118] (Encoder operation) The encoder operation is the same as in the first example, so the explanation will be omitted.

[0119] (Decoder operation) The decoder operation is the same as in the first example, so the explanation will be omitted.

[0120] (Fifth Example) Figure 18 shows a fifth example of the syntax of picture-level metadata in the hierarchical data structure of encoded stream Te and encoded stream Fe. This is an example of storing the syntax that was stored in FPPS in Figure 13 in FSPS and processing it.

[0121] The metadata FSPS in Figure 18 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0122] By storing the syntax previously stored in FPPS in Figure 13 in FSPS and sending it, it becomes unnecessary to send metadata for each frame. Also, "channel_v and channel_cb" are sent only if "multi_color_channel_update_flag" is "true". By making it a mechanism to send them as needed, the amount of data transmitted is reduced. If "multi_color_channel_update_flag" is "false", "channel_v" and "channel_cb" used in the previous sequence are used.

[0123] Note that “multi_color_channel_update_flag” may be held by both FSPS and FPPS, one each. In this case, even if the “multi_color_channel_update_flag” in FSPS is “false”, if the “multi_color_channel_update_flag” in FPPS is “true”, decoding will be performed based on the “channel_v” and “channel_cb” in FPPS.

[0124] (Encoder operation) The encoder operation is the same as in the second example, so the explanation is omitted.

[0125] (Decoder operation) The decoder operation is the same as in the third example, so the explanation will be omitted.

[0126] (Sixth Example) Figure 19 shows a sixth example of the syntax of picture-level metadata in the hierarchical data structure of encoded stream Te and encoded stream Fe. This is an example of storing the syntax that was stored in FPPS in Figure 15 in FSPS and processing it.

[0127] The metadata FSPS in Figure 19 has the following parameters set: “fpps_feat_pic_parameter_set_id” is set to the FPPS identifier at the picture level. “fpps_feat_seq_parameter_set_id” is set to the FSPS identifier at the sequence level. “packed_feat_val_max” is set to the maximum value of the packed features when “dequant_bypass_flag” is “false”. “packed_feat_val_min” is set to the minimum value of the packed features when “dequant_bypass_flag” is “false”.

[0128] By storing the syntax previously stored in FPPS in Figure 15 in FSPS and sending it, it becomes unnecessary to send metadata for each frame. Also, POS is sent only when "multi_color_channel_update_flag" is "true". By making it a mechanism to send only when necessary, the amount of data transmitted is reduced. If "multi_color_channel_update_flag" is "false", the POS used in the previous sequence is used.

[0129] Note that "multi_color_channel_update_flag" can also be held by FSPS and FPPS, one each. In this case, even if the FSPS's "multi_color_channel_update_flag" is "false", if the FPPS's "multi_color_channel_update_flag" is "true", it will be decoded based on the FPPS's pos.

[0130] (Encoder operation) The encoder operation is the same as in the third example, so the explanation is omitted.

[0131] (Decoder operation) The decoder operation is the same as in the third example, so the explanation will be omitted.

[0132] In the syntax shown in Figures 11, 13, 15, 17, 18, and 19, if all features are the same as in the previous frame, feature encoding is skipped. However, if only some features are different, the other parts that are the same as the features in the previous frame are also encoded, resulting in wasted effort. Therefore, when encoding the syntax, the feature map conversion unit 112 may refer to the syntax value of the same feature in the previous frame in units of the vertical and horizontal width of a single feature, and use information (flags) indicating that it is the same as the feature in the feature map of the previous frame as the syntax for encoding. Alternatively, the feature map conversion unit 112 may refer to the syntax value of the same feature in the previous frame in units of the vertical and horizontal width of a single feature, and use information (flags) indicating that it is the same as the feature located to the left between multiple feature maps arranged in one direction as the syntax for encoding. This improves encoding efficiency. This information is encoded by the encoding unit 114 and transmitted to the video decoding device 13.

[0133] Furthermore, the decoding unit 131 decodes information indicating that one of the feature quantities in the feature map is the same as the feature quantity in the feature map of the previous frame, or the same as the feature quantity located to the left between multiple feature maps arranged in one direction. This also improves decoding efficiency.

[0134] Furthermore, the encoding unit 113 may encode the difference between one feature in the generated feature map and the feature in the feature map of the previous frame. In this way, if a feature differs from the feature in the previous frame in terms of vertical and horizontal dimensions, only the difference is encoded. This is expected to improve entropy coding. Furthermore, the encoding unit 113 may encode the difference between one feature in the generated feature map and the feature located to the left of multiple feature maps arranged in one direction. For example, in the arrangement of feature maps shown on the left side of Figure 5, the difference with the feature in the leftmost feature map is encoded. By using this information, it is possible to select whether the feature is the same as the feature in the previous frame or the feature in the feature map located to the left. Note that, as shown in Figures 13 and 15, a flag or information indicating whether a feature is the same as the same feature in the previous frame in terms of vertical and horizontal dimensions may be provided as syntax. This can improve coding efficiency. Furthermore, the decoding unit 131 may decode one feature in the feature map by decoding the difference between it and the feature in the feature map of the previous frame, or the difference between it and the feature located to the left of multiple feature maps arranged in one direction. This can improve decoding efficiency.

[0135] Even within the range of the vertical and horizontal dimensions of a single feature, there may always be regions that do not change. In this case, the encoding unit 113 may encode a flag or information indicating that it does not change at the column or row level, or it may encode a number of zeros from left to right between multiple feature maps arranged in one direction.

[0136] The feature map conversion unit 112 may perform at least one of the following: scalar quantization of one or more feature maps using multiple feature quantities extracted from image T; difference quantization with the feature map of the previous frame; and difference quantization with the leftmost feature quantity between multiple feature maps arranged in one direction. For example, in the arrangement of feature maps shown on the left side of Figure 5, difference quantization is performed with the feature quantity of the leftmost feature map. This can improve encoding efficiency.

[0137] Furthermore, the encoding unit 113 may encode information indicating which quantization method was used. For example, a flag or information may be used to indicate whether or not to perform scalar quantization, differential quantization, or quantization on a frame-by-frame basis or on a feature patch-by-feature basis. This can improve encoding efficiency. Also, the decoding unit 131 can improve the decoding efficiency of the video decoding device 13 by decoding the information indicating which quantization method was used.

[0138] In the explanation so far, we have distributed features to three channels, Y, U, and V, based on the "luma_or_chroma" flag. However, it is also possible to distribute features to two channels. For example, features may be distributed to two channels, Y and U, or to two channels, Y and V, based on the "luma_or_chroma" flag. In this case, zero will be set for channels that are not to be distributed.

[0139] (Generation of Encoded Stream Te) Figure 20 is a block diagram showing the configuration of the encoding unit 114. The encoding unit 114 is a video encoding unit that performs video encoding of image T. In Figure 20, the encoding control unit 1141 determines the maximum size of the encoding block, which is the processing unit when motion compensation prediction processing (inter-frame prediction processing) or intra-prediction processing (intra-frame prediction processing) is performed, and performs processing to determine the upper limit of the number of layers when the maximum size encoding block is divided hierarchically.

[0140] Furthermore, the encoding control unit 1141 performs the process of selecting an encoding mode suitable for each hierarchically divided encoding block from among one or more available encoding modes (one or more intra encoding modes, one or more inter encoding modes (including direct mode inter encoding modes)).

[0141] When the block division unit 1142 receives a video signal representing an image T, it divides the image T represented by the video signal into encoding blocks of the maximum size determined by the encoding control unit 1141, and performs a process of hierarchically dividing the encoding blocks until it reaches the upper limit number of layers determined by the encoding control unit 1141.

[0142] If the coding mode selected by the coding control unit 1141 is the intra coding mode, the changeover switch 1143 outputs the coded blocks divided by the block division unit 1142 to the intra prediction unit 1144. If the coding mode selected by the coding control unit 1141 is the inter coding mode, the changeover switch 1143 outputs the coded blocks divided by the block division unit 1142 to the motion compensation prediction unit 1145. When the intra prediction unit 1144 receives the coded blocks divided by the block division unit 1142 from the changeover switch 1143, it performs intra prediction processing on those coded blocks using the intra prediction parameters output from the coding control unit 1141 and generates a predicted image.

[0143] If the encoding control unit 1141 has selected the direct mode inter-encoding mode as the encoding mode suitable for the encoded block divided by the block division unit 1142, the motion compensation prediction unit 1145 generates a spatial direct mode spatial direct vector from the motion vectors of the encoded blocks located around the encoded block. The motion compensation prediction unit 1145 then generates a time direct mode time direct vector from the motion vectors of the encoded pictures that the encoded block can reference, selects the direct vector that has a higher correlation between the reference images from among the spatial direct vector and the time direct vector, and uses that direct vector to perform motion compensation prediction processing on the encoded block to generate a predicted image.

[0144] Furthermore, if the encoding control unit 1141 selects an inter-encoding mode other than direct mode as an encoding mode suitable for the encoding block divided by the block division unit 1142, the motion compensation prediction unit 1145 searches for motion vectors from the encoding block and the reference image stored in the motion compensation prediction frame memory 1152, and uses those motion vectors to perform motion compensation prediction processing on the encoding block to generate a predicted image.

[0145] The subtraction unit 1146 generates a difference image (= encoded block - predicted image) by subtracting the predicted image generated by the intra prediction unit 1144 or the motion compensation prediction unit 1145 from the encoded block divided by the block division unit 1142. The transformation quantization unit 1147 transforms the difference image generated by the subtraction unit 1146 in units of the transformation block size included in the predicted difference encoding parameters output from the encoding control unit 1141. For example, an orthogonal transformation process such as DCT (Discrete Cosine Transform) or KL transform for which a basis design has been made in advance for a specific learning sequence is performed. Then, the transformation quantization unit 1147 quantizes the transformation coefficients of the difference image using the quantization parameters included in the predicted difference encoding parameters, and outputs the quantized transformation coefficients as compressed data of the difference image.

[0146] The inverse quantization inverse transform unit 1148 inversely quantizes the compressed data output from the transform quantization unit 1147 using the quantization parameters included in the predicted difference coding parameters output from the coding control unit 1141. Then, the inverse quantization inverse transform unit 1148 inversely transforms the compressed data in inverse quantization block size units included in the predicted difference coding parameters. For example, an inverse transform process such as inverse DCT (inverse discrete cosine transform) or inverse KL transform is performed. As a result, the compressed data after the inverse transform process is output as a local decoded predicted difference signal.

[0147] The adder 1149 adds the local decoded prediction difference signal output from the inverse quantization inverse transform 1148 to the prediction signal indicating the predicted image generated by the intra prediction 1144 or the motion compensation prediction 1145. This generates a local decoded image signal indicating the local decoded image. The intra prediction memory 1150 is a memory that stores the predicted image for video encoding. The intra prediction memory 1150 stores the local decoded image indicated by the local decoded image signal generated by the adder 1149, which is used by the intra prediction 1144 in the next intra prediction process. The intra prediction memory 1150 is a storage medium such as RAM.

[0148] The loop filter unit 1151 compensates for the coding distortion contained in the locally decoded image signal generated by the adder unit 1149, and outputs the locally decoded image indicated by the locally decoded image signal after coding distortion compensation to the motion compensation prediction frame memory 1152 as a reference image. The motion compensation prediction frame memory 1152 stores the locally decoded image after filtering by the loop filter unit 1151 as a reference image to be used in the next motion compensation prediction process by the motion compensation prediction unit 1145. The motion compensation prediction frame memory 1152 is a storage medium such as RAM.

[0149] The variable-length coding unit 1153 performs variable-length coding on the compressed data output from the conversion quantization unit 1147, the coding mode and prediction difference coding parameters output from the coding control unit 1141, and the intra-prediction parameters output from the intra-prediction unit 1144 or the inter-prediction parameters output from the motion compensation prediction unit 1145. This generates a coded stream Te, which is a bitstream in which the coded data of the compressed data, coding mode, prediction difference coding parameters, intra-prediction parameters, or inter-prediction parameters are multiplexed.

[0150] (Generation of Image Td) Figure 21 is a block diagram showing the configuration of the decoding unit 133. The decoding unit 133 is a video decoding unit that decodes video including image T. In Figure 21, the variable-length decoding unit 1331 decodes the compressed data, encoding mode, prediction difference encoding parameters, intra-prediction parameters or inter-prediction parameters related to each hierarchically divided encoding block from the encoded data multiplexed in the encoded stream Te, and outputs the compressed data and prediction difference encoding parameters to the inverse quantization inverse transform unit 1335. Furthermore, the variable-length decoding unit 1331 outputs the encoding mode and intra-prediction parameters or inter-prediction parameters to the changeover switch 1332.

[0151] The changeover switch 1332 outputs the intra prediction parameters output from the variable-length decoding unit 1331 to the intra prediction unit 1333 if the encoding mode for the encoded block output from the variable-length decoding unit 1331 is the intra encoding mode, and outputs the inter prediction parameters output from the variable-length decoding unit 1331 to the motion compensation prediction unit 1334 if the encoding mode is the inter encoding mode. The intra prediction unit 1333 uses the intra prediction parameters output from the changeover switch 1332 to perform intra prediction processing for the encoded block and generate a predicted image.

[0152] If the encoding mode for an encoded block output from the variable-length decoding unit 1331 is an inter-encoding mode of direct mode, the motion compensation prediction unit 1334 generates a spatial direct vector of spatial direct mode from the motion vectors of decoded blocks located around the encoded block. The motion compensation prediction unit 1334 then generates a time direct vector of time direct mode from the motion vectors of decoded pictures that the encoded block can reference, selects the direct vector that has a higher correlation between the reference images from among the spatial direct vector and the time direct vector, and uses that direct vector to perform motion compensation prediction processing on the encoded block to generate a predicted image.

[0153] Furthermore, if the encoding mode for the encoded block output from the variable-length decoding unit 1331 is an inter-encoding mode other than direct mode, the motion compensation prediction unit 1334 uses the motion vector included in the inter-prediction parameters output from the variable-length decoding unit 1331 to perform motion compensation prediction processing for the encoded block and generate a predicted image.

[0154] The inverse quantization inverse transform unit 1335 uses the quantization parameters included in the predicted difference coding parameters output from the variable-length decoding unit 1331 to inverse quantize the compressed data relating to the coding blocks output from the variable-length decoding unit 1331. Then, the inverse quantization inverse transform unit 1335 inversely transforms the inversely quantized compressed data in units of the transformation block size included in the predicted difference coding parameters. For example, an inverse transform process such as inverse DCT (inverse discrete cosine transform) or inverse KL transform is performed. As a result, the compressed data after the inverse transform process is output as a decoded predicted difference signal (a signal indicating the difference image before compression).

[0155] The adder 1336 generates a decoded image signal representing the decoded image by adding the decoded prediction difference signal output from the inverse quantization inverse transform 1335 to a prediction signal representing the prediction image generated by the intra prediction 1333 or the motion compensation prediction 1334. The intra prediction memory 1337 is a memory that stores the prediction image generated when decoding the video. The intra prediction memory 1337 stores the decoded image represented by the decoded image signal generated by the adder 1336 as the image to be used in the next intra prediction process by the intra prediction 1333. The intra prediction memory 1337 is a storage medium such as RAM.

[0156] The loop filter unit 1338 compensates for encoding distortion contained in the decoded image signal generated by the adder unit 1336, and outputs the image Td indicated by the decoded image signal after encoding distortion compensation to the motion compensation prediction frame memory 1339 as a reference image. The motion compensation prediction frame memory 1339 stores the image Td after filtering by the loop filter unit 1338 as a reference image to be used in the next motion compensation prediction process by the motion compensation prediction unit 1334. The motion compensation prediction frame memory 1339 is a storage medium such as RAM.

[0157] Next, the hardware configuration that realizes the functions of the video encoding device 11 and the video decoding device 13 will be described. The functions of the feature extraction unit 111, the feature map conversion unit 112, and the encoding unit 113 of the video encoding device 11 are realized by a processing circuit. That is, the video encoding device 11 includes a processing circuit for executing the processes from step ST1 to step ST5 shown in Figure 4. The processing circuit may be dedicated hardware, or it may be a CPU (Central Processing Unit) that executes a program stored in memory.

[0158] The functions of the decoding unit 131 and the feature map inverse conversion unit 132 of the video decoding device 13 are realized by a processing circuit. That is, the video decoding device 13 includes a processing circuit for executing the processes from step ST1A to step ST4A shown in Figure 6. The processing circuit may be dedicated hardware, or it may be a CPU that executes a program stored in memory.

[0159] Figure 22A is a block diagram showing the hardware configuration for realizing the functions of the video encoding device 11 and the video decoding device 13. Figure 22B is a block diagram showing the hardware configuration for executing the software that realizes the functions of the video encoding device 11 and the video decoding device 13. In Figures 22A and 22B, the feature extraction unit 111 of the video encoding device 11 acquires an image T from an external device (e.g., a camera device, a data server, etc.) via the input interface 100. The encoding unit 113 of the video encoding device 11 outputs the encoded stream to the network 12 via the output interface 101. The memory 115 shown in Figure 2 is the memory 104 shown in Figure 22B. Alternatively, the memory 115 may be an external storage device from which data is read from the encoding unit 113 via the input interface 100 and written by the feature map conversion unit 112 via the output interface 101.

[0160] The decoding unit 131 of the video decoding device 13 acquires the encoded stream from the video encoding device 11 via the input interface 100. The feature map inverse transform unit 132 outputs the feature map Fd to the analysis device 15 via the output interface 101. The memory 134 shown in Figure 3 is the same as the memory 104 shown in Figure 22B. Alternatively, the memory 134 may be an external storage device from which data is read from the feature map inverse transform unit 132 via the input interface 100 and from which data is written from the decoding unit 131 via the output interface 101.

[0161] If the processing circuit is a dedicated hardware processing circuit 102 as shown in Figure 22A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The features extraction unit 111, the feature map conversion unit 112, and the encoding unit 113 of the video encoding device 11 may be implemented in separate processing circuits, or these functions may be implemented together in a single processing circuit. Similarly, the decoding unit 131 and the feature map inverse conversion unit 132 of the video decoding device 13 may be implemented in separate processing circuits, or these functions may be implemented together in a single processing circuit.

[0162] When the processing circuit is the processor 103 shown in Figure 22B, the functions of the feature extraction unit 111, feature map conversion unit 112, and encoding unit 113 of the video encoding device 11 are realized by software, firmware, or a combination of software and firmware. Similarly, the functions of the decoding unit 131 and feature map inverse conversion unit 132 of the video decoding device 13 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 104.

[0163] The processor 103 reads and executes a program stored in the memory 104 to realize the functions of the feature extraction unit 111, the feature map conversion unit 112, and the encoding unit 113 of the video encoding device 11. For example, the video encoding device 11 includes a memory 104 for storing a program in which steps ST1 to ST5 shown in Figure 4 are executed when executed by the processor 103. These programs cause the computer to execute the procedures or methods of processing performed by the feature extraction unit 111, the feature map conversion unit 112, and the encoding unit 113. The memory 104 may be a computer-readable storage medium in which a program is stored that causes the computer to function as the feature extraction unit 111, the feature map conversion unit 112, and the encoding unit 113.

[0164] Similarly, the processor 103 realizes the functions of the decoding unit 131 and the feature map inverse conversion unit 132 of the video decoding device 13 by reading and executing a program stored in the memory 104. For example, the video decoding device 13 includes a memory 104 for storing a program in which steps ST1A to ST4A shown in Figure 6 are ultimately executed when executed by the processor 103. These programs cause the computer to execute the procedures or methods of processing performed by the decoding unit 131 and the feature map inverse conversion unit 132. The memory 104 may be a computer-readable storage medium in which a program is stored that causes the computer to function as the decoding unit 131 and the feature map inverse conversion unit 132.

[0165] Memory 104 includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM) (registered trademark), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs, and the like.

[0166] Some functions of the feature extraction unit 111, feature map conversion unit 112, and encoding unit 113 of the video encoding device 11 may be implemented by dedicated hardware, while other functions may be implemented by software or firmware. For example, the function of the feature extraction unit 111 may be implemented by a processing circuit 102, which is dedicated hardware, and the functions of the feature map conversion unit 112 and encoding unit 113 may be implemented by the processor 103 reading and executing a program stored in memory 104. Similarly, some functions of the decoding unit 131 and feature map inverse conversion unit 132 of the video decoding device 13 may be implemented by dedicated hardware, while other functions may be implemented by software or firmware. For example, the function of the decoding unit 131 may be implemented by a processing circuit 102, which is dedicated hardware, and the function of the feature map inverse conversion unit 132 may be implemented by the processor 103 reading and executing a program stored in memory 104. In this way, the processing circuit can implement the above functions by hardware, software, firmware, or a combination thereof.

[0167] The video encoding device 11 or the video decoding device 13 may be implemented by a computer system that reads and executes a program to realize its function. The term "computer system" refers to a computer system built into either the video encoding device 11 or the video decoding device 13, and includes hardware such as an operating system or peripheral devices. Furthermore, the computer system may acquire the program via a network such as the Internet or a communication line such as a telephone line, or it may acquire the program from a computer-readable storage medium or device.

[0168] Furthermore, part or all of the video encoding device 11 or video decoding device 13 may be implemented using an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 or video decoding device 13 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology, an integrated circuit using that technology may be used.

[0169] As described above, the video encoding device 11 according to Embodiment 1 includes a feature map conversion unit 112 that selects and packs either a single feature map or multiple feature maps using a plurality of feature quantities extracted from an image T, and an encoding unit 113 that encodes using the selected single or multiple feature maps. As a result, the video encoding device 11 can efficiently encode feature maps.

[0170] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 uses a plurality of feature quantities extracted from the image T to encode metadata that allows the number of feature maps to be selected, either in sequence or picture units. This enables the video encoding device 11 to efficiently encode feature maps.

[0171] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 encodes information about the update frequency when the number of feature maps is changed, using a plurality of feature quantities extracted from the image T. This enables the video encoding device 11 to efficiently encode feature maps.

[0172] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 encodes a syntax that indicates which feature map is being used when selecting and packing multiple feature maps using multiple feature quantities extracted from an image T. This enables the video encoding device 11 to efficiently encode feature maps.

[0173] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 uses a plurality of feature quantities extracted from the image T to select and pack three feature maps, encoding metadata to select whether to start with the first map or the second or subsequent maps, and metadata to select whether to start with the second or third map. This allows the video encoding device 11 to efficiently encode feature maps.

[0174] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 encodes metadata indicating that one of the feature quantities in the generated feature map is the same as the feature quantity in the feature map of the previous frame, or the same as the feature quantity located to the left between multiple feature maps arranged in one direction. This enables the video encoding device 11 to efficiently encode feature maps.

[0175] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 encodes the difference between one feature quantity in the generated feature map and the feature quantity in the feature map of the previous frame, or the difference between one feature quantity and the feature quantity located to the left of a plurality of feature maps arranged in one direction. This allows the video encoding device 11 to efficiently encode feature maps.

[0176] In the video encoding device 11 according to Embodiment 1, the feature map conversion unit 112 performs at least one of the following: scalar quantization of one or more feature maps using multiple feature quantities extracted from an image; difference quantization with the feature map of the previous frame; and difference quantization with the left feature quantity between multiple feature maps arranged in one direction. As a result, the video encoding device 11 can efficiently encode feature maps.

[0177] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 encodes information indicating which quantization method was used for quantization. This enables the video encoding device 11 to efficiently encode feature maps.

[0178] The video encoding device 11 according to Embodiment 1 includes a memory 115 for storing feature maps. This allows the video encoding device 11 to efficiently encode feature maps.

[0179] The video encoding device 11 according to Embodiment 1 includes an encoding unit 114 that performs video encoding of image T, and an intra-prediction memory 1150 that stores a predicted image for video encoding. As a result, the video encoding device 11 can efficiently encode feature maps.

[0180] The video decoding device 13 according to Embodiment 1 includes a decoding unit 131 that decodes metadata, which allows the selection of the number of feature maps composed of multiple feature quantities extracted from an image T, on a sequence or picture basis, and a feature map inverse conversion unit 132 that, based on the decoded metadata, inversely converts one or more packed feature maps back into multiple feature quantities extracted from the image. As a result, the video decoding device 13 can efficiently decode feature maps.

[0181] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes metadata regarding the update frequency when the number of feature maps is changed. This enables the video decoding device 13 to efficiently decode the feature maps.

[0182] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes information indicating which frame the encoded feature map is in. This allows the video decoding device 13 to efficiently decode the feature map.

[0183] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes information to select whether to start with the first feature map or the second or subsequent maps, and information to select whether to start with the second or third map, when three feature maps have been generated. This allows the video decoding device 13 to efficiently decode the feature maps.

[0184] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes information indicating that one feature quantity in the feature map is the same as a feature quantity in the feature map of the previous frame, or the same as a feature quantity located to the left between multiple feature maps arranged in one direction. This enables the video decoding device 13 to efficiently decode the feature maps.

[0185] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes the difference between one feature quantity in the feature map and the feature quantity in the feature map of the previous frame, or the difference between one feature quantity and the feature quantity located to the left of a plurality of feature maps arranged in one direction. As a result, the video decoding device 13 can efficiently decode the feature maps.

[0186] In the video decoding device 13 according to Embodiment 1, the feature map inverse transform unit 132 performs at least one of the following: inverse scalar quantization of one or more feature maps, difference inverse quantization with the feature map of the previous frame, and difference inverse quantization with the left feature quantity between multiple feature maps aligned in one direction. This enables the video decoding device 13 to efficiently decode feature maps.

[0187] In the video decoding device 13 according to Embodiment 1, the decoding unit 131 decodes information indicating which quantization method was used for quantization. This enables the video decoding device 13 to efficiently decode the feature map.

[0188] The video decoding device 13 according to Embodiment 1 includes a memory 134 for storing feature maps. This allows the video decoding device 13 to efficiently decode feature maps.

[0189] The video decoding device 13 according to Embodiment 1 includes a decoding unit 133 that decodes video including images, and an intra-prediction memory 1337 that stores a predicted image generated when decoding the video. As a result, the video decoding device 13 can efficiently decode feature maps.

[0190] The video encoding method according to Embodiment 1 comprises steps ST2 to ST4 in which the feature map conversion unit 112 selects and packs either one feature map or multiple feature maps using a plurality of feature quantities extracted from the image T, and step ST5 in which the encoding unit 114 encodes using the selected one or multiple feature maps. By executing the video encoding method, the video encoding device 11 can efficiently encode the feature maps.

[0191] The video decoding method according to Embodiment 1 includes steps ST1A in which the decoding unit 131 decodes metadata that allows the selection of the number of feature maps composed of a plurality of feature quantities extracted from an image T, in sequence or picture units, and steps ST2A to ST4A in which the feature map inverse transformation unit 132 inversely transforms one or more packed feature maps into a plurality of feature quantities extracted from the image based on the decoded metadata. The video decoding device 13 can efficiently decode the feature maps by executing the video decoding method described above.

[0192] The video decoding device 13 according to Embodiment 1 includes a task processing unit that performs tasks using feature quantities inversely transformed by the feature map inverse transformation unit 132. As a result, the task processing unit performs task processing using efficiently decoded feature maps, thereby improving the analysis performance of the video decoding device 13.

[0193] The image transmission system according to Embodiment 1 includes a video decoding device 13 comprising: a decoding unit that decodes information on a sequence or picture basis, where the number of feature maps composed of multiple feature quantities extracted from an image can be selected; a feature map inverse conversion unit that, based on the decoded information, inversely converts one or more packed feature maps back into multiple feature quantities extracted from an image; and an analysis device 15 that performs a task using the feature quantities generated by the video decoding device 13. As a result, the analysis device 15 can perform task processing using the efficiently decoded feature maps, thereby improving the performance of the analysis device 15.

[0194] In the video encoding device 11 according to Embodiment 1, the encoding unit 113 uses a plurality of feature quantities extracted from the image T to determine metadata that allows the number of feature maps to be selected, using either a supervised or unsupervised method. This enables the video encoding device 11 to efficiently encode feature maps.

[0195] While embodiments have been described with reference to the drawings, the specific configurations of the video encoding and decoding devices according to this disclosure are not limited to those described above. For example, as described in Reference 1 below, it is possible to replace each of the processes such as the intra-prediction unit 1144, intra-prediction unit 1333, motion compensation prediction unit 1145, and motion compensation prediction unit 1334 with a method using deep learning. In other words, the video encoding and decoding devices according to this disclosure can be modified in various ways without departing from the gist of this disclosure. (Reference 1) Institute of Electronics, Information and Communication Engineers, "How Deep Learning Has Changed Information and Systems Research: Special Feature 5. (Video Coding) The Spread of Deep Learning in Video Coding and Distribution", IEICE Journal, Vol. 105 No. 5 pp. 381-386 May 2022

[0196] The video encoding device described herein is applicable to fields in which image data is encoded and encoded data obtained through encoding is handled. The video decoding device described herein is applicable to fields in which encoded data is decoded and image data obtained through decoding is handled.

[0197] Furthermore, even if only some of the encoding and decoding processes using the syntax described above are implemented, improvements in encoding and decoding efficiency can be expected. Therefore, the following notes are provided, divided into several concepts. Any combination of the following notes is possible.

[0198] (Note 1) A video encoding device for encoding feature maps, comprising: a feature map conversion unit that selects and packs either one feature map or multiple feature maps using a plurality of feature quantities extracted from an image; and an encoding unit that encodes using the selected one or multiple feature maps. (Note 2) The video encoding device according to Note 1, wherein the encoding unit encodes information that allows the number of feature maps to be selected on a sequence or picture basis using a plurality of feature quantities extracted from an image. (Note 3) The video encoding device according to Note 1 or Note 2, wherein the encoding unit encodes information regarding the update frequency when the number of feature maps is changed using a plurality of feature quantities extracted from an image. (Note 4) The video encoding device according to any one of Notes 1 to 3, wherein the encoding unit encodes information indicating which feature map it is when selecting and packing multiple feature maps using a plurality of feature quantities extracted from an image. (Note 5) The video encoding device according to any one of Notes 1 to 3, wherein the encoding unit encodes information to select whether to start with the first or second or later feature map, and information to select whether to start with the second or third feature map, when selecting and packing three feature maps using multiple feature quantities extracted from an image. (Note 6) The video encoding device according to any one of Notes 1 to 3, wherein the encoding unit encodes information indicating that one feature quantity in the generated feature map is the same as a feature quantity in the feature map of the previous frame, or the same as a feature quantity located to the left between multiple feature maps arranged in one direction. (Note 7) The video encoding device according to any one of Notes 1 to 3, wherein the encoding unit encodes the difference between one feature quantity in the generated feature map and a feature quantity in the feature map of the previous frame, or the difference between one feature quantity in the generated feature map and a feature quantity located to the left between multiple feature maps arranged in one direction. (Note 8) The feature map conversion unit performs at least one of the following: scalar quantization of one or more feature maps using multiple feature quantities extracted from an image; difference quantization with the feature map of the previous frame; and difference quantization with the left feature quantity between multiple feature maps arranged in one direction, as described in any one of Notes 1 to 3.(Note 9) The video encoding device according to Note 8, wherein the encoding unit encodes information indicating which quantization method was used for quantization. (Note 10) The video encoding device according to any one of Notes 1 to 3, Notes 6 and 7, further comprising a memory for storing feature maps. (Note 11) The video encoding device according to any one of Notes 1 to 3, Notes 6, Notes 7 and 10, further comprising a video encoding unit for performing video encoding of an image, and a memory for storing a predicted image when performing video encoding. (Note 12) A video decoding device for decoding feature maps, comprising: a decoding unit that decodes information on the number of feature maps composed of a plurality of feature quantities extracted from an image in sequence or picture units; and a feature map inverse transformation unit that, based on the decoded information, inversely transforms one or more packed feature maps into a plurality of feature quantities extracted from an image. (Note 13) The video decoding device according to Note 12, wherein the decoding unit decodes information on the update frequency when the number of feature maps is changed. (Note 14) The video decoding device according to Note 12 or Note 13, wherein the decoding unit decodes information indicating which frame the encoded feature map is in. (Note 15) The video decoding device according to any one of Notes 12 to 14, wherein, when three feature maps are generated, the decoding unit decodes information to select whether to start with the first or second or later feature map, and information to select whether to start with the second or third feature map. (Note 16) The video decoding device according to any one of Notes 12 to 14, wherein the decoding unit decodes information indicating that one feature quantity in the feature map is the same as a feature quantity in the feature map of the previous frame, or the same as a feature quantity located to the left between multiple feature maps arranged in one direction. (Note 17) The video decoding device according to any one of Notes 12 to 14, wherein the decoding unit decodes the difference between one feature quantity in the feature map and a feature quantity in the feature map of the previous frame, or the difference between one feature quantity located to the left of a plurality of feature maps arranged in one direction.(Note 18) The feature map inverse transform unit performs at least one of the following: inverse scalar quantization of one or more feature maps, difference inverse quantization with the feature map of the previous frame, and difference inverse quantization with the left feature quantity between a plurality of feature maps arranged in one direction, the video decoding device according to any one of Notes 12 to 14. (Note 19) The decoding unit decodes information indicating which quantization method was used, the video decoding device according to Note 18. (Note 20) The video decoding device according to Note 12, comprising a memory for storing feature maps. (Note 21) The video decoding device according to Note 12, comprising a video decoding unit for decoding video including images, and a memory for storing predicted images generated when decoding video. (Note 22) A video encoding method by a video encoding device that encodes feature maps, comprising: a step of a feature map conversion unit selecting and packing one feature map or multiple feature maps using a plurality of feature quantities extracted from an image; and a step of an encoding unit encoding using the selected one or multiple feature maps. (Note 23) A video decoding method by a video decoding device that decodes feature maps, comprising: a step of a decoding unit decoding information on a sequence or picture basis that allows the selection of the number of feature maps composed of a plurality of feature quantities extracted from an image; and a step of a feature map inverse conversion unit inversely converting the packed one or multiple feature maps into a plurality of feature quantities extracted from an image based on the decoded information. (Note 24) The video decoding device according to Note 12, further comprising a task processing unit that performs a task using the feature quantities inversely converted by the feature map inverse conversion unit. (Note 25) A video decoding device comprising: a decoding unit that decodes information on a sequence or picture basis, in which the number of feature maps composed of multiple feature quantities extracted from an image can be selected; a feature map inverse conversion unit that, based on the decoded information, inversely converts one or more packed feature maps into multiple feature quantities extracted from an image; and an analysis device that performs a task using the feature quantities generated by the video decoding device.(Note 26) The video encoding device described in Note 2, wherein the encoding unit determines information on the number of feature maps that can be selected using a supervised or unsupervised method, using a plurality of feature quantities extracted from the image.

[0199] Furthermore, any component of the embodiment can be modified or any component of the embodiment can be omitted.

[0200] 1 Image transmission system, 11 Video encoding device, 12 Network, 13 Video decoding device, 14 Display device, 15 Analysis device, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104, 115 Memory, 111 Feature extraction unit, 112 Feature map conversion unit, 113, 114 Encoding unit, 131, 133 Decoding unit, 132 Feature map inverse conversion unit, 134 Memory, 1141 Encoding control unit, 1142 Block division unit, 1143, 1332 Switching switch, 1144, 1333 Intra prediction unit, 1145, 1334 Motion compensation prediction unit, 1146 Subtraction unit, 1147 Transformation quantization unit, 1148 Inverse quantization inverse conversion unit, 1149, 1336 Addition unit, 1150, 1337 Intra prediction memory, 1151 Loop filter section, 1152, 1339 Motion-compensated prediction frame memory, 1153 Variable-length coding section, 1331 Variable-length decoding section, 1335 Inverse quantization inverse transform section, 1338 Loop filter section.

Claims

1. A video encoding device for encoding feature maps, comprising: a feature map conversion unit that selects and packs either a single feature map or multiple feature maps using a plurality of feature quantities extracted from an image; and an encoding unit that encodes using the selected single or multiple feature maps.

2. The video encoding device according to claim 1, wherein the encoding unit encodes information on a sequence or picture basis, where the number of feature maps can be selected, using a plurality of feature quantities extracted from the image.

3. The video encoding device according to claim 1 or claim 2, wherein the encoding unit encodes information on the update frequency when the number of feature maps is changed, using a plurality of feature quantities extracted from the image.

4. The video encoding device according to any one of claims 1 to 3, wherein the encoding unit encodes information indicating which feature map it is when selecting and packing multiple feature maps using multiple feature quantities extracted from an image.

5. The video encoding device according to any one of claims 1 to 3, wherein the encoding unit, when selecting and packing three feature maps using a plurality of feature quantities extracted from an image, encodes information for selecting whether to start with the first map or the second or later map, and information for selecting whether to start with the second map or the third map.

6. The video encoding device according to any one of claims 1 to 3, wherein the encoding unit encodes information indicating that one of the generated feature quantities in the feature map is the same as a feature quantity in the feature map of the previous frame or the same as a feature quantity located to the left between a plurality of feature maps arranged in one direction.

7. The video encoding device according to any one of claims 1 to 3, wherein the encoding unit encodes the difference between one feature quantity in the generated feature map and a feature quantity in the feature map of the previous frame, or the difference between one feature quantity located to the left of a plurality of feature maps arranged in one direction.

8. The motion image encoding device according to any one of claims 1 to 3, wherein the feature map conversion unit performs at least one of the following: scalar quantization of one or more feature maps using a plurality of feature quantities extracted from an image; difference quantization with the feature map of the previous frame; and difference quantization with the left feature quantity between a plurality of feature maps arranged in one direction.

9. The video encoding device according to claim 8, wherein the encoding unit encodes information indicating which quantization method was used for quantization.

10. A video encoding device according to any one of claims 1 to 3, 6, and 7, comprising a memory for storing a feature map.

11. A video encoding device according to any one of claims 1 to 3, 6, 7, and 10, comprising: a video encoding unit that performs video encoding of an image; and a memory that stores a predicted image for video encoding.

12. A video decoding device for decoding feature maps, comprising: a decoding unit that decodes information on a sequence or picture basis, allowing the selection of the number of feature maps composed of multiple feature quantities extracted from an image; and a feature map inverse conversion unit that, based on the decoded information, inversely converts one or more packed feature maps into multiple feature quantities extracted from an image.

13. The motion image decoding apparatus according to claim 12, wherein the decoding unit decodes information on the update frequency when the number of feature maps is changed.

14. The motion image decoding apparatus according to claim 12 or 13, wherein the decoding unit decodes information indicating which frame the encoded feature map is.

15. The motion image decoding apparatus according to any one of claims 12 to 14, wherein, when three feature maps have been generated, the decoding unit decodes information to select whether to start with the first feature map or a second or subsequent feature map, and information to select whether to start with the second feature map or a third feature map.

16. The motion image decoding apparatus according to any one of claims 12 to 14, wherein the decoding unit decodes information indicating that one feature quantity in the feature map is the same as a feature quantity in the feature map of the previous frame or the same as a feature quantity located to the left between a plurality of feature maps arranged in one direction.

17. The motion image decoding apparatus according to any one of claims 12 to 14, wherein the decoding unit decodes the difference between one feature quantity in the feature map and a feature quantity in the feature map of the previous frame, or the difference between one feature quantity located to the left of a plurality of feature maps arranged in one direction.

18. The motion image decoding apparatus according to any one of claims 12 to 14, wherein the feature map inverse transform unit performs at least one of the following: inverse scalar quantization of one or more feature maps, difference inverse quantization with the feature map of the previous frame, and difference inverse quantization with the left feature quantity between a plurality of feature maps aligned in one direction.

19. The motion image decoding apparatus according to claim 18, wherein the decoding unit decodes information indicating which quantization method was used for quantization.

20. The motion image decoding device according to claim 12, comprising a memory for storing a feature map.

21. The video decoding device according to claim 12, comprising: a video decoding unit for decoding video including images; and a memory for storing predicted images generated when decoding video.

22. A video encoding method using a video encoding device for encoding feature maps, comprising: a feature map conversion unit selecting and packing either a single feature map or multiple feature maps using a plurality of feature quantities extracted from an image; and an encoding unit encoding using the selected single or multiple feature maps.

23. A video decoding method using a video decoding device for decoding feature maps, comprising: a decoding unit decoding information on a sequence or picture basis, wherein the number of feature maps composed of a plurality of feature quantities extracted from an image is selectable; and a feature map inverse transformation unit inversely transforming one or more packed feature maps into a plurality of feature quantities extracted from an image based on the decoded information.

24. The motion image decoding apparatus according to claim 12, further comprising a task processing unit that performs a task using the feature quantities inversely transformed by the feature map inverse transformation unit.

25. A video decoding device comprising: a decoding unit that decodes information on a sequence or picture basis, where the number of feature maps composed of multiple feature quantities extracted from an image can be selected; a feature map inverse transformation unit that, based on the decoded information, inversely transforms one or more packed feature maps into multiple feature quantities extracted from an image; and an analysis device that performs a task using the feature quantities generated by the video decoding device.

26. The video encoding device according to claim 2, wherein the encoding unit determines information on the number of feature maps that can be selected using a supervised or unsupervised method, using a plurality of feature quantities extracted from the image.

Images