Systems and methods for non-linear transformation for video coding and decoding in feature coding for machines

Abstract

Systems and methods presenting improvements in encoding and decoding of video for machine-based applications are provided which include modified feature conversion with non-linear transform of tensors of features and / or temporal consistency processing of tensors at an encoder and complimentary inverse processes at the decoder.

Description

SYSTEMS AND METHODS FOR NON-LINEAR TRANSFORMATION FOR VIDEO CODING AND DECODING IN FEATURE CODING FOR MACHINES Statement of Related Applications

[0001] The present application claims the benefit of priority to U. S. Provisional Application serial no. 63 / 710,275 filed on October 22, 2024, and entitled Systems and Methods for Non-linear Transformation for Feature Coding for Machines, and also claims the benefit of priority to U. S. Provisional Application serial no. 63 / 711,741 filed on October 25, 2024, and entitled Systems and Methods for Temporally Consistent Feature Conversion for Feature Coding for Machines, the disclosures of which are hereby incorporated by reference in their entireties.Background

[0002] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines and for machine use rather than human viewing. Indeed, an exemplary system or solution employing thousands of cameras would produce significant amounts of video that cannot be monitored by humans in a cost-effective manner. Because of the significant volume of the data, an efficient compression system is desirable for Machine-to-Machine (M2M) communication. To standardize the coding for machines to facilitate a Machine-to-Machine (M2M) communication system more efficiently, MPEG issued a Call for Proposal (CfP) for Coding for Machines in 2022.

[0003] Currently, M2M communications are typically managed by two distinct systems, namely, edge computing or local computing systems and remote computing or Video Coding for Machines (VCM) systems. In the context of edge computing systems, the complete Convolutional Neural Network (CNN) is typically executed on edge devices. However, end devices often lack the computational capacity to run extensive networks, requiring billions of operations for inference result calculations. In contrast, in video coding for machine systems, the video is typically compressed initially and transmitted to a base server or cloud server with more computing resources to execute the full CNN. The calculated results are then subsequently relayed back to the edge devices. However, video coding for machine systems may not be able to support compute offload and may fail to fully leverage the potential of end devices, as they execute the complete CNN on the base server. To overcome these limitations for M2M communication and to enable collaborative intelligence and split computing for M2M connections, MPEG has issued a Call for Proposal (CfP) for Feature Coding for Machines (FCM) technology in 2023. As used herein, the term Feature Coding for Machines or FCM is intended to apply generally to systems and methodsfor feature coding of video and similar input data for use in transferring information in systems primarily intended for machine consumption and / or machine to machine applications without regard to any one particular standard.

[0004] Figure 1 is a simplified block diagram providing a high-level illustration of a CODEC for a feature coding for machine (FCM) system. It will be appreciated that, as used herein, the terms feature coding for machines and FCM refer generally to any machine-to-machine application using feature representation of data, such as video, and is not intended to be limited to any particular coding standard. A resource limited edge device receives input video 110 and uses a split neural network part 1 115 (also referred to herein as NN front end or head) to extract features of the video which are encoded by FCM encoder 120. The encoded features are transmitted over a channel to a base server which decodes the features. The base server includes an FCM decoder 130 which provides decoded feature maps 135 to a neural network part 2 140 (also referred to herein as NN back end or tail) which provides output to inference results 145.

[0005] Referring to Figure 1, in FCM systems, a Neural Network is first split into two parts, i.e., NN Part 1 (Split CNN - Head) 115 and NN Part 2 (Split CNN - Tail) 140. Instead of running the full CNN on the end devices, only the NN Part 1 115 is executed on the resource-limited devices, and the NN Part 2 140 is preferably executed on the base server which is expected to have more computing resources. Since the NN Part 1 115 is generally executed on the edge devices, instead of transmitting videos to the base server (like in Video Coding for Machine), the intermediate feature data extracted from NN Part 1 is sent to the base server. However, the volume of intermediate feature data is usually much larger than the video itself and needs efficient compression before sending it to the base server. An example of such a system is depicted in Figure 1.

[0006] Referring to Fig. 2, which is a block diagram further illustrating an exemplary FCM system, an FCM Encoder 205 which receives feature maps 210 from an NN part 1 115 consists of three main modules, e.g., feature reduction 215, feature conversion 220, and inner codec 225 for encoding sequences.

[0007] The FCM architecture employs a split neural network, typically a CNN. In the context of CNNs, input data, output data and transformations with the neural network are generally represented using tensors. A tensor is the primary data structure used by a CNN and is a mathematical representation of a concept in the form of an n-dimensional array. For example, tensors can represent concepts such as scalar (0-dimensional tensor), vector (1-dimensional tensor), matrix (2-dimensional tensor), and an nd-array (n-dimensional tensor). Typical tensors occurring in the CNNs are 3-dimensional, e.g. [c, w, h], where c represents the number of channels, and w and h represent the width and height of each channel.

[0008] As noted in Figure 1, the FCM encoder provides an encoded bitstream over a communication link (e.g., wired or wireless communication link) to a base receiver including an FCM decoder. Referring to Fig. 2, FCM decoder 235 consists of three principal modules, e.g., an inner codec 240, inverse feature conversion 245, and feature restoration 250 for decoding the sequences with feature maps 255. The input of the FCM encoder 205 is intermediate feature data (Xt) extracted from NN Part 1 115 (Fig. 1), and the output of the FCM decoder is reconstructed intermediate feature data (Xt'), as shown in Figure 2, which is applied to NN Part 2 140 (Fig. 1). After reconstruction of the intermediate feature data (Xt'), the NN Part 2 is executed to generate the inference results. Feature data usually consists of multiple layers. And each layer usually consists of multiple channels. For instance, in one example, the backbone of detectron2 (NN Part 1) may consist of 4 layers and each layer consists of 256 channels. The details of all the FCM encoder and decoder modules of Figure 2 are described in more detail below.

[0009] The feature reduction module 215 of the FCM encoder 205 takes original feature maps 210 and / or layers as input and outputs a reduced number of feature maps and / or layers representing the original data. Essentially, feature reduction module 215 takes the data from many layers from the NN Part 1 and conveys it as a single layer. The feature reduction module 215 can be a neural network, auto-encoder, or just classical statistical approaches like PCA or clustering methods to reduce the number of feature maps and / or layers.

[0010] The feature conversion module 220 of the FCM encoder 205 receives reduced feature maps and / or layers as input and converts the feature tensor data from floating point 32-bit to unsigned integers, such as 8bit or lObit integers. During feature conversion from Floating Point 32-bit to 8-bit or 10-bit, min-max normalization may be applied, followed by uniform quantization according to bit width. In some cases, depending on the data, it may also pack the feature maps into one frame. Then, the packed frame or converted feature maps and / or layers are converted by the feature conversion module to a suitable video format, such as YUV 4:0:0, before sending it to the inner codec 225.

[0011] The inner codec (encoder) module 225 takes the video format signal representing the feature maps as input and outputs a bitstream to transmit over a communication channel 230 to the receiver 235. The inner codec (encoder) 225 could be a Neural Network encoder or any traditional video encoder such as HEVC, VVC, AVI, AVC, VVenC, or a combination of both Neural Network encoder and traditional encoder.

[0012] Turning to the FCM decoder 235, the Inner Codec (Decoder) module 240 receives the encoded bitstream as input and decompresses the bitstream file to a suitable format, such as YUV 4:0:0. Like the inner codec (encoder) module 225, the inner codec (decoder) 240 can beimplemented by a Neural Network based decoder or any traditional decoder like HEVC, VVC, VVdeC or the combination of both Neural Network based decoder and traditional decoder.

[0013] The inverse feature conversion module 245 takes the decoded YUV 4:0:0 from Inner Codec 240 as input and unpacks (if necessary) and converts the unsigned integers to floating point values. Min-max value sent from the encoder may be used to convert the unsigned integer to Floating Point 32-bit. The feature restoration module 250 receives feature tensors from the inverse feature conversion module 245 as input and restores the feature tensors from reduced size to the original size. Like the feature reduction module 215, the feature restoration module 250 can also be a Neural Network, Auto-Decoder, or just classical statistical approaches like Inverse PCA or declustering to restore the original size of feature maps and layers.

[0014] Documents which may provide relevant background information for the present disclosure include: Yuan Zhang, Manouchehr Rafie, Shan Liu, Christopher Hollmann, “[N00164] Call for Proposals on Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / W G 2, Jan. 2022; C. Rosewarne (Canon) and Y. Zhang (China Telecom), “[N00282] Call for Proposals on Feature Compression for Video Coding for Machines.” ISO / IEC JTC 1 / SC 29 / WG 2, Jan. 2023; WG 04 MPEG Video coding, “[N00460] Algorithm description of FCTM.” Jan. 2024; and Y. Wu, A. Kirillov, F. Massa, W. Y. Lo, and R. Girshick, “Detectron2,” 2019, Feb. 2024 (Online).Summary of the Disclosure

[0015] In one aspect of the present disclosure a video encoder, and associated methods for video encoding, for machine-based video applications is provided. The video encoder includes a split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal. A feature reduction module is coupled to the neural network front end, receives the feature maps and generates at least one reduced feature map representing the original plurality of feature maps. A feature conversion module receives the reduced feature maps and converting the reduced feature maps to a video format. The encoder further includes a forward non-linear transform module receiving the converted feature maps and performing a non-linear transform on the feature tensors of the converted feature maps and an inner encoder receiving the transformed feature maps and generating an encoded bitstream representing the plurality of feature maps.

[0016] In some embodiments the non-linear transform is characterized by the equation:

[0017] Alternatively, the non-linear transform is characterized by the equation:2,yi=- x arctan(r X xi), Vx;- G x

[0018] Preferably, the video encoders of the present disclosure provide an identification of the applied non-linear transform applied in the bitstream. The identification can be provided in a header, such as a picture header, a sequence header, or a block header. Additionally or alternatively, the identification can be signaled as supplemental enhancement information (SEI) in the bitstream.

[0019] In some embodiments, a video encoder for machine-based video applications of the present disclosure may further include a preprocessing step prior to forward non-linear transformation. The preprocessing step may perform entropy reduction transformation of the tensors of the feature maps. In some cases, the entropy reduction transformation further comprises the steps:determining standard deviation and mean of input tensors;normalize the values of the input tensors;apply transformation characterized by:x = where (|x| <1,0, x)orx = where (|x| <1, x11, x), when n> 0; anddenormalize the values of x.

[0020] The video encoders of the present disclosure may additionally or alternatively include a feature conversion module which performs temporal consistency processing. In some embodiments, the temporal consistency processing, includes quantizing feature tensors is characterized by the equation:where absmax(x) denotes the absolute maximum value of the feature tensor, x.

[0021] The present disclosure also includes video decoders, and video decoding methods, for machine based video applications for decoding a bitstream generated by any of encoders and encoding methods described herein. In some embodiments, a decoder includes a modified inversefeature conversion module including non-linear inverse transform processing of decoder tensor values.

[0022] In cases where an identification of a non-linear transform applied by an encoder is signaled in the bitstream, the decoder may select an inverse non-linear transform based on the signaled identification.

[0023] In some cases, the non-linear transform is characterized by the equation:

[0024] In some cases, the non-linear transform is characterized by the equation:1 7TXi=- x tan(- X yi), Vy;- e y

[0025] Embodiments of video decoders may additionally or alternatively include inverse temporal consistency processing. In some embodiments, inverse temporal consistency processing is characterized by the following equation whereby the quantized unsigned values, %<?ucmttzedareconverted back to the original feature tensor values x:absmax(x).Brief Description of the Figures

[0026] For the purpose of illustrating the inventive aspects of the current disclosure, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present disclosure and disclosed invention(s) is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:

[0027] Figure 1 is a simplified block diagram of a video coding and decoding (CODEC) system for machine video applications, such as Feature Coding for Machines (FCM);

[0028] Figure 2 is block diagram of a video CODEC system for machine video applications, such as Feature Coding for Machines (FCM) of Fig. 1 and further illustrating the components of the encoder and decoder systems;

[0029] Figure 3 is block diagram of a CODEC system for machine video applications, such as Feature Coding for Machines (FCM), with supplemental encoder processing for entropy reduction transformation and forward non-linear processing, in accordance with the present disclosure;

[0030] Figure 4 is a flow chart of an algorithm for an exemplary method for entropy reduction transformation in accordance with the present disclosure;

[0031] Figure 5 is a flow chart of an algorithm for forward non-linear transform of a feature signal for encoding, in accordance with the present disclosure;

[0032] Figure 6 is a flow chart of an algorithm for decoder side non-linear inverse transform of an encoded feature signal in accordance with the present disclosure;

[0033] Figure 7 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0034] Figure 8 is a graph illustrating the results of applying temporally consistent feature conversion in accordance with the present disclosure.Detailed Description of Disclosed Embodiments

[0035] The present disclosure describes improvements to video coding and decoding systems and methods for machine-based video applications, such as Feature Coding for Machines (FCM) and other M2M video applications.

[0036] The present systems and methods for coding video for machine-based applications described in this disclosure include three new modules, which can be used alone or in combination, and which improve upon known FCM systems, such as described in connection with Figs. 1 and 2. Embodiments of the present disclosure include, for example, encoder side pre-processing using entropy reduction transformation, encoder side forward non-linear transformation, and decoder side inverse non-linear transformation. It has been found that the use of non-linear transformation applied to feature conversion in accordance with the present disclosure provides improved bitrate accuracy when compared to linear transformation which has been used previously in FCM encoders.

[0037] At the encoder side, preprocessing and forward non-linear transformation can preferably replace and / or supplement the traditional feature conversion module of an FCM encoder as described in connection with Fig. 2, as further described below. Similarly, on the decoder side, inverse non-linear transformation preferably replaces and / or supplements the traditional inverse feature conversion module of an FCM decoder, as described in further detail below.

[0038] Figure 3 is a simplified block diagram of an FCM encoder and decoder system in accordance with the present disclosure. The encoder includes a neural network part 1 310, feature reduction 315, feature conversion 320 and inner codec 325, which operate substantially as described in like numbered components in Fig. 2. In addition, the encoder of the present invention includes a supplemental processing module 322 interposed between feature conversion 320 and the inner codec 325 which can implement the encoder side entropy reduction transformation (ERT) preprocessing and / or forward non-linear transform processing of the present disclosure, as described in further detail below. It will be appreciated that the functionality of supplementalprocessing module 322 can be integrated into the feature conversion module 320 but is described herein as a separate processing module for clarity and ease of understanding.

[0039] Referring to Figure 3, supplemental encoder side processing 322 may include a preprocessing step which includes a method of entropy reduction transformation. Algorithm- 1 presented below and further illustrated in the flow chart of Fig. 4, is preferably used as a preprocessing step at an encoder following (or part of) feature conversion to reduce the entropy for the feature tensor and to reduce the bitrate. This step is optional as a preprocessing step for the forward non-linear transform described below as Algorithm 2.

[0040] Algorithm 1 - Entropy Reduction Transformation (ERT)

[0041] An entropy reduction transformation (ERT) method in accordance with the present disclosure is illustrated in the flow chart of Figure 4 and can be described mathematically and with pseudocode as follows:Input: Tensor x (Fig. 4, step 405)Output: Transformed tensor x1. Compute the original standard deviation and mean (Fig. 4, step 410):where N is the number of elements in x, xi is the i-th element of x, porgis the mean, and aorgis the standard deviation.2. Normalize x (Fig. 4, step 415):3. Apply transformation (Fig. 4, step 420):x = where (|x| <1,0, x)orx = where (|x| <1, x11, x), when n> 0

[0042] In the first part of this transformation, after standardizing the input, if a value is less than 1, it will be set to 0, otherwise its value is maintained. In the second part of this transformation, after standardizing the input, if a value is less than 1, values of x will be exponentially dampened rather than simply defaulting to 0. For values greater than 1, the value of x is maintained.4. Denormalize (Fig. 4, step 425):5. Return the transformed x. (Fig. 4, step 430).

[0043] Algorithm 1, or entropy reduction transform (ERT), is preferably a pre-processing step for the presently disclosed forward non-linear transform on the encoder side, referred to herein as Algorithm-2. The forward non-linear transform process is illustrated in the flow chart of Figure 5 and is generally implemented in the supplemental processing stage 322 at the encoder illustrated in Figure 3. Alternatively, this functionality can be integrated into the feature conversion module 320. The use of non-linear transform can provide improved bitrate accuracy when compared to linear transformation methods previously used in FCM encoders, e.g., min-max normalization or z-score normalization for machine-based video encoding, such as feature coding for machines. Traditional min-max normalization or z-normalization applies a linear transformation, even though not all feature values are equally important for the downstream task. In contrast, a nonlinear transformation allows for entropy reduction in the feature activation values by carefully shaping the transformation function to minimize redundancy without compromising downstream task performance.

[0044] The present forward non-linear transform algorithm of Fig. 5 can be further described mathematically and with pseudocode as follows:Input (Step 505): Tensor x, control parameter r, bit depth nOutput: Transformed tensor to range [0, 2n-l]1. (Optional) Step 1 (Fig. 5, step 510): Preprocessing to reduce entropy. This can be performed, for example, by applying Algorithm 1, ERT (Fig. 4) to the input tensor x 2. Step 2 (Fig. 5, step 515): Scale x to range [-1, 1], preferably using the function:3. Step 3 (Fig. 5, step 520): Apply non-linear transformation using one of the following exemplary options:Option2Option 2: yi=- x arctan(r X xi), Vx;- G x4. Step 4 (Fig. 5, step 525): Scale to [0,1] rangey+1y=—5. Step 5 (Fig. 5, step 530): Convert to n-bit unsigned integersmax_val = 2n-1zi = round(yi x max_val), Vy G y6. Step 6 (Fig. 5, step 535): Return the output z G [0, 2n— 1]

[0045] In step 3, the selection of Option 1 and Option 2 may be based on the desired properties of these options. Option 1 provides stronger compression of large values while keeping values near zero almost linear. Option 2, on the other hand, provides a bounded, smooth mapping using the arctangent function and ensures robustness to outliers. It effectively saturates extreme values and normalizes data to a fixed range (typically [-1,1]). The main drawback with Option 2, however, is that information in the saturated regions is lost, as the large inputs become indistinguishable once mapped near the bounds.Inverse Non-linear Transformation

[0046] A third process used in accordance with the present disclosure is a non-linear inverse transform on the decoder side to reconstruct the original feature tensor values processed by the forward non-linear transform of Algorithm 2. This inverse non-linear transformation method is referred to herein as Algorithm 3 and is further detailed below and illustrated in the flow chart of Fig. 6. It should be noted that when forward transform is applied at the encoder, it is preferable for the ID of the forward transform to be signaled in the encoded bitstream to facilitate the decoder applying the correct corresponding inverse transformation.

[0047] Algorithm 3 can be expressed mathematically with the following pseudocode as follows. Figure 6 illustrates the non-linear inverse transform of Algorithm 3 and is referenced below:Input (Fig. 6, step 605): Tensor z ∈ [0, 2n-1], control parameter r, bit depth n Output: reconstructed Tensor x∈ [0, 1]1. Step 1 (Fig. 6, step 610): Convert / -bit unsigned integers to [0,1]max_val = 2n-12. Step 2 (Fig. 6, step 615): Scale to range [-1, 1]x = 2×y -13. Step 3 (Fig. 6, step 620): Apply inverse transformation using one of the following options:Option 1:1 7TOption 2: xi=1 / r× tan(π / 2× yi), ∀yi∈ y4. Step 4 (Fig. 6, step 625): Convert to [0,1] range5. Step 5 (Fig. 6, step 630): return the output x G [0, 1]

[0048] In implementing the present ERT and forward non-linear transform algorithms at the encoder side, it is preferable to signal the existence and implementation of these algorithms to the decoder so an appropriate inverse transform can be performed by the decoder. An exemplary syntax for a bitstream suitable for such signaling is provided below.Exemplary Bitstream SyntaxSymbols:• ert flag = denotes if Algorithm-1 is applied to reduce the entropy of the feature map as a pre-processing step. (Boolean value of 1 -bit, True or False)• control_factor = denotes the shape of the transformation function as defined in Algorithm- 2 & Algorithm-3 as symbol r. (3 -bit unsigned integer, range 1 to 9)• transformation_function_id = denotes which transformation is applied on the encoder side so that the corresponding inverse transformation could be applied on the decoder side to reconstruct the original values. (3-bit unsigned integer, range 1 to 9 where 0 is reserved to signal that linear transformation has been applied on the encoder side)

[0049] As an alternative approach, the inverse transformation function could be sent to the decoder in Sequence Parameter Set (SPS) or using VVC Supplemental Enhancement Information (VSEI).

[0050] In accordance with the present disclosure, a Temporally Consistent Feature Conversion process can be implemented on the encoder side and the corresponding Temporally Consistent Inverse Feature Conversion process implemented at the decoder side to improve the machine vision task. These two improved modules will preferably replace and / or supplement the existing Feature Conversion (Fig. 3, 320) (described above) and the corresponding decoder module Inverse Feature Conversion 345 (described above).Encoder-side: Temporally Consistent Feature Conversion

[0051] The existing feature conversion procedures generally use the following min-max normalization method to convert the floating-point values to unsigned integer.(x — min) x (2bltdepth— 1)(max — min)However, this normalization procedure does not take account of temporal consistency, which may introduce extra bitrate overhead to represent the feature tensor values. To solve this issue, the presently improve feature conversion module 320 may implement the following equation to increase the temporal consistency across frames without sacrificing the system performance.X+ 1quantized= roundabsmax(x)

[0052] In this above equation, absmax(x) denotes the absolute maximum value of the feature tensor, x. This absmax(x) will be signaled to decoder in floating-point to recover the original feature tensor values. Some of the exemplary methods of how absmax(x) can be signaled include:1. Signal absmax(x) per frame.2. Signal average of absmax(x) over an Intra-Period (or GOP).3. Signal absmax(x) per Intra-Period of the video codec.4. Signal absmax(x) for every I-frame and for P / B frames only signal the absmax(x), if the difference between absolute maximum value of two consecutive frame exceeds a predefined threshold.Decoder-side: Temporally Consistent Inverse Feature Conversion

[0053] When the present temporally consistent feature conversion is applied on the encoder side, an inverse method is applied at the decoder. This method of temporally consistent inverse feature conversion can be implemented as part of inverse feature conversion 345. On the decoder-side, using the following equation, the quantized unsigned values, xquantizedare converted back to the original feature tensor values x.absmax(x)

[0054] Preferably, the parameter absmax(x) is extracted from the bitstream. Different examples of bitstream syntax for signaling absmax(x) is shown below.Exemplary Bitstream SyntaxSymbols:• abs_max = absolute maximum value of the feature tensor, (floating point 32-bit)

[0055] It should be noted that abs_max could be sent for every frame or with the I-frame (intra frame) only. It’s also possible to send the average of abs_max or only the difference in abs_max if there is a significant change of abs_max value in two or more consecutive frames to reduce the bitrate overhead.Result &

[0056] The presently disclosed temporally consistent feature conversion method achieves improved temporal consistency across the frame by mapping the mean value in floating-point to the unsigned number,For instance, if the mean of the feature tensor is 0 and the required bit-depth is 10, then the mean value for the feature tensor will be mapped to 511 and thus will increase the temporal coherence. This approach reduces the bitrate overhead without sacrificing the machine task accuracy.

[0057] An example analysis is shown in Figure 8, which is a graph in which the x-axis represents the frame number, and the y-axis represents the mean value of the frame in unsigned integers (bitdepth=10). The bottom curve in the graph, illustrated in a dashed line, illustrates the mean value of the frame after applying the traditional method, and the upper curve, depicted as a solid line, shows the mean value of the frame after applying the proposed methods described herein.Results

[0058] The following table shows the bitrate reduction compared to the FCTMv4 and our proposed method. In this case, the present methods improve the existing normalization method by almost 7%. Thus, in this example, the present systems and methods achieved the same level vision task accuracy with 7% less bits.

[0059] Some embodiments of the present disclosure may include / and or be embodied by non-transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein. Such non-transitory computer program products (i.e., physically embodied computer program products) may store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations of the FCM decoder and / or FCM encoder may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0060] Any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more server devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art. Appropriate software coding can readilybe prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0061] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to, a magnetic disk, an optical disc (e.g., CD, CD-R, DVD, DVD-R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine-readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0062] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0063] Examples of a computing device include, but are not limited to, an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk.

[0064] FIG. 7 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 700 within which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure.Computer system 700 includes a processor 704 and a memory 708 that communicate with each other, and with other components, via a bus 712. Bus 712 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0065] Processor 704 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 704 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 704 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), general purpose GPU, Tensor Processing Unit (TPU), analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC).

[0066] Memory 708 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, and any combinations thereof. In one example, a basic input / output system 716 (BIOS), including basic routines that help to transfer information between elements within computer system 700, such as during start-up, may be stored in memory 708. Memory 708 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 720 embodying any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 708 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof.

[0067] Computer system 700 may also include a storage device 724. Examples of a storage device (e.g., storage device 724) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 724 may be connected to bus 712 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 724 (or one or more components thereof) may be removably interfaced with computer system 700 (e.g., via an external port connector (not shown)). Particularly, storage device 724 and an associated machine-readable medium 728 may provide nonvolatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 700. In one example, software 720 mayreside, completely or partially, within machine-readable medium 728. In another example, software 720 may reside, completely or partially, within processor 704.

[0068] Computer system 700 may also include an input device 732. In one example, a user of computer system 700 may enter commands and / or other information into computer system 700 via input device 732. Examples of an input device 732 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 732 may be interfaced to bus 712 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 712, and any combinations thereof. Input device 732 may include a touch screen interface that may be a part of or separate from display 736, discussed further below. Input device 732 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0069] A user may also input commands and / or other information to computer system 700 via storage device 724 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 740. A network interface device, such as network interface device 740, may be utilized for connecting computer system 700 to one or more of a variety of networks, such as network 744, and one or more remote devices 748 connected thereto. Examples of a network interface device include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 744, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g., data, software 720, etc.) may be communicated to and / or from computer system 700 via network interface device 740.

[0070] Computer system 700 may further include a video display adapter 752 for communicating a displayable image to a display device, such as display device 736. Examples of a display device include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter 752 and display device 736 may be utilized in combination with processor 704 to provide graphicalrepresentations of aspects of the present disclosure. In addition to a display device, computer system 700 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 712 via a peripheral interface 756. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0071] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0072] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

What is claimed is:

1. A video encoder for machine-based video applications comprising:a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal;a feature reduction module coupled to the neural network front end and receiving the at least one feature map generating at least one reduced feature map representing the original plurality of feature maps;a feature conversion module receiving the reduced feature maps and converting the reduced feature maps to a video format;a forward non-linear transform module receiving the converted feature maps and performing a non-linear transform on the feature tensors of the converted feature maps; andan inner encoder receiving the feature maps from the forward non-linear transform module and generating an encoded bitstream representing the plurality of feature maps.

2. The video encoder for machine-based video applications of claim 1, wherein the nonlinear transform is characterized by the equation:

3. The video encoder for machine-based video applications of claim 1, wherein the nonlinear transform is characterized by the equation:2,yi=- x arctan(r X xi), Vx;- G x4. The video encoder of any of claims 1-3, wherein an identification of the non-linear transform applied by the encoder is signaled in the bitstream5. The video encoder for machine-based video applications of claim 4, further comprising a preprocessing step prior to forward non-linear transformation.

6. The video encoder for machine-based video applications of claim 5, wherein the preprocessing step performs entropy reduction transformation of the tensors of the feature maps.

7. The video encoder for machine-based video applications of claim 6, wherein entropy reduction transformation further comprises:determining standard deviation and mean of input tensors;normalize the values of the input tensors;apply transformation characterized by:x = where (|x| <1,0, x)orx = where (|x| <1, x11, x), when n> 0; anddenormalize the values of x.

8. The video encoder of any of claims 1-7, wherein the feature conversion module further comprises temporal consistency processing.

9. The video encoder of any of claim 8, wherein the temporal consistency processing, includes quantizing feature tensors is characterized by the equation:where absmax(x) denotes the absolute maximum value of the feature tensor, x.

10. A video decoder for machine-based video applications for decoding a bitstream generated by an encoder of any of claims 1-9 and comprising a modified inverse feature conversion module including non-linear inverse transform processing of decoder tensor values.

11. The video decoder of claim 10, wherein an identification of a non-linear transform is signaled in the bitstream and the decoder selects an inverse non-linear transform based on the signaled identification.

12. The video decoder of claim 10, wherein the non-linear transform is characterized by the equation:

13. The video decoder of claim 10, wherein the non-linear transform is characterized by the equation:1 71Xi=- x tan(- X yi), Vyfe y14. The video decoder of any of claims 10-13, further comprising inverse temporal consistency processing.

15. The video decoder of claim 14, wherein inverse temporal consistency processing is characterized by the following equation whereby the quantized unsigned values,xquantlzedare converted back to the original feature tensor values x:absmax(x).

16. A video encoder for machine-based video applications comprising:a split neural network front end, the split neural network front end receiving an input image signal and generating a plurality of feature maps comprising a plurality of feature tensors representing the input image signal;a feature reduction module coupled to the neural network front end and receiving the at least one feature map generating at least one reduced feature map representing the original plurality of feature maps;a feature conversion module with temporal consistency processing receiving the reduced feature maps and converting the reduced feature maps to a video format, wherein temporal consistency processing includes quantizing feature tensors with the processwhere absmax(x) denotes the absolute maximum value of the feature tensor, x and an inner encoder receiving the feature maps from the feature conversion module and generating an encoded bitstream representing the plurality of feature maps.

17. A video decoder for decoding a bitstream generated by the encoder of claim 12 and comprising a modified inverse feature conversion module including inverse temporal consistency processing using the following equation whereby the quantized unsigned values, xquantizedare converted back to the original feature tensor values xabsmax(x).

Images