Spherical vector quantization for neural audio coding

Abstract

The invention relates to a method for coding an audio signal, comprising: quantifying a latent representation from an analysis of the audio signal by a neural network, on at least one quantization stage, each quantization stage comprising: -- selecting a codeword, by comparing a vector from the latent representation and codewords belonging to a predefined spherical dictionary, -- scaling the selected codeword by an optimized gain factor to form a quantized vector; -- generating an index representing the quantized vector; multiplexing the generated indices to form at least part of a bitstream representing the audio signal.

Description

Spherical vector quantization for neural audio coding

[0001] This document relates to the general field of audio signal encoding and decoding. In particular, it relates to the optimized quantization of a latent space (or signal transformed by a neural network) for the encoding and decoding of an audio signal.

[0002] In a lossy speech or audio signal coding (or compression) system, the general objective is to find a representation of the signal that is as suitable as possible for coding at the lowest possible bit rate at a given quality, or conversely with the best possible quality at a given bit rate, taking into account constraints such as complexity, storage quantity, or algorithmic delay.

[0003] Historically, this field has developed according to two approaches.

[0004] One approach to speech signal coding is dominated by a temporal approach based on linear prediction (short-term and long-term) to model the speech signal. The unmodeled portion (innovation or prediction residual) is coded in addition to the model parameters. Auditory perception is generally taken into account through the use of perceptual weighting. The CELP model (for Code Excited Linear Prediction) remains the reference model for good-quality, low-bitrate, "conventional" speech coding.

[0005] A second approach concerns the coding of audio signals, which relies primarily on a frequency-domain approach using sub-bands or transforms to achieve coding gain and take advantage of psychoacoustic characteristics and the limitations of auditory perception. MDCT (Modified Discrete Cosine Transform) coding, with critical decimation, remains the reference technique for conventional low-bitrate audio coding.

[0006] Speech coding and audio coding have subsequently converged with "universal" coding models, particularly through time-combining (CELP) and frequency-combining (MDCT) approaches.

[0007] The state of the art in "conventional" speech and audio coding is very extensive, and numerous standardized or proprietary codecs have been developed, incorporating various technical advancements and addressing diverse applications and constraints. The most recent standards, such as MPEG-D USAC, MPEG-H 3D Audio, 3GPP EVS, and IETF Opus, provide an overview of the latest advancements and currently represent the state of the art in "conventional" coding.

[0008] Artificial intelligence-based approaches, particularly those using neural networks such as WaveNet and VQ-VAE (Vector Quantized-Variational Auto-encoder), have spurred the development of neural network-based speech and audio coding methods. Deep learning has shown great potential in other areas related to audio coding, such as text-to-speech synthesis, image and video processing, and audio signal analysis and processing in general.

[0009] A particular feature of neural methods applied to speech and audio is that they often work directly in the time domain (on the waveform).

[0010] However, there are also models operating in the frequency domain.

[0011] Two approaches to speech and neural audio coding stand out.

[0012] One approach involves encoding features to feed into a decoding process using a generative model. In this case, representations, such as the spectrogram according to the Mel scale, are estimated and encoded, and a neural network decoder (e.g., WaveNet or WaveRNN) reconstructs the waveform from this encoded representation.

[0013] A second approach involves end-to-end coding, inspired by the VQ-VAE approach, where the codec is decomposed into an analysis part, a quantizer, and a synthesis part, all of which are learned end-to-end. The principle is to "embed" the input distribution (of high dimensionality) into a domain called "latent space" (generally of lower dimensionality), which is then discretized.

[0014] Neural audio coding is currently at a sufficient level of maturity for neural codecs to be deployed in conversational applications. It is also used in other applications, such as tokenization, which involves replacing an audio signal with codes or tokens (generally the binary representation in latent space) that are then used for advanced processing such as model-based audio synthesis, etc.

[0015] The codecs called "SoundStream," "EnCodec," and "DAC" are representative methods of neural audio coding. These codecs are described respectively in the following articles: - N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec," IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2021; - Défossez et al., "High fidelity neural audio compression," arXiv preprint arXiv:2210.13438, 2022; - R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, "High-fidelity audio compression with improved RVQGAN," Proc. NIPS, Article 1214, 2023.

[0016] Their low bitrate performance (in the order of 3 to 8 kbit / s) rivals "conventional" speech and audio coding approaches like EVS or Opus which operate at much higher bitrates (in the order of 12 to 24 kbit / s).

[0017] These codecs, SoundStream, Encodec and DAC, are very similar. We will subsequently focus more specifically on the "DAC" method, as it is more recent and more efficient, in the case of a mono (i.e., one-dimensional) signal and for the case of signals sampled at 44.1 kHz.

[0018] Implementing a codec using neural networks involves two distinct phases: the training phase and the inference phase. During the training phase, the model's parameters are trained using training data. The inference phase refers to the deployment and use of the previously trained model.

[0019] Laillustrates the different steps implemented in a DAC type codec.

[0020] During the inference phase, i.e. when using the already "trained" codec, the encoding (block 100) of a signal frame of the input signal x is broken down into two parts:

[0021] - an analysis section (block 101) by a neural network which converts the input signal x into a latent representation z also called "latent space". This latent space represents in compact form the important characteristics of the input signal.

[0022] - a quantization / coding part (block 102) of the latent representation z generating a binary train I made up of multiplexed quantization indices.

[0023] Similarly, in the inference phase, the decoding (block 110) that reconstructs the audio signal is broken down into two parts: - inverse quantization / decoding of the latent space (block 111) from the binary stream I; - a synthesis section using a synthesis neural network (block 112). The architecture of this section is symmetrical (in reverse order) to the analysis section (101), with transposed convolutions instead of convolutions.

[0024] The synthesis generates an approximate value of the signal x, denoted .

[0025] During the training phase, the quantization blocks (102, 111) are optimized using learned and stored dictionaries. The selected codeword is updated with a moving average (with a forgetting factor) or by direct dictionary optimization to minimize distortion ("codebook loss"), often with a cost function linking the analysis and quantization parts ("commitment loss"); unused dictionary entries can be replaced with candidates from the current batch of the training set. Quantization is bypassed in the backpropagation phase for the analysis / synthesis networks. A variable number of stages (called "residual stages") are selected during training to train a single model representing multiple rates.Furthermore, during training, the codec also includes additional metric calculation blocks (block 121) and a complementary neural network called a "discriminator" (block 120), based on the principle of Generative Adversarial Networks (GANs), to obtain different cost functions for training. Details regarding the training process are not discussed here, but the detailed description below will explain the training of the neural network parameters and the latent space quantifiers.

[0026] In the DAC example, the architecture of the analysis and synthesis sections is described below with reference to Figures 8a and 8b. The signal frame x consists of 512 samples at a frequency of 44.1 kHz and the latent space has a dimension of 1024.

[0027] The analysis is performed by the encoder (block 101) and the synthesis by the decoder (block 112). The encoder consists of a series of convolutional layers (blocks 901 and 907), encoding blocks ("EncoderBlock") (blocks 902 to 905), and a SNAKE activation function (block 906). The decoder consists of a series of convolutional layers (blocks 921 and 927), decoding blocks ("DecoderBlock") (blocks 922 to 925), a SNAKE activation function (block 926), and finally a nonlinear hyperbolic tangent function (block 928). The encoding block "EncoderBlock" (block 940) consists of residual units "ResidualUnit" (blocks 941 to 943), a SNAKE activation function (block 944), and a convolution layer (block 945). The decoding block "DecoderBlock" (block 950) consists of residual units "ResidualUnit" (blocks 953 to 955), a SNAKE activation function (block 951), and a transposed convolution layer (block 952).The residual unit (block 930) is composed of convolutional layers (blocks 932 and 934), SNAKE activation functions (blocks 931 and 934), and the input of the residual unit is summed to the output of the residual unit (block 935). The tensor dimensions are given in square brackets with the notation [B, C, T] where B corresponds to the batch size (equal to 1 by default), C corresponds to the number of channels, and T is the length of the time frame.

[0028] Note that a complete DAC implementation (blocks 100 and 110) is provided in: https: / / github.com / descriptinc / descript-audio-codec

[0029] The (102, 111) quantization used in SoundStream and EnCodec is a multi-stage vector quantization called "Residual VQ" or RVQ (for "Residual Vector Quantization"). In DAC, a more advanced version is used; the different quantization stages also include linear preprocessing (projection onto an 8-dimensional subspace) and linear postprocessing (expansion to the latent space dimension).

[0030] Before going any further, let's review some basics on quantification.

[0031] The book by Gersho A. and Gray RM, "Vector Quantization and Signal Compression," Kluwer Academic Publishers, 1992, offers a comprehensive review of scalar and vector quantization methods. Vector quantization (VQ) is a widely used solution in digital signal compression. It represents an input vector by a vector of the same dimension chosen from a finite set called a "dictionary." A quantizer with M "codewords" is a non-bijective mapping from the set of input vectors, usually the n-dimensional Euclidean real space (or a subspace), to a finite subset C of this space with M distinct elements: C = {c0, c1, ..., c1}. M-1The elements of the dictionary C are codewords that serve as output points (reconstruction). The quantizer's bit rate R (or its resolution) is defined by log2(M) bits per vector. In vector quantization, a block of n samples is treated as a vector of dimension n. According to source coding theory, when the dimension becomes very large, the performance of vector quantization (VQ) approaches the source's bit rate-distortion bound. Dictionaries of vector quantizers can be designed using statistical methods such as the Generalized Lloyd Algorithm (GLA). This algorithm is based on the necessary optimality conditions. The statistical vector quantizers thus obtained have no structure, which makes their exploration computationally expensive and memory-intensive. Indeed, the complexity of both coding and storage is proportional to n². R .

[0032] To implement a vector quantizer, there are three operations: two for encoding and one for decoding. An input vector is encoded by first choosing a codeword from a dictionary. Generally, the most "similar" codeword is chosen, and then the index of this codevector is determined to be transmitted or stored. At the decoder, the codeword is determined from its index.

[0033] The exponential growth of complexity, depending on the vector dimensions and data rate, limits the use of unstructured vector quantizers to low dimensions and / or low data rates to enable real-time implementation. For an unstructured vector quantizer, nearest neighbor search requires an exhaustive search of all dictionary elements to select the dictionary element that minimizes a distance measurement between itself and the input vector.

[0034] Indexing and decoding are generally performed by simple table reads but are memory-intensive. To overcome size and dimensionality constraints, several variants of vector quantization were studied. These attempt to address the lack of dictionary structure and thus manage to reduce complexity, generally at the expense of quality. However, the performance-complexity trade-off is improved, making it possible to increase the range of resolutions and / or dimensions over which quantization can be applied cost-effectively. Many structured vector quantization methods are reviewed in the aforementioned work by Gersho and Gray. In particular, the RVQ-type quantization of SoundSteam and EnCodec corresponds to a known multi-stage quantization, the difference being that this method is integrated into the learning of a neural codec.

[0035] The definition of two distortion (or "similarity") criteria used in vector quantization is recalled below: - Quadratic error between x and y of dimension n, which corresponds to the square of the L2 norm or Euclidean distance in dimension n: d(x,y) = || x – y || 2 = (x1-y1) 2 + … + (x n -y n ) 2 Cosine similarity measure between x and y: d(x,y) = x T .y / (|| x || . || y ||)where x T .y is the dot product between x and y. This measure actually corresponds to the cosine of the angle between the vectors x and y. It therefore measures the similarity of the two vectors based on their orientation, ignoring their respective magnitudes.

[0036] Note that if the two vectors x and y are normalized (to a norm of 1), the quadratic error and the cosine similarity are equivalent.

[0037] Other distances exist, such as the L1 standard (or Manhattan distance) or the Linf standard (or Chebyshev distance). These are not used in the rest of the description.

[0038] Laillustre the operation of multi-stage quantization of RVQ type (block 102 without entropic coding) and lad describes the associated decoding (block 111 without entropic decoding).

[0039] Depending on the number of bits R per frame, this process operates with a series of K quantization blocks (denoted 2-B1, 2-B2, … 2-BK, where 2-Bi is the i-th block) in cascade, where the latent space signal z is encoded by a first stage (block 2-B1-1), then the quantization error of the first stage e1 is calculated (block 2-B1-2, …) to obtain the input of the next stage (block k=2), and so on. The indices i k from each stage are multiplexed to form the binary train I of the current frame (block 200).

[0040] In the EnCodec described in the aforementioned article, the bit rate of a stage is 10 bits for an input signal (latent space) of dimension n=128. For a mono input at 24 kHz, EnCodec defines stages of M=1024 inputs (10 bits per frame of 320 samples at 24 kHz, or 0.75 kbit / s), and can combine up to 32 of these (for a bit rate of 24 kbit / s). RVQ quantization involves storing K=32 dictionaries (blocks 2-C1 to 2-CK=2-C32), with one dictionary per stage. Each dictionary C k is of dimension n=128 and size M=1024. Quantization in the k-th stage is performed by simply finding the nearest neighbor between e k and all the code words from the C dictionary k , by minimizing the quadratic error (the Euclidean distance).

[0041] By default, the number of successive stages combined is predefined: K=2, 4, 8, 16 or 32 stages, for a rate of 1.5, 3, 6, 12 or 24 kbit / s (or R = 20, 40, 80, 160, 320 bits per frame).

[0042] During decoding, as shown in Figure 3, the number of stages K is determined based on the data rate R. The binary stream I is demultiplexed, and its indices are separated (block 300). Each stage (blocks 3-B1, 3-B2, … 3-BK) provides the stored codeword, which is then decoded by reading from the dictionary C. k associated (blocks 2-C1 to 2-CK), and all codewords The decoded elements are successively added (blocks 352, …, 35K) to obtain the decoded signal. of latent space in the current framework.

[0043] Laet lamontent the equivalent operation for the DAC projection quantification approach, referred to here as "RPVQ" (for Residual Projected VQ).

[0044] In the DAC described in the aforementioned article, the bit rate of a stage is 10 bits for an input signal projected in dimension n=8. For a mono input at 44.1 kHz, DAC defines stages of M=1024 inputs (10 bits per frame of 512 samples at 44.1 kHz, or approximately 0.86 kbit / s), and can combine up to 9 of these (for a maximum bit rate of 7.75 kbit / s). RVQ quantization with projection / expansion, hereafter denoted RPVQ, involves storing K=9 dictionaries (blocks 4-C1 to 4-CK=4-C9), with one dictionary per stage. Each dictionary C k is of dimension n=8 and size M=1024, but it is also necessary to store the projection and expansion matrices of respective sizes 1024x8+8 and 8x1024+1024 (+8 and +1024 due to a bias defined in these operations; the dimension of the bias corresponds to the output dimension of the linear layer). Quantization in the k-th stage (block 4-Bk-2) is performed by nearest neighbor search between e kand all the code words from the C dictionary k , by minimizing a "cosine similarity" metric that corresponds to the normalized dot product.

[0045] The indices i k from each stage are multiplexed to form the binary train I of the current frame (block 400).

[0046] In the decoding stage, shown in Figure 5, the number of stages K is determined based on the data rate R. The binary stream I is demultiplexed and its indices are separated (block 500), and each stage (blocks 5-B1, … 5-BK) provides the stored and decoded codeword (block 5-Bk) by reading from the dictionary C. k associated (block 4-Ck) before applying the expansion operation (block 4-Bk-3), and all the vectors thus decoded are successively added (blocks 5-Bk-2) to obtain the decoded signal of latent space in the current framework.

[0047] The RVQ latent space quantization method and its more advanced variant, RPVQ, offer several advantages: the implicit structures of the latent space are learned implicitly through vector quantization, and the multi-stage approach allows for hierarchical (scalable) coding with enhancement stages whose number depends on the target throughput. The RPVQ approach can be interpreted as a kind of successive principal component decomposition and allows for a reduction in the dimensionality of the quantization dictionaries.

[0048] These quantization methods, however, present several problems. It is well known to those skilled in the art that vector quantization using stochastic dictionaries requires significant storage; each RVQ stage (of 10 bits) requires n x M stored values, where M = 1024 is the size of the dictionary (in number of code words) and n is the dimension of the latent space. In the case of DACs, dictionary storage is reduced to 1024 x 8 stored values ​​per stage, to which must be added the projection and expansion matrices (1024 x 8 + 8 and 8 x 1024 + 1024 per stage), which remains significant for the entire set of K = 9 stages. These quantization parameters, represented in floating-point (32 bits), require, for example, approximately 0.88 MB of storage.

[0049] Furthermore, the complexity of the nearest neighbor search (which must be performed exhaustively by minimizing the quadratic error for SoundStream and EnCodec, and the cosine similarity for DAC between the input e j and all the code words c k (i)) is significant for the 10-bit throughput, and it increases linearly with the number of stages and the throughput.

[0050] Finally, while stochastic dictionaries can represent the distribution of a source observed during training, they can lead to source mismatch problems when the coded test data deviates from this distribution. More "universal" approaches may be of interest to improve generalization capabilities in the encoding of the latent space.

[0051] Therefore, there is a need to optimize latent space coding techniques for the field of neuronal audio coding / decoding. Exposed

[0052] This document improves upon the state of the art.

[0053] We are particularly interested in the application of algebraic methods of spherical vector quantization.

[0054] This leads us to provide a brief overview of this structured vector quantization method. Two forms of quantization based on algebraic structures are studied here in particular: - Quantization using permutation codes - Quantization using point networks (regular or irregular).

[0055] In vector quantization using "permutation codes," introduced in the article "Permutation codes for sources" by Berger T., Jelinek F., and Wolf JK, IEEE, Trans. On Information Theory, Vol. IT 18, No. 1, pp. 160-169, January 1972, codewords are obtained by permuting the components of the first (in the lexicographical order sense) codeword, called the leader (or direction vector). These components take their values ​​from an alphabet A = {a0, a1, …, aq-1} of size q (an alphabet A with q values ​​such that a0 > a1 > …> aq-1). The components ai are real numbers or integers. The weight wi (where i is an index) is the number of repetitions of the letter ai in the alphabet. The n components of the leader decrease from position 0 to position (n-1).

[0056] The leader vector generally has no sign constraints, and the permutation code is said to be type I. The other code vectors are obtained by permutations. In this case, the leader vector can be called a signed leader (in the sense that the letters ai of the alphabet A can be positive or negative).

[0057] There is another type of permutation code (Type II). The leader vector also has the same form as before, but its components must be positive. The other code vectors are also obtained by permuting the components, assigning them all possible sign combinations. In this case, the leader vector can also be called the absolute leader (although, as we will see later, this term is slightly more restricted in the case of codes derived from dot networks).

[0058] The vector quantifier "permutation code" has been extended to the composition (or union) of permutation codes in Lu L., Cohen G., Godlewski Ph., "Composite Permutation codes for Vector Quantization", IEEE International Symposium on Information Theory, pp 237, 27 Jun-1 Jul 1994.

[0059] Note that the dictionary associated with a permutation code is a spherical dictionary if all leaders have the same norm.

[0060] A regular network of points in dimension n is a linear structure whose points are all obtained by linear combination of basis vectors in the form: y = k . G where G is a generating matrix and k is an integer vector.

[0061] Lattices of points are algebraic structures related to numerous mathematical problems, such as close packings of spheres, groups, and error-correcting codes. They have many applications in digital communications (vector quantization, coded modulation, cryptography, etc.). For a more in-depth presentation, see the reference work by J.H. Conway and N.J.A. Sloane, "Sphere Packings, Lattices and Groups," 3rd Edition, Springer, 1998.

[0062] A vector quantizer dictionary can be defined from a point lattice by selecting a finite subset. This operation is generally called "shaping" or "truncation." The region defining the truncation is often centered on the origin and can be spherical, pyramidal, etc., but a dictionary can also be defined on the surface of such a region. A point lattice dictionary is generally suitable for representing a uniform source, and in this sense, point lattice quantization is a vector generalization of uniform scalar quantization.

[0063] The advantage of point network quantization is that it can exploit the existence of fast nearest neighbor search algorithms that exploit the algebraic structure in regular point networks, before virtually any dictionary storage.

[0064] In particular, spherical vector quantization by point networks was studied in the article by Lamblin C., Adoul JP - "Algorithm of Spherical Algebraic Vector Quantization by the E8 Gosset Network", Annales Des Télécommunications, n° 3-4, 1988., in the case of the Gosset network in dimension 8.

[0065] Recall that the Gosset lattice can be defined in the form: where D8 is the point network such that:

[0066] The Gosset lattice can also be defined according to its generating matrix: y = [k1k2… k8]G RE8 Or

[0067] The line vectors v1, v2, … , v8 are the basis vectors of the network.

[0068] One point In checks the following properties: - All elements are integers - The sum of all the elements is a multiple of 4 - All elements have the same parity (they are either all even or all odd) - The sum of is a multiple of 8.

[0069] This last property can be interpreted geometrically by decomposing the Gosset lattice into discrete spheres (or orbits) of index m centered at the origin and of radius , with m integer. The points of an RE8 sphere can be used to define a spherical dictionary in dimension 8.

[0070] A special property of the Gosset lattice is that any permutation of a point in RE8 results in a point that remains in RE8 (on the same sphere). Therefore, each sphere can be decomposed as a union of permutation codes, where a specific vector called the "leader" (or direction vector) allows us to generate a subset of equivalent points on the same sphere through permutation.

[0071] A “leader” is an integer vector whose elements are sorted in descending order; this vector can be a “signed leader” or an “absolute leader”. Codewords associated with a signed leader are generated solely by permutation, while codewords associated with an absolute leader are generated by permutation and sign reversal, with the additional constraint in RE8 that the parity of the total number of negative elements is identical to the predefined parity of the absolute leader, denoted ISIG in the aforementioned article by Lamblin and Adoul.

[0072] As shown in the aforementioned article by Lamblin and Adoul, it is possible to enumerate all the leaders belonging to each of the successive spheres of the RE8 network. The permutations of the components of a given leader are all points of the network on the same sphere.

[0073] It is possible to design a spherical vector quantization dictionary from the RE8 network, by selecting a complete sphere or by selecting leaders (which must be scaled so that they are all of the same norm).

[0074] The article by Lamblin and Adoul details nearest neighbor search, indexing, and decoding algorithms based on such dictionaries defined by a predefined set of leaders. The dictionary is then a union of permutation codes (of leaders brought to the same norm if they are not on the same sphere). The permutations are unconstrained for a signed leader, but they are constrained to satisfy the sign parity constraint of the absolute leader if they are signed permutations from an absolute leader.

[0075] The index of a codeword is typically decomposed into two parts: the rank of the permutation of a leader, and the offset (or shift) of the cardinality of the leader.

[0076] It is also possible to define quasi-spherical truncation quantization dictionaries, by retaining all the spheres of the lattice up to a certain radius and then adding pre-selected leaders as explained in the article S. Ragot, B. Bessette, and R. Lefebvre, "Low-Complexity Multi-Rate Lattice Vector Quantization with Application to Wideband TCX Speech Coding at 32 kbit / s," Proc. ICASSP, 2004. Several vector quantization dictionaries are defined, each being a union of permutation codes (of leaders on different spheres) with RE8 lattice-specific sign parity constraints.

[0077] The invention relates to a method for encoding an audio signal comprising: - performing a quantization of a latent representation resulting from an analysis of the audio signal by a neural network, on at least one quantization stage, with for each quantization stage: - the selection of a codeword, by comparison between a vector from the latent representation and codewords belonging to a predefined spherical dictionary, - the scaling of the selected codeword by an optimized gain factor in order to form a quantized vector; - the generation of an index representing the quantized vector; - multiplexing the generated indices to form at least part of a binary train representing the audio signal.

[0078] The described method aims to efficiently compress an audio signal by exploiting an intermediate representation called a "latent representation," generated through prior analysis of the audio signal by a neural network. This latent representation captures the essential properties of the audio signal, thus enabling its efficient compression.

[0079] The quantization of this latent representation proposed by the invention is performed on at least one quantization stage. At each relevant stage, a vector from the latent representation is subjected to spherical vector quantization. This quantization consists of choosing, from among the vectors ("codewords") of a pre-established dictionary where all codewords lie on the same sphere (i.e., all have the same norm), the codeword that most closely resembles the vector from the latent representation. For this purpose, a similarity measure, such as cosine similarity, is generally used to determine the codeword closest to the vector from the latent representation.

[0080] Once the codeword is selected, it is scaled by multiplication with an optimized gain factor to precisely match the initial vector. This gain can be an "instantaneous" gain calculated to minimize the error between the vector from the initial latent representation and the quantized vector, or a fixed "optimized" gain.

[0081] Each quantized vector obtained is represented by a compact numerical index corresponding to its position in the spherical dictionary. If "instantaneous" gain is used, indices encoding this gain can also be determined in addition to the position indices in the dictionary. These indices are then multiplexed to form at least part of a binary stream representing the compressed audio signal.

[0082] In one embodiment, the process first applies a projection of the vector extracted from the initial latent representation onto a reduced-dimensional subspace. This operation significantly reduces the dimension of the latent vector, thereby simplifying subsequent processing, decreasing algorithmic complexity, and reducing memory requirements during encoding and decoding.

[0083] The major advantage of this method lies in the judicious combination of projection and spherical vector quantization. This combination significantly reduces algorithmic complexity and memory requirements while ensuring excellent audio reproduction quality after decompression. The method is particularly well-suited to applications requiring low transmission rates while preserving high sonic fidelity.

[0084] Each stage can also include the application of a linear expansion to the quantized vector.

[0085] This expansion consists of reducing the quantized vector, which is in a reduced-dimensional subspace, to the initial dimension of the original latent vector. This inverse operation can be performed by multiplying the quantized vector by a predefined expansion matrix stored in memory.

[0086] The main advantage of this expansion is that it allows for the progressive and precise reconstruction of the initial latent representation from the quantized results of each stage. Indeed, since each stage operates on a reduced-dimensional vector, this characteristic makes it possible to work backward to the initial dimension before combining the results from the different stages. This ensures a consistent and accurate accumulation of the contributions from each quantization step.

[0087] The quantization of the latent representation can be of the multi-stage type and can also include, for each stage, the updating of the vector from said latent representation, by subtraction with the quantized vector of said stage.

[0088] Thus, at each stage of the multi-stage quantization of the latent representation, the input vector is updated by a specific operation. After performing spherical vector quantization of the latent vector projected to a given stage, the resulting quantized vector is subtracted from the initial vector from the latent representation, forming a new vector representing the residual quantization error. This new vector, called the residual vector, then becomes the input for the next stage of quantization. In this way, each successive stage progressively refines the latent representation by addressing only the residual error left by the previous steps.

[0089] This subtractive update method progressively improves the accuracy of the quantization. By successively addressing the residual error of each stage, the quality of the final representation is gradually enhanced. This strategy allows for precise control of the reconstructed audio signal's quality, while also offering flexibility in bitrate selection, as the number of stages can be adjusted according to the desired bitrate constraints.

[0090] The predefined spherical dictionary can be derived from a Gosset lattice, for example of the RE8 type.

[0091] The Gosset lattice is a regular algebraic structure particularly well-suited to spherical quantization, as it allows for an optimal arrangement of points (codewords) on a reduced-dimensional (8-dimensional) sphere. This algebraic structure simplifies the search for the optimal codeword, thus reducing the complexity of encoding and decoding. Furthermore, thanks to its intrinsic structure, the amount of memory required to store this dictionary is reduced compared to unstructured dictionaries, enabling a more efficient and less expensive implementation. The RE8 lattice structure, for example, provides a good distribution of codewords, ensuring high-quality audio signal reconstruction after decoding, even at low bit rates.

[0092] The projected vector can be obtained by linear projection.

[0093] The linear projection operation consists of transforming the initial vector from the latent representation into a lower-dimensional vector. This projection is performed by multiplying the initial latent vector by a predefined matrix, called the projection matrix, which serves to reduce the complexity of the vector to be quantified while preserving its main characteristics, with the possible addition of a bias (or vector shift) – the bias being defined in the sense of linear layers of neural networks.

[0094] The selection of the optimal codeword can be performed by maximizing a dot product, the exact definition of this dot product being able to be modified depending on the point network used – for example for RE8 the dot product is modified with a penalty related to the sign parity constraint of RE8.

[0095] This operation involves comparing the projected vector, obtained after linear projection, to each leader vector in the predefined spherical dictionary by calculating their dot product. This operation is equivalent to identifying the code word whose direction is closest to that of the vector projected through a leader vector.

[0096] The main advantage of this approach is that it greatly simplifies the process of selecting the optimal code word.

[0097] The optimized gain factor can be determined "instantaneously" (i.e., adaptively, as a function of the projected vector) by minimizing a quadratic error between the projected vector and the selected codeword.

[0098] Using a quadratic error measure to determine the optimized gain factor has the advantage of being simple to calculate, while providing an efficient and accurate representation of the original projected latent vector. Indeed, the algebraic simplicity of this metric promotes fast and efficient implementation.

[0099] The projected latent vector can undergo a preliminary centering operation by subtracting a predefined average value.

[0100] This characteristic helps reduce the inherent bias of the data to be encoded. Indeed, by subtracting this average value, the process ensures that the vectors to be encoded have a mean of zero (or close to zero), which improves the efficiency of spherical vector quantization, which assumes that the values ​​to be encoded are centered.

[0101] The codeword selection and codeword scaling operations can be performed jointly in a single operation.

[0102] The predefined spherical dictionary can be formed from a set of absolute leaders or absolute direction vectors in a point network, and in which the selection of the codeword includes: - sorting the components of the vector from the latent representation in descending order and storing the corresponding permutation, - calculating a sign parity associated with the components of said vector from the latent representation, - comparing the vector from the latent representation with each absolute leader, - applying the permutation and the sign parity to a selected absolute leader to obtain the corresponding quantified codeword.

[0103] A gain factor can be encoded by scalar quantization on a predefined number of bits that are part of the binary stream, and the index corresponding to this gain is multiplexed with the index of the codeword in this binary stream, the value of the quantized gain factor then being taken as the optimized gain factor.

[0104] The invention also relates to a method for decoding a coded audio signal comprising: - demultiplexing a received binary stream in order to obtain at least indices representing quantized vectors; - decoding the indices obtained by selecting the corresponding codewords in a predefined spherical dictionary; - applying scaling by an optimized gain factor to each decoded codeword; - reconstructing a latent vector from the decoded and scaled codeword; - synthesizing the decoded audio signal from the reconstructed latent vector using a synthesis neural network.

[0105] In one embodiment, a linear expansion is applied to the decoded and scaled codeword before the latent vector reconstruction step.

[0106] Several reconstructed latent vectors can be successively added together to reconstruct a latent representation of the audio signal.

[0107] The invention also relates to a coding device comprising a processing circuit configured to implement the aforementioned coding process.

[0108] The invention also relates to a decoding device comprising a processing circuit configured to implement the aforementioned decoding process.

[0109] The invention also relates to a computer program comprising instructions capable of being executed by a processor to carry out the aforementioned encoding process and / or the aforementioned decoding process.

[0110] The invention also relates to a processor-readable storage medium that stores a computer program containing instructions for executing the aforementioned encoding process and / or the aforementioned decoding process.

[0111] Other features, details, and advantages will become apparent upon reading the detailed description below and analyzing the attached drawings, on which: Fig. 1

[0112] illustrates an example of encoder and decoder using neural networks with latent space quantization; Fig. 2

[0113] illustrates an example of multi-stage vector (unstructured) quantization of latent space according to the state of the art; Fig. 3

[0114] illustrates an example of multi-stage vector (unstructured) quantization decoding of the latent space according to the state of the art; Fig. 4

[0115] illustrates an example of multi-stage vector (unstructured) quantization of latent space according to the state of the art; Fig. 5

[0116] illustrates an example of multi-stage vector (unstructured) quantization decoding of the latent space according to the state of the art; Fig. 6

[0117] illustrates an example of the realization of multi-stage spherical vector quantization of latent space according to the invention; Fig. 7

[0118] illustrates an example of multi-stage spherical vector quantization decoding of latent space according to the invention; Fig. 8a, Fig. 8b, Fig. 8c

[0119] ,,illustrate an example of the implementation of the analysis, synthesis and quantification parts of an example of a neural audio codec used according to the invention; Fig. 9

[0120] illustrates examples of structural realization of an encoding device and a decoding device according to an embodiment of the invention.

[0121] We first describe the implementation of a first embodiment according to the invention with reference to figures 6 and 7.

[0122] In this embodiment, we take as an example the encoder (analysis network) 101 and the decoder (synthesis network) 112 from the DAC codec with pre-trained parameters according to the complete DAC implementation given in: https: / github.com / descriptinc / descript-audio-codec, retaining only the DAC model pre-trained at 44.1 kHz as an example of implementation – this model may be referred to as the “original DAC” codec.

[0123] The encoding 102 and decoding 111 operations of the latent space are replaced (at least in one quantization stage) by the methods according to the invention. This embodiment has the advantage of not requiring any retraining of the autoencoder (except for quantization). However, since the training of this autoencoder is performed end-to-end including quantization, replacing the quantization after the fact, at inference, will necessarily be suboptimal. The case where the invention is inserted into a fully retrained codec, taking into account the quantization of the latent space according to the invention, is addressed in the second embodiment described later.

[0124] As illustrated in the, the quantization of the latent space z is implemented by using here a multi-stage structure, consisting of K quantifiers or quantization blocks, denoted 6-Bk, k=1, …, K, with a successive calculation of the quantization error after each stage in 6-Bk-9.

[0125] We detail here the general case of the kth stage. Indeed, the invention makes it possible to replace in a "modular" way any existing stage (4-B1 to 4-BK) of the DAC codec, where K=9 in the embodiment example.

[0126] In the first embodiment, the projection operations in 6-BK-1 and expansion in 6-BK-8 correspond exactly to blocks 4-BK-1 and 4-BK-3 of the "original" DAC codec (available with pre-trained parameters). Therefore, we find a projected error ε k of dimension 8 as in DAC and a vector z of the latent space of dimension 1024.

[0127] The invention is based on quantization (block 6-Bk-4) via a spherical vector quantization dictionary, with scaling (block 6-Bk-6) by an optimized gain (6-Bk-5). This quantization produces a quantization index i k preferentially on 10 bits, but this 10-bit rate may be different in variants of the embodiment (examples at 8 or 12 bits are described later).

[0128] The implementation of the 6-Bk-4 block according to the invention is detailed below.

[0129] According to the invention, spherical vector quantization is used in at least one latent space quantization stage. This method consists of constraining the dictionary to have codewords that all have the same norm, which can be lossless and, by convention, set to the value of 1. If the norm of the vectors in the spherical dictionary is different from 1, it suffices to adjust the adaptation gain factor used according to the invention accordingly. Quantization then amounts to encoding only the orientation of the input vector, and the nearest neighbor search is greatly simplified. When the error criterion (distortion) is the quadratic error (or square of the L2 norm), the error between x (input) and y (codeword) can indeed be minimized in the form: || x – y|| 2 = || x || 2 + || y || 2 – 2 x T .y

[0130] Since the first two terms are constant, minimizing the error amounts to maximizing the dot product, which in fact corresponds directly to the cosine similarity up to a factor (equal to the norm of ||x||, which is a constant factor in the search). The result of the search by maximizing x T .y does not depend on the norm of || x ||.

[0131] For a given spherical dictionary, and a search for the nearest neighbor according to the maximization of the scalar product, it remains to optimize the quantification performance to apply a scale factor denoted g to the orientation denoted y resulting from the search.

[0132] It is known to those skilled in the art that the optimal "instantaneous" gain is obtained by minimizing, for given x and y and for the squared error: min g || x – g. y|| 2 = || x || 2 + g 2 .|| y || 2 – 2 .gx T .yce which gives, when || y ||=1, g= xT .y

[0133] This gain value g must be quantized, either as a prefix value or using scalar quantization (uniform or non-uniform) over a predetermined number of bits. At low bit rates (e.g., 10 bits in dimension 8, or 1.25 bits / dimension), it is advantageous not to use any bit budget to encode this gain g and to allocate the entire budget to the codeword y resulting from the search.

[0134] When the optimal gain g (for a given input vector x) is not encoded on a predetermined number of bits, a unique "optimized" gain must be determined, which is pre-stored and not transmitted. This optimized gain can, for example, be obtained to minimize the average quantization error. Various methods can be used to determine the optimal gain factor g to use.

[0135] Assuming that the input source of block 6-Bk-4 follows an independent and identically distributed (i.id.) Gaussian distribution, it is possible to perform a random sampling of Gaussian vectors x1, …, x N in dimension n and with the same variance as the source, and to test a series of possible values ​​for the fixed gain g (with a "grid search" type optimization) by measuring the mean squared error ( || x1 – g. y1 || 2 + …+ || x N – g. y N || 2 ) / N after quantification and to retain the gain g which minimizes this error.

[0136] In another method used in the first embodiment, the gain g applied in each 6-Bk-5 block is determined, for example, by using the pre-trained dictionaries of the original DAC codec (block 4-Ck) and measuring the average norm of the codewords in 4-Ck, then decomposing the optimized gain g into two gain factors: g = γ. g corr, where γ is the fixed gain optimized on a Gaussian source in dimension n=8 (with a random draw for example with N=100000) of unit variance, and g corr is a correction factor which is the ratio between the average norm of the code words in the predefined dictionary (block 4-Ck) and the average norm of the Gaussian vectors in dimension n=8 and with unit variance (this average norm is approximately 2.74). This form of decomposition is given in Tables 2 and 3.

[0137] Other methods for determining the optimized gain factor g are possible.

[0138] According to the invention, all types of spherical dictionaries are conceivable. Two variants are described here in more detail.

[0139] In the first case, quantification relies on the Gosset network in dimension 8, as studied in the aforementioned article by Lamblin and Adoul.

[0140] According to the invention, it is possible to use absolute leaders which allow for nearest neighbor search and more efficient indexing.

[0141] An implementation of the three algorithms enabling spherical vector quantization in the RE8 lattice is described above:

[0142] The search for the nearest neighbor

[0143] Nearest Neighbor Indexing

[0144] Decoding the clue

[0145] The first two (nearest neighbor search and indexing) correspond to a realization of block 6-Bk-4 (noted Q k for quantification at the k level), while the third defines a realization of block 7-Bk-1 (denoted D k for decoding at level k).

[0146] In the preferred embodiment, the search for the nearest neighbor is based on absolute (normalized) leaders of RE8.

[0147] We define the search for the nearest neighbor of an input vector from block 6-Bk-4 which gives the code word output from the same block. To simplify the notation, the input will be noted and the exit .

[0148] The spherical dictionary is defined according to the invention as a list of absolute (non-normalized) L leaders with the corresponding predefined sign parity denoted where k=1,…,L.

[0149] In the description below, we adopt, for an arbitrary initial vector the following notation: indicates the absolute value operation applied to each of the components of which amounts to asking Furthermore, sorting the components of a vector in descending order will give a vector which will be noted Thus, we will adopt the convention that an absolute leader can be noted in the form because its components are by definition all positive and sorted in descending order.

[0150] Finding the nearest neighbor in such a dictionary follows these steps: 1. Determine the absolute vector obtained by sorting the components of , which amounts to swapping the elements of 1. Store the inverse permutation in memory. 2. Calculate the sign parity. defined as: Or gives the parity of x (0 if even, 1 if odd) and sign(x) gives the sign of x with sign(x)=1 if x<0, 0 otherwise. 3. Calculate the dot product with the absolute leaders L (which are standardized here) Or if And if 4. Find the best absolute leader by searching for the value of k=1,..,L where is maximal – the index of the best leader is denoted k*5. Reconstruct the codeword with the following steps: a. Initialize the codeword from the absolute leader at index k*: , j=1,…8 and reverse the sign of if a. Apply the inverse permutation (stored in step 1) to b. Apply the signs of the entry : reverse the sign of if for j=1,…8

[0151] The steps defined above are very close to the nearest neighbor search algorithm using absolute leaders described in the aforementioned article by Lamblin and Adoul. However, the absolute leaders are not all constrained to be on the same RE8 sphere. The calculation of the dot product therefore includes an explicit normalization by the factor .

[0152] In some variants, it will be possible to pre-store normalized absolute leaders. / for k=1, …, L, which avoids a division at each dot product calculation.

[0153] We now describe a method for indexing codewords based on absolute leaders, according to the invention. This method is not described in the article by Lamblin and Adoul.

[0154] To simplify the notation, the index i k from block 6-Bk-4 (noted Q) k for quantification at level k) is here simply noted "i".

[0155] The index i of a codeword is determined as follows: Where: - The term is the sign code of This sign code is defined according to the invention as follows: if the absolute leader is even, is an integer defined in binary representation (bit by bit) by "juxtaposing" the sign bits (1 for , 0 if 0) at non-zero positions (where 0); if the absolute leader is odd, is an integer defined in binary representation (bit by bit) by "juxtaposing" the first 7 sign bits of (1 for , 0 if 0 for j=1,…,7). For example, in the case of an even absolute leader like [2, 2, 0, 0, 0, 0, 0, 0], the sign code associated with the signed permutation [0, -2, 0, 0, 2, 0, 0, 0] (which has no sign parity constraint) will be obtained from the sign bits [1, 0] = [sign(y2), sign(y5)] from the only two non-zero positions (2 and 5), which gives in binary the code '10' (in base 2) which corresponds to the integer value "2" (in base 10). For example, in the case of an odd absolute leader like [1, 1, 1, 1, 1, 1, 1, 1], the sign code associated with the signed permutation [-1, 1, -1, -1, 1, 1, 1, -1] (which has an even number of negative signs satisfying the leader parity constraint) will be obtained from the sign bits [1, 0, 1, 1, 0, 0, 0, 1] = [sign(y1), …, sign(y8)], by removing the last sign bit, which gives in binary the code '1011000' (in base 2) which corresponds to the integer value "88" (in base 10). - The term gives the number of permutations of the absolute leader - The term is the rank (or index) of the absolute leader permutation associated with In the preferred embodiment, this rank is determined according to Schalkwijk's formula, as in the aforementioned article by Lamblin and Adoul. - The term is the cardinality of the absolute leader which corresponds to the number of possible signed permutations in RE8.

[0156] We now describe the decoding of index i k in block 7-Bk-1 (noted D) k for decoding at level k). To simplify the notation, the index i k The input to block 7-Bk-1 is here simply noted as "i".

[0157] The decoding of index i is described in the following steps: 1. Find the index of the absolute leader by successive comparison of the value of i with the pre-calculated and stored quantities (offset table or cardinality shift) given by for k* = 1 to K+1 with the convention that for k* = 1 the sum is zero, 2. Update the index i by subtracting the cardinality offset corresponding to the absolute leader of index found in the previous step: 3. Extract the sign code and the rank of the permutation starting from the index i from step 2: this is equivalent to calculating the integer division of i by , the quotient corresponds to and the rest to 4. Decode the rank of the permutation to find the absolute decoded vector 5. Reverse the sign of the components of based on the sign code representing the whole in binary form and by reversing the sign of the component if the sign code indicates a sign bit of 1 for the index component For example, in the case of an even absolute leader like [2, 2, 0, 0, 0, 0, 0, 0], the sign code will be decomposed into base 2 using 2 bits in '10' because there are only 2 non-zero positions in the absolute leader. If the permutation rank resulted in the absolute decoded vector [0, 2, 0, 0, 2, 0, 0, 0], then applying the sign bits will lead to [0, -2, 0, 0, 2, 0, 0, 0] because the first sign bit '1' corresponds to the first non-zero position in the decoded absolute vector, and the second sign bit '0' corresponds to the second non-zero position. For example, in the case of an odd absolute leader like [1, 1, 1, 1, 1, 1, 1, 1], the sign code This will give the value '1011000' in base 2 as 7 bits, and we will apply the first 7 signs to the first 7 positions, the sign of the last being determined by the sign parity constraint of the leader. Here the code '1011000' contains 3 bits set to 1, while the absolute leader has even sign parity (ISIG=0), which implies that the sign of the last component is necessarily negative. We thus reconstruct: [-1, 1, -1, -1, 1, 1, 1, -1].

[0158] In the first embodiment, the complete DAC model was trained including RPVQ quantization and replacing at least one quantization stage with spherical vector quantization requires several adaptations: not only must the spherical dictionary be scaled by a factor or gain, but the input before spherical vector quantization must be "centered".

[0159] Optionally, the invention also includes centering blocks (i.e., subtraction by an average) and offset blocks (by the same average), respectively in blocks 6-Bk-3 and 6-Bk-7, the average being given by block 6-Bk-2. In an implementation of the first embodiment, the optimized fixed value of the average vector in dimension 8 (block 6-Bk-2, k=1, …, 9) is given in Table 1 for each of the stages k = 1 to 9.

[0160] Furthermore, in an implementation of the first embodiment, the 10-bit RE8 dictionary defined in Table 2 will preferably be used. This dictionary has only one absolute leader, and the general form of the nearest neighbor search and indexing methods is then greatly simplified. In alternative implementations, another dictionary (such as the other 10-bit dictionary denoted "alt." for alternative) may be used, or the 10-bit bit rate may be changed, for example, by using an 8-bit or 12-bit dictionary as defined in Table 2.

[0161] In Table 2, the 8-bit dictionary includes two leaders from sphere 1 of RE8 (240 points) and one additional leader (16 points) from sphere 2 of RE8, for a total of M = 256 codewords. At 10 bits, two dictionary variants are defined with M = 1024 codewords: a dictionary with a single absolute leader (incomplete sphere 2 of RE8) and a dictionary (denoted 'alt') with four absolute leaders that mixes several spheres of RE8 (spheres 1, 3, 5, and 10). At 12 bits, the (incomplete) dictionary with M = 4080 codewords (instead of the possible 4096) includes sphere 2 of RE8 (2160 points), one absolute leader from sphere 1 (128 points), and another absolute leader from sphere 3 (1792 points).

[0162] Finally, in an implementation of the first embodiment, the optimized fixed value of the correction gain can be defined as in Table 3, the fixed adaptation gain of the spherical dictionary being given in the 6-Bk-5 block, k=1, …, 9 in the form: gk = γR..gcorr where γR depends on the flow rate R and is given in Table 2 and gcorr is given in Table 3 for each stage k. In variants, the gain γR can be integrated into the spherical dictionary in 6-Bk-4 and the correction gain in 6-Bk-5, ultimately obtaining the same scaling of the predefined spherical dictionary in 6-Bk-4.Recall that in the implementation example described here, γR is the fixed gain optimized for the R-bit dictionary on a Gaussian source in dimension n=8 (with random sampling, for example, with N=100000), and gcorr is a correction factor that is the ratio between the average norm of the codewords in the predefined dictionary (block 4-Ck) and the average norm of the Gaussian vectors in dimension n=8. However, in other versions, different methods of determining the gain gk can be used, with a directly defined value of gk or a value of gk decomposed into several sub-factors. The optimized fixed values ​​of the correction gain in Table 3 are obtained empirically from the quantization dictionaries available in the DAC model. The gains presented correspond, for a stage k, to the average norm of the codewords of stage k divided by 2.74, a value corresponding to the average norm of an iid Gaussian vector in dimension 8.

[0163] Table 1: Optimized fixed value of the mean vector (block 6-Bk-2, k=1, …, 9)kµ k1[-0.11520223, -0.12549783, -0.09464508, 0.16919054, -0.0787252, 0.11072443, 0.09925366, 0.02332132]2[-0.03650802, 0.05348687, 0.10868465, -0.02770074, 0.02295331, -0.00783541, 0.11034289, -0.05973633]3[0.03956118, 0.02418371, 0.06306202, -0.0390567, 0.04374486, 0.009958, -0.07310977, -0.05468717]4[-0.01159949, -0.00974658, 0.01323013, -0.00341866, 0.04071631, 0.0255354, -0.0304246, -0.00231216]5[-0.04118184, -0.00945165, 0.01324847, -0.05061552, -0.02626047, -0.03228011, -0.00290333, -0.03157319]6[-0.00160538, 0.0416545, -0.00642506, -0.02625478, -0.03411976, -0.02458269, -0.01719413, 0.01888132]7[-0.00762004, -0.01692576, 0.03200294, 0.0160811, 0.01259751, -0.01412598, 0.00069721, -0.00634967]8[-0.00645784, -0.05261763, -0.00584335, 0.02851553, -0.04655663, -0.01716808, 0.04031012, -0.00355523]9[0.00228948, -0.02568473, -0.00589426, 0.00665507, 0.00271465, 0.02891909, 0.00439095, -0.04728379]

[0164] Table 2: Absolute leaders with additional parameters (parity, cardinality) according to the R throughput – at 10 bits, an alternative case is also defined. RGain opt. throughput (γ R )KDE Leader Index (Absolute Leader - Unnormalized) Parity (ISIG) k Cardinality )82.290[2, 2, 0, 0, 0, 0, 0, 0]01121[1, 1, 1, 1, 1, 1, 1, 1]01282[4, 0, 0, 0, 0, 0, 0, 0]016102.450[3, 1, 1, 1, 1, 1, 1, 1]1102410 (alt.)2.400[1, 1, 1, 1, 1, 1, 1, 1]01281[6, 2, 0, 0, 0, 0, 0, 0]02242[4, 4, 4, 0, 0, 0, 0, 0]04483[8, 4, 0, 0, 0, 0, 0, 0]0224122.510[1, 1, 1, 1, 1, 1, 1, 1]01281[4, 0, 0, 0, 0, 0, 0, 0]0162[2, 2, 2, 2, 0, 0, 0, 0]011203[3, 1, 1, 1, 1, 1, 1, 1]110244[2, 2, 2, 2, 2, 2, 0, 0]01792

[0165] Table 3: Optimized fixed value of the matching gain; the value of 2.74 corresponds to the average norm of an iid Gaussian vector in dimension 8. The gain gk (block 6-Bk-5, k=1, …, 9) is given by gk = γR. gcorr, where g is the index of the stage. corr110,641 / 2,7429,030 / 2,7438,294 / 2,7447,840 / 2,7457,505 / 2,7467,247 / 2,7476,985 / 2,7486,681 / 2,7496,337 / 2,74

[0166] During the nearest neighbor search step in a 6-Bk-4 block, the optimal gain that minimizes the error is obtained through the dot product. We therefore also obtain a scaling factor. which was moved from block 6-Bk-4 to 6-Bk-5 to be quantized. The encoding of this gain on a predetermined number of bits (for example 2 bits) in the 6-Bk-5 block leads to transmitting a gain quantization index jk (for example on 2 bits) which is multiplexed in 600. In an implementation of the first embodiment, the gain quantization can use a non-uniform logarithmic type quantization with reconstruction values ​​{-3 dB, 0 dB, 3 dB, 6 dB} around a predefined value such as the value of gcorr given in Table 3. Other forms of scalar quantization are possible, for example a scalar dictionary { gk,1, …, gk,Q} (where Q is the number of possible gains) can be trained by K-means taking as the source to be encoded the "instantaneous" gain g from the 6-Bk-4 block, knowing the spherical dictionary defined in 6-Bk-4 in quantization.

[0167] If the spherical quantization in 6-Bk-4 is different from the RE8 quantization, the principle of calculating the optimal gain remains the same.

[0168] The optimal "instantaneous" gain is obtained by minimizing, for given x and y and for the squared error:min g || x – g. y|| 2 = || x || 2 + g 2 .|| y || 2 – 2 .gx T .yce which gives, when || y ||=1, = x T .y where x is the input of 6-Bk-4 and y is the output of 6-Bk-4.

[0169] In this variant, we can choose to take an 8-bit spherical dictionary in 6-Bk-4 and a 2-bit scalar quantization of the gain in 6-Bk-5 (for example driven by K-means), which gives an overall budget of 10 bits in the k stage.

[0170] As illustrated in the diagram, the decoding of the latent space is implemented by using a multi-stage structure, consisting of K decoding blocks, denoted 7-Bk, k=1, …, K, with successive reconstructions of the latent space after each stage in 7-Bk-7 (k>1).

[0171] The decoding consists, for a given block k, of applying in 7-Bk-1 the decoding of index i k Demultiplexed to 700, the same blocks are then repeated at the corresponding quantization stage with scaling (6-Bk-6) using the matching gain (6-Bk-5). Optionally, if the average (in 6-Bk-2) was subtracted from the quantization, this average is added in 6-Bk-8. Finally, the same expansion block as in the encoder (6-Bk-8) is applied.

[0172] In variants, if the adaptation gain is coded on a predetermined (non-zero) number of bits, the index j k is multiplexed.

[0173] We now describe the implementation of a second embodiment according to the invention, this one also refers to figures 6 and 7 (with adaptations), as well as to the.

[0174] In this second embodiment, it is assumed that the DAC model is also modified as in the first embodiment, with two main differences:

[0175] The average (blocks 6-Bk-2) with the associated operations (6-Bk-3 and 6-Bk-7) may not be implemented.

[0176] The 6-Bk-5 gain is learned by the complete training of the modified DAC model, which also includes model auxiliary functions such as GANs defined with the DAC model in the aforementioned available implementation.

[0177] Thus, in this embodiment, the average (blocks 6-Bk-2) is not necessary because it is implicitly learned and integrated into the linear operations of projection (6-bk-1) and expansion (6-Bk-8).

[0178] The invention according to the second embodiment is illustrated with reference to Figure 6. The latent space quantization (102) and decoding (111) blocks, corresponding respectively to Figures 6 and 7, are summarized here in the top figure with a succession of quantization stages. Each stage is illustrated in the bottom figure as a computation graph in 960, which in fact represents block 6-Bk, where 961, 962, 963, and 964 correspond respectively to 6-Bk-1, 6-Bk-4, 6-Bk-5 / 6-Bk-6 (as a whole), and 6-Bk-8.

[0179] The advantage of defining an adaptation gain factor in block 963 is that it allows the spherical quantization dictionary to be "parameterized." Empirically, it has been verified that if this gain is not present, the complete training of the modified DAC model (with spherical vector quantization in at least one stage, but without a gain factor on the spherical dictionary) performs worse than according to the invention. In theory, the projection layer in 961 and the expansion layer in 964 could automatically adapt to the spherical dictionary in 962; in practice, it turns out that adding a gain factor in 963 is important for better performance.

[0180] In this second embodiment, the complete architecture including blocks 101, 112 and 102+111 as aux, 8c, is re-trained from a predetermined audio training base.

[0181] This leads to a modified DAC codec, where the source code of the original DAC codec is modified to replace the latent space quantization (102+111) with spherical codes and at least one adaptation gain (typically one gain per stage in multi-stage quantization). Training can, for example, be performed with the EARS speech dataset defined in the article by Richter et al., “EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation,” Proc. ISCA Interspeech, 2024, pp. 4873–4877.

[0182] In variants of the invention, the adaptation gain (block 6-Bk-5 in the encoder and decoder) may be integrated directly into the spherical dictionary or into the quantization (block 6-Bk-4) and decoding (6-Bk-1) block; in this case the scaling (multiplication) operation in block 6-Bk-6 (in the encoding and decoding) will be integrated into the quantization (block 6-Bk-4) and decoding (6-Bk-1) block.

[0183] Similarly, in variants of the invention, the operations of subtracting and adding the mean (blocks 6-Bk-3 and 6-Bk-7), if necessary, may be integrated into the blocks of projection and expansion which are linear operations (blocks 6-Bk-1 and 6-Bk-8).

[0184] In variants of the invention, spherical vector quantization according to the invention can be applied in any residual quantization stage, and the number of residual quantization stages applying the invention can be from 1 to K where K is the number of residual stages.

[0185] In embodiments of the invention, spherical vector quantization with at least blocks of type 6-Bk-4, 6-Bk-5, and 6-Bk-7 can be applied directly in the latent space of a neural network model with an analysis component and a synthesis component using neural networks, without going through projection steps (6-Bk-1 and 6-Bk-8). Depending on the neural network model used, however, the use of averaging (blocks 6-Bk-2, 6-Bk-3, and 6-Bk-7) can be added.

[0186] In variations of the invention, point lattices other than RE8 may be used to define spherical quantization dictionaries. However, in this case, the nearest neighbor search, indexing, and decoding algorithms will be different because the methods described above are specific to the RE8 structure. Reference may be made, for example, to the following two articles for cases of 16 or 24 dimensions with the Barnes-Wall and Leech point lattices (respectively): - C. Lamblin, JP Adoul, D. Massaloux and S. Morissette, "Fast CELP coding based on the Barnes-Wall lattice in 16 dimensions," Proc. ICASSP, vol. 1, 1989. - Adoul, J.-P. and Barth, M., “Nearest neighbor algorithm for spherical codes from the Leech lattice”, IEEE Trans. Inf. Theory, vol. 34, no. 5, 1988 Furthermore, in variants of the invention, the quantification dimension may be different from dimension 8, using a different point network than RE8.

[0187] Furthermore, the invention may more generally rely on spherical codes such as permutation codes as defined in ITU-T Recommendation G.729.1 (initially for discrete frequency spectrum coding by standardized sub-bands).

[0188] Finally, in other variants, blocks 6-Bk-4 and 7-Bk-1 could use a non-spherical dictionary derived from a dot lattice, such as one of the dictionaries (Q2, Q3, or Q4 at 8, 12, or 16 bits) described in the article by S. Ragot, B. Bessette, and R. Lefebvre, "Low-Complexity Multi-Rate Lattice Vector Quantization with Application to Wideband TCX Speech Coding at 32 kbit / s," Proc. ICASSP, 2004. In this case, blocks 6-Bk-4 and 7-Bk-1 must incorporate additional gain that scales the input / output of block 6-Bk-4 and the output of block 7-Bk-1. Since the dictionary is no longer spherical, finding the nearest neighbor requires a "truncating" operation for rounded points in RE8 that fall outside the dictionary. This variant necessitates optimizing for an additional gain.

[0189] We have illustrated on the, a DCOD encoding device and a DDEC decoding device, in the sense of the invention, these devices being dual to each other (in the sense of "reversible") and connected to each other by a communication network RES.

[0190] The DCOD encoding device includes a processing circuit typically comprising: - a memory MEM1 for storing instruction data of a computer program as defined in the invention (these instructions being distributed between the DCOD encoder and the DDEC decoder) as well as data calculated during a learning phase; - an interface INT1 for receiving an audio signal of origin x; - a processor PROC1 for receiving this signal and processing it by executing the computer program instructions stored in the memory MEM1, for the purpose of encoding it; in particular, the processor being capable of driving a neural network-based analysis module and a latent representation quantification module as described with reference to Figures 6 and 7; and - a communication interface COM 1 for transmitting the encoded signals via the network.

[0191] The DDEC decoding device comprises its own processing circuit, typically including: - a MEM2 memory for storing instruction data from a computer program as defined in the invention (these instructions can be distributed between the DCOD encoder and the DDEC decoder as previously described) as well as data calculated during a learning phase; - a COM2 interface for receiving coded signals from the RES network for decoding by compression as defined in the invention; - a PROC2 processor for processing these signals by executing the computer program instructions stored in the MEM2 memory, for decoding purposes; in particular, the processor being capable of driving a latent space decoding module as described with reference to Figures 6 and 8 and a neural network-based synthesis module; and - an INT2 output interface for delivering the decoded audio signal. .

Claims

A method for encoding an audio signal comprising: performing a quantization of a latent representation obtained from an analysis of the audio signal by a neural network, on at least one quantization stage, with, for each quantization stage:-- the selection of a codeword, by comparison between a vector from said latent representation and codewords belonging to a predefined spherical dictionary,-- the scaling of the selected codeword by an optimized gain factor in order to form a quantized vector;-- the generation of an index representing the quantized vector; multiplexing the generated indices to form at least part of a binary train representing the audio signal. A method according to any one of the preceding claims, wherein the vector from said latent representation is a vector projected onto a reduced-dimensional subspace, each stage also comprising the application of a linear expansion to the quantized vector. A method according to any one of the preceding claims, wherein the quantization of the latent representation is of the multi-stage type and also includes, for each stage, updating the vector from said latent representation, by subtraction with the quantized vector of said stage. A method according to one of the preceding claims, wherein the predefined spherical dictionary is derived from a Gosset lattice. A method according to any one of the preceding claims, wherein the projected vector is obtained by linear projection. A method according to any one of the preceding claims, wherein the codeword selection is performed by maximizing a dot product. A method according to any one of claims 2 to 6, wherein the optimized gain factor is determined by minimizing a quadratic error between the projected vector and the selected code word. A method according to any one of claims 2 to 7, wherein the projected vector undergoes a preliminary centering operation by subtracting a predefined average value. A method according to any one of the preceding claims, wherein the codeword selection and codeword scaling operations are performed jointly in a single operation. A method according to any one of the preceding claims, wherein the predefined spherical dictionary is formed from a set of absolute leaders or absolute direction vectors in a point lattice, and wherein the codeword selection comprises: sorting the components of the vector from the latent representation in descending order and storing the corresponding permutation, calculating a sign parity associated with the components of said vector from the latent representation, comparing the vector from the latent representation with each absolute leader, and applying the permutation and sign parity to a selected absolute leader to obtain the corresponding quantified codeword. A method according to any one of the preceding claims, wherein a gain factor is encoded by scalar quantization on a predefined number of bits forming part of the binary stream and the index corresponding to this gain is multiplexed with the index of the codeword in this binary stream, the value of the quantized gain factor then being taken as the optimized gain factor. A method for decoding a coded audio signal comprising: demultiplexing a received binary stream to obtain at least indices representing quantized vectors; decoding the indices obtained by selecting the corresponding codewords from a predefined spherical dictionary; applying scaling by an optimized gain factor to each decoded codeword; reconstructing a latent vector from the decoded and scaled codeword; synthesizing the decoded audio signal from the reconstructed latent vector using a synthesis neural network. A decoding method according to the preceding claim, wherein several reconstructed latent vectors are successively added together to reconstruct a latent representation of the audio signal. Coding device comprising a processing circuit configured to implement the coding method according to any one of claims 1 to 11. Decoding device comprising a processing circuit configured to implement the decoding process according to claim 12 or 13. A processor-readable storage medium that stores a computer program containing instructions for executing the coding process according to any one of claims 1 to 11 and / or the decoding process according to claim 12 or 13.

Images