Controlling of reverberators

Abstract

An apparatus for spatial rendering of reverberation, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to perform: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

Description

[0001] CONTROLLING OF REVERBERATORS

[0002] Field

[0003] The present application relates to apparatus and methods for controlling of reverberators, but not exclusively for controlling of reverberators in augmented reality and / or virtual reality apparatus.

[0004] Background

[0005] Reverberation refers to the persistence of sound in a space after the actual sound source has stopped. Different spaces are characterized by different reverberation characteristics. For conveying spatial impression of an environment, reproducing reverberation perceptually accurately is important. Room acoustics are often modelled with individually synthesized early reflection portion and a statistical model for the diffuse late reverberation. Figure 1 depicts an example of a synthesized room impulse response where the direct sound 101 is followed by discrete early reflections 103 (or reflection echoes) which have a direction of arrival (DOA) and diffuse late reverberation 105 which can be synthesized without any specific direction of arrival. The delay d1(t) 102 in Figure 1 can be seen to denote the direct sound arrival delay from the source to the listener and the delay d2(t) 104 can denote the delay from the source to the listener for one of the early reflections (in this case the first arriving reflection). Additionally the delay d3(t) 106 can denote the delay from the source the onset of the diffuse late reverberation.

[0006] One method of reproducing reverberation is to utilize a set of D loudspeakers (or virtual loudspeakers reproduced binaurally using a set of head-related transfer functions (HRTF)). The loudspeakers are positioned around the listener somewhat evenly. Mutually incoherent reverberant signals are reproduced from these loudspeakers, producing a perception of surrounding diffuse reverberation.

[0007] The reverberation produced by the different loudspeakers has to be mutually incoherent. In a simple case the reverberations can be produced using the different channels of the same reverberator, where the output channels are uncorrelated but otherwise share the same acoustic characteristics such as reverberation time and level (specifically, the diffuse-to-direct ratio or reverberant-to-direct ratio or diffuse-to-total ratio or diffuse-to-source ratio or any other suitable parameter for representing reverberation energy or level). Such uncorrelated outputs sharing the same acoustic characteristics can be obtained, for example, from the output taps of a feedback delay network (FDN) reverberator with suitable tuning of the delay line lengths and mixing matrix, or from a reverberator based on using decaying uncorrelated noise sequences by using a different uncorrelated noise sequence in each channel. In this case, the different reverberant signals effectively have the same features, and the reverberation is typically perceived to be similar in all directions.

[0008] Summary

[0009] There is provided according to a first aspect an apparatus for spatial rendering of reverberation, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to perform: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0010] Wherein obtaining at least one threshold may cause the apparatus to perform: obtaining at least one first threshold associated with the at least one reverberation input signal level; and obtaining at least one further threshold associated with the at least one reverberation delay line signal level, wherein the apparatus caused to perform controlling the at least one reverberator is further caused to perform controlling the at least one reverberator at least based on: a comparison between the at least one reverberation input signal level against the at least one first threshold; and a comparison between the at least one reverberator delay line signal level against the at least one further threshold.

[0011] The at least one reverberator may comprise at least two delay lines, wherein the apparatus caused to perform determining the at least one reverberation delay line signal level may be further caused to perform: obtaining outputs from the at least two delay lines; and generating at least one reverberation delay line signal level based on a combination of the outputs from the at least two delay lines.

[0012] The apparatus caused to perform obtaining at least one audio signal may be further caused to perform obtaining at least two audio signals, and the apparatus caused to perform determining at least one reverberation input signal level may be caused to perform combining the at least two audio signals.

[0013] The at least one first threshold may be equal to the at least one further threshold. The apparatus caused to perform obtaining the at least one threshold may be caused to perform: obtaining a listener distance, the listener distance defining a distance between a listener and an audio environment; and generating the at least one threshold based on the listener distance.

[0014] The apparatus caused to perform controlling the at least one reverberator may be caused to perform at least one of: switching on or off the at least one reverberator based on the comparison; culling the at least one reverberator based on the comparison; reinstating the at least one digital reverberator based on the comparison; and attenuating at least one output of the at least one reverberator based on the comparison.

[0015] The at least one reverberator may be a late reverberation.

[0016] The at least one reverberator may be a digital reverberator.

[0017] The apparatus caused to perform configuring at least one digital reverberator based on at least one reverberation parameter may be further caused to perform at least one of: determining a number of reverberator output channels; determining feedback delay line lengths; determining feedback attenuation filter coefficients; determining reverberation ratio control filter coefficients; and determining a pre-delay line length.

[0018] According to a second aspect there is provided a method for spatial rendering of reverberation, the method comprising: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0019] Obtaining at least one threshold may further comprise: obtaining at least one first threshold associated with the at least one reverberation input signal level; and obtaining at least one further threshold associated with the at least one reverberation delay line signal level, wherein controlling the at least one reverberator further comprises controlling the at least one reverberator at least based on: a comparison between the at least one reverberation input signal level against the at least one first threshold; and a comparison between the at least one reverberator delay line signal level against the at least one further threshold.

[0020] The at least one reverberator may comprise at least two delay lines, wherein determining at least one reverberation delay line signal level may further comprise: obtaining outputs from the at least two delay lines; and generating at least one reverberation delay line signal level based on a combination of the outputs from the at least two delay lines.

[0021] Obtaining at least one audio signal may further comprise obtaining at least two audio signals, and determining at least one reverberation input signal level may comprise combining the at least two audio signals.

[0022] The at least one first threshold may be equal to the at least one further threshold. Obtaining the at least one threshold may further comprise: obtaining a listener distance, the listener distance defining a distance between a listener and an audio environment; and generating the at least one threshold based on the listener distance.

[0023] Controlling the at least one reverberator may further comprise at least one of: switching on or off the at least one reverberator based on the comparison; culling the at least one reverberator based on the comparison; reinstating the at least one digital reverberator based on the comparison; and attenuating at least one output of the at least one reverberator based on the comparison.

[0024] The at least one reverberator may be a late reverberation.

[0025] The at least one reverberator may be a digital reverberator.

[0026] Configuring at least one digital reverberator based on at least one reverberation parameter may comprise at least one of: determining a number of reverberator output channels; determining feedback delay line lengths; determining feedback attenuation filter coefficients; determining reverberation ratio control filter coefficients; and determining a pre-delay line length.

[0027] According to a third aspect there is provided an apparatus for spatial rendering of reverberation, the apparatus comprising means configured to: obtain at least one audio signal; configure at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determine at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtain at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; control the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0028] The means configured to obtain at least one threshold may be further configured to: obtain at least one first threshold associated with the at least one reverberation input signal level; and obtain at least one further threshold associated with the at least one reverberation delay line signal level, wherein the means configured to control the at least one reverberator may further be configured to control the at least one reverberator at least based on: a comparison between the at least one reverberation input signal level against the at least one first threshold; and a comparison between the at least one reverberator delay line signal level against the at least one further threshold.

[0029] The at least one reverberator may comprise at least two delay lines, wherein the means configured to determine at least one reverberation delay line signal level may further be configured to: obtain outputs from the at least two delay lines; and generate at least one reverberation delay line signal level based on a combination of the outputs from the at least two delay lines.

[0030] The means configured to obtain at least one audio signal may further be configured to obtain at least two audio signals, and determine at least one reverberation input signal level may comprise combining the at least two audio signals.

[0031] The at least one first threshold may be equal to the at least one further threshold. The means configured to obtain the at least one threshold may further be configured to: obtain a listener distance, the listener distance defining a distance between a listener and an audio environment; and generate the at least one threshold based on the listener distance.

[0032] The means configured to control the at least one reverberator may further be configured to at least one of: switch on or off the at least one reverberator based on the comparison; cull the at least one reverberator based on the comparison; reinstate the at least one digital reverberator based on the comparison; and attenuate at least one output of the at least one reverberator based on the comparison.

[0033] The at least one reverberator may be a late reverberation.

[0034] The at least one reverberator may be a digital reverberator. The means configured to configure at least one digital reverberator based on at least one reverberation parameter may be configured to at least one of: determine a number of reverberator output channels; determine feedback delay line lengths; determine feedback attenuation filter coefficients; determine reverberation ratio control filter coefficients; and determine a pre-delay line length.

[0035] According to a fourth aspect there is provided an apparatus an apparatus for spatial rendering of reverberation, the apparatus comprising: obtaining circuitry configured to obtain at least one audio signal; configuring circuitry configured to configure at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining circuitry configured to determine at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining circuitry configured to obtain at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling circuitry configured to control the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0036] According to a fifth aspect there is provided a computer program comprising instructions [or a computer readable medium comprising instructions] for causing an apparatus, for spatial rendering of reverberation, the apparatus caused to perform at least the following: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0037] According to a sixth aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus, for spatial rendering of reverberation, to perform at least the following: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0038] According to a seventh aspect there is provided an apparatus, for spatial rendering of reverberation,, the apparatus comprising: means for obtaining at least one audio signal; means for configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; means for determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; means for obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; means for controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0039] According to an eighth aspect there is provided a computer readable medium comprising instructions for causing an apparatus, for spatial rendering of reverberation, to perform at least the following: obtaining at least one audio signal; configuring at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal; determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level; obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; controlling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

[0040] An apparatus comprising means for performing the actions of the method as described above. An apparatus configured to perform the actions of the method as described above. A computer program comprising program instructions for causing a computer to perform the method as described above.

[0041] A computer program product stored on a medium may cause an apparatus to perform the method as described herein.

[0042] An electronic device may comprise apparatus as described herein.

[0043] A chipset may comprise apparatus as described herein.

[0044] Embodiments of the present application aim to address problems associated with the state of the art.

[0045] Summary of the Figures

[0046] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings in which:

[0047] Fig.1 shows a model of room acoustics with regard to the room impulse response; Fig.2 shows schematically an example a reverberator which includes an example feedback delay network (FDN) according to some embodiments;

[0048] Fig.3 shows schematically an example apparatus comprising the example reverberator as shown in Fig. 2 according to some embodiments;

[0049] Fig.4 shows schematically an example reverberator parameter determiner as shown in Fig.3 in further detail according to some embodiments;

[0050] Fig.5 shows a flow diagram of the operation of the example reverberator parameter determiner as shown in Fig.4;

[0051] Fig.6 shows schematically an example binaural renderer as shown in Fig.3 in further detail according to some embodiments; Fig.7 shows an example renderer incorporating the example apparatus as shown in Fig.3; Fig.8 shows schematically an example early reflection processor and renderer as shown in Fig. 3 according to some embodiments;

[0052] Fig.9 shows a flow diagram of the operation of the echo cull determiner as shown in Fig.2 according to some embodiments;

[0053] Fig.10 shows an example system within which some embodiments can be implemented; and

[0054] Fig.11 shows an example device suitable for implementing the apparatus shown in previous figures. Embodiments of the Application

[0055] The following describes in further detail suitable apparatus and possible mechanisms for echo cull control in rendering audio scenes.

[0056] In a virtual acoustics rendering system, reverberation is typically rendered as a combination of a certain number of distinct early reflections (or reflection echoes) and a stochastic model for the late reverberation. The early reflection synthesis is thus typically position-dependent in that it varies with source and listener positions, while the late reverberation synthesis is not. Together these two can create a plausible reverberation rendering for a physical or virtual space. The reverberation rendering is combined (summed) with direct sound rendering, which involves distance gain attenuation, air absorption filtering, and directional reproduction (binaural or loudspeaker) of the direct sound component that directly propagates to the ears of the listener without reflecting or reverberating in the space.

[0057] To produce a good quality reverberation output the late reverberation is configured such that the transition from the early reflections to the late reverberation is perceived as smooth and continuous, without noticeable gaps or energy fluctuation. The rendering is implemented by constructing the impulse responses offline (or as a background process) then the part of the impulse response corresponding to the later part can be processed to ensure it fits well with the early reflections.

[0058] Thus digital reverberators are configured in such a manner to emulate realistic acoustic spaces by combining an early reflection renderer, which reproduces time and direction varying discrete specular reflections with air- and surface-dependent absorption properties and a feedback delay network (FDN) to reproduce the coarse, stochastic decay of the late reverberation stage.

[0059] Even though the FDN is a computationally efficient method for rendering reverberation especially if the number of delay lines is set to a low value, computational loads for rendering reverberation can become onerous, for example where several acoustic environments, each with their own reverberator, are running in parallel.

[0060] There have been proposed methods for disabling reverberators based on, for example, the level of reverberation, the distance of the reverberant environment from the listener, and whether there are acoustic connections from the listener environment to the reverberant environment. However, such methods fail to determine how long to run any reverberator in order to ensure that no audible artefacts are generated by the enabling / disabling of the reverberator as there can still be echoes in the delay lines of a reverberator.

[0061] In other words, incorrect control and switching of reverberators can cause audible artefacts. For example an audible artefact can be generated by cutting the reverberant tail of a sound which sounds unnatural and reduces the immersion and plausibility of the VR or AR listening experience.

[0062] The concept thus as described in further detail by the following examples and embodiments relates to apparatus and methods of rendering of reverberation providing a mechanism for culling (or controlling or switching on / off) reverberators which do not contribute significantly to the audible signal and thus such that no audible artefacts occur. This can be achieved by monitoring at least one reverberation input signal parameter relative to a first (or input) threshold level and a reverberation delay line signal relative to a second (or further or output or delay line) threshold level, and not executing the reverberator (in other words culling the reverberator or switching off the reverberator or controlling the reverberator to be off) when the input signal level is less than the first threshold and the delay line signal level is less than the second threshold.

[0063] Furthermore in some embodiments the apparatus and methods can squelch, substantially attenuate or zero out the content of the delay lines of a reverberator which is stopped in order to prevent any ‘residual echoes’ stored in the FN from being output at a later time when the reverberator is re-started.

[0064] In some embodiments the first and second threshold values can be adjusted or modified based on listening feedback and whether the cull becomes audible. The thresholds furthermore can, in some embodiments, be adjusted to determine fixed values at which cull is not audible with a certain set of test signals.

[0065] In some embodiments the first or second threshold values can be dynamically adjusted based on levels of other signals in the system. For example the threshold values can be adjusted based on the levels of the direct sound and / or early reflections and / or other acoustic effects. In such a manner the threshold values can be adaptively increased as the level of other sounds increases as the other sounds are likely to mask the reverberation and changes in the rendered audio output due to stopping it.

[0066] In some embodiments the apparatus and methods can furthermore adaptively adjust the first or second threshold levels or values based on at least one of: a listener distance to the acoustic environment (for example the source associated with the reverberator);

[0067] a determination of acoustic connections from the listener environment to the reverberator environment; and

[0068] acoustic materials at the acoustic connections from the listener environment to the reverberator environment.

[0069] As discussed earlier, reverberation is typically rendered as a combination of at least two echo-generating components. A so-called early reflection echo synthesis component generates a certain number of perceptually distinct echoes, and a late reverberation synthesis component generates a stream of echoes which are relatively indistinct but adhere to the overall decay properties of a stochastic model for the late reverberation.

[0070] Reflection echo synthesis is typically spatially dynamic in that the levels and directions of arrival (DoAs) of the reflections depend on the source and listener positions and orientations. A reflection processor can be configured to produce a discrete number of echoes that are precise and independently varied in their intensity and coloration, as determined by attenuation with distance, air absorption filtering, reflection surface absorption filtering, and which are specular relative to features and geometry of the modelled room which in turn determines their encoded DoA. These echoes correspond to early reflections depicted in the impulse response (as shown by the reference 103 in the example impulse responses shown in Fig. 1). The reflection processor is external to and running in parallel with the reverberator. Herein, the echoes produced by the reflection processor are referred to as reflection echoes.

[0071] Late reverberation, in contrast to early reflections, is not considered to be spatially dynamic in that the echoes produced in late reverberation synthesis do not vary with source-listener orientation. Each reverberator in the reverberators can be configured to produce, by way of a feedback network, a decaying stream of many echoes which increase in number (density) while decreasing in intensity (loudness) over time, as characterized by the decay properties of stochastic late reverberation (as shown by reference 107 in the example impulse responses shown in Fig. 1) according to the characteristics of the virtual or physical room the reverberator relates to. This is achieved by the feedback architecture of the reverberator, in which an input audio signal passes through the network, splitting into numerous paths which form “echoes” which are separated in time by independent delay lines, all of which subsequently recirculate through the network, being further divided among the delay lines, subsequently splitting into more echoes with each recirculation through the network, and so on. These echoes can be made to have only a loose correspondence to the geometry of the virtual room so do not represent geometrically precise (specular) reflections, nor do they convey the attenuation characteristics of specific reflection surfaces.

[0072] In some embodiments the system can comprise several reverberators, each of which models the characteristics of a room or Acoustic Environment (AE). The rooms (AEs) can have connections to each other, which means that sound sources in any room can contribute to any reverberator. Also, connected or second order reverberation can be implemented by feeding the output of a reverberator into another reverberator.

[0073] Fig. 2 shows an example reverberator system 200 which could be employed in some embodiments. The reverberator 200 is shown herein employing a feedback network (FN) 258 implemented as a feedback delay network (FDN). As depicted in Fig.2, in some embodiments, each reverberator comprises a pre-delay line z−m205, configured to receive and delay the input audio signal 201.

[0074] The reverberator 200 can furthermore comprise a reverberation ratio control filter GEQratio203 which is configured to receive the pre-delay line output.

[0075] The reverberator 200 furthermore comprises a number / ) of feedback delay lines 251 and corresponding feedback delay line attenuation filters GEQd253. The output signals from GEQd253 are sent to inputs of a feedback matrix A 257. The outputs of the feedback matrix A 257 are sent to D signal combiners (adders) 254 which sum the outputs of the feedback matrix 257 with the output of GEQratio203 to be used as inputs to each of the feedback delay lines z−m251.

[0076] The reverberator 200 thus comprises multiple recirculating delay lines associated with the feedback network 258. The feedback matrix A 257 is used to control the recirculation gain and routing within the network.

[0077] Signals from the output taps 259 of the feedback loops combiners are furthermore routed to D signal multipliers 261 which in turn produce the reverberant audio output signals 210.

[0078] Outputs of delay line attenuation filters GEQd253 are routed directly to a corresponding signal multiplier among the multipliers 261.

[0079] The reverberator 200 furthermore comprises a first threshold crossing determiner 277. The first threshold determiner 277 is configured to receive the input from the pre- delay line 250 and listener position distance from AE 290 and compare the level of the input to a first threshold value.

[0080] The reverberator 200 further contains a second threshold crossing determiner 271 which is configured to receive an input from a combiner (adder) 270, which receives its input from the delay lines 251. The combiner 270 can, for example be considered to generate an ‘intermediate output’ signal associated with the reverberant audio signal to be used by the threshold crossing determiner 271.

[0081] In some embodiments the combiner 270 can be any suitable combiner, for example a uniform or weighted combiner.

[0082] Additionally in some embodiments a listener distance from the AE 290 is input. In some embodiments the outputs of the first threshold crossing determiner 277 and second threshold crossing determiner 271 are routed to a cull determiner 280. The cull determiner is configured to determine or make decisions when the reverberator 200 processing can be stopped. The control or stopping of the reverberator can be implemented by controlling a switch 282, or other suitable means for switching, located for example between the first threshold determiner 277 and the reverberation ratio control filter GEQratio203 which prevents signal flow into the reverberator network 258. The switch 282, (or the suitable means for switching), can in some embodiments be located after the reverberation ratio control filter GEQratio203. The cull determinator 280, in some embodiments, can be further connected via control signals 284 to the delay lines 251 so as to be able to zero (or otherwise squelch or substantially attenuate) the state of the delay lines 251 (in other words insert zero values into them) when the reverberator is stopped.

[0083] The zeroing of the delay lines has the effect that no past signal output is generated by the reverberator when it is re-started, even when the second threshold in threshold crossing determinator 271 is larger than zero.

[0084] The embodiments, by employing two threshold crossing determiners 277 and 271, enable a cull determination such that when the first input crossing determiner 277 indicates that input signal is less than a threshold level, the inspection of the second threshold crossing determiner 271 ensures that the reverberator 200 is run long enough so that all content from the feedback network 258 is reproduced. In embodiments not employing a second threshold crossing determiner 271 configured to inspect the content of the delay lines 251, could result in a situation where the feedback network 258 is stopped while there still is audible content in the delay lines 251, leading to artifacts generated by the cutting of reverberant tails while there is audio signal energy within them.

[0085] In some embodiments the listener distance from AE 290 can be input to the threshold crossing determiners 277 and 271 so that the threshold values (first threshold value in 277 and second threshold value in 271) can be adjusted depending on how ‘far’ the reverberator is from the listener. This makes it possible to adjust the threshold values depending on the listener distance to the AE. The rationale for this is that the farther the reverberator is from the listener, the more the reverberant signal will be attenuated when it is reproduced to the listener, which means that signal levels in the input 250 and delay lines 251 needs to be larger to be audible. Thus, in some embodiments, the threshold values in 277 and 271 can be increased dependent on an increasing listener-to-AE distance. Any suitable relationship can be employed between the threshold values and the listener-to-AE distance, for example a linear relationship between the threshold values and the listener-to-AE distance, a non-linear relationship etc.

[0086] In some embodiments, each reverberator 200 is configured to receive reverberator parameters which comprise a delay length mpre, in samples, for pre-delay line z−m205, coefficients of a reverberation ratio control filter GEQratio203, delay lengths mdfor each of D feedback delay lines z−m251, coefficients for each of D feedback attenuation filters GEQd253, coefficients for the feedback matrix A 257. The reverberator parameters also comprise output channel gains gdwhich are used to configure D signal multipliers 261.

[0087] The reverberator 200 parameters are related to the reverberation characteristics of the acoustic environment or room which the reverberator relates to. In a scene with several rooms there can be several reverberator 200 instances forming a set of the reverberators 200. Each reverberator 200 can have a different set of parameter values if the acoustic environments are different.

[0088] In some embodiments the attenuation filter GEQd is a graphic equalizer (EQ) filter using M biquad HR band filters. In the case of octave-band filtering, M = 10. Thus, the reverberator parameters corresponding to each graphic EQ filter comprise the feedforward and feedback coefficients for 10 biquad IIR filters, the gains for biquad band filters, and the overall gain. The feedback delay line attenuation filters GEQd 253 can be implemented in some embodiments as cascades of second-order section IIR filters and can facilitate controlling the energy decay rate at different frequencies. In some embodiments the coefficients of feedback matrix A 257 form a square lossless matrix and can be hardcoded in software code rather than provided as parameters. The size of the matrix (and the corresponding number of delay lines D) can be adjusted depending on the desired tradeoff between reverberation quality (e.g. modal density, temporal and spatial diffuseness, diffuse onset time) and computational complexity.

[0089] The feedback attenuation filters GEQd253 are designed such that they attenuate the signal by the desired amount with each pass through the FN 258 such that the desired reverberation time ( RT60) is achieved. The reverberation time depends on the reverberation time of the acoustic environment the reverberator models.

[0090] The number of delay lines D can be adjusted depending on quality requirements and the desired tradeoff between reverberation quality and computational complexity. In an embodiment, an efficient implementation with D = 15 delay lines is used. This makes it possible to define the coefficients of the feedback matrix A 257 as proposed by Rocchesso in Maximally Diffusive Yet Efficient Feedback Delay Networks for Artificial Reverberation, IEEE Signal Processing Letters, Vol. 4. No. 9, Sep 1997, in terms of a Galois sequence facilitating efficient implementation.

[0091] The reverberator 200 can thus produce 15 nearly uncorrelated outputs which are subsequently encoded to different spatial directions defined by the directional configuration. The output signals are then reproduced using loudspeakers (or alternatively virtual loudspeakers that are convolved with HRTFs or yet alternatively encoded to ambisonics which is then decoded to a binaural or loudspeaker format) that are positioned in the corresponding spatial directions, and the levels of which are controlled with channel gain coefficients. The resulting reverberant audio signals have acoustical characteristics according to the reverberator parameters 304, namely a desired frequency-dependent rate of decay and level.

[0092] Fig.3 shows an example system or apparatus representing a reverberator processing system 300 suitable for rendering middle- and late-stage reverberation implementing some embodiments and comprising at least one reverberator 200 as shown in Fig.2. The reverberator processing system comprises inputs such as audio signal 201, reverberation configuration specification 302, and directional configuration specification 312.

[0093] The reverberator processing system 300 further comprises a reverberant signal combiner 310 configured to receive reverberant audio signals 210 from reverberators 305 (or the multiple reverberators 200 shown in Fig.2) as an input, along with listener position 900 and room geometry 906.

[0094] The reverberant signal combiner 310 is configured to combine the reverberant audio signals 210 according to the listener position 900 and room geometry 906.

[0095] For example, the reverberant signal combiner 310 is configured to combine (mix or otherwise form) the reverberant audio signals 210 from the reverberator which corresponds to the listener AE based on listener position 900 and room geometry 906 with other reverberant audio signals 210, if such signals are audible to this AE and listener position 900. An example when reverberant audio signals 210 are combined by the reverberant signal combiner 310 is when the listener is transitioning between two environments. In some embodiments the reverberant audio signals 210 from an environment may not be combined to combined reverberant audio signals 316 at all (in other words they may be omitted) if the environment is far from the listener position 900.

[0096] The reverberator processing system 300 further comprises a binaural renderer 309 configured to render reverberant binaural signals 314 with late reverberation that is perceived according to the reverberant characteristics specified in the reverberation configuration specification 302 and directional characteristics specified in the directional configuration specification 312, and combined (formed) by the reverberant signal combiner 310 based on listener position 900 and room geometry 906. The reverberation configuration specification 302 and directional configuration specification 312 can, for example, be obtained from a bitstream or from a listening space description format (LSDF) input to the renderer.

[0097] In some embodiments, the reverberation configuration specification 302 comprises suitable parameters for configuring the reverberator 200. Suitable reverberation configuration specification 302 includes, for example, the reverberation times RT60k) in frequency bands (where k is the frequency band index), reverberant-to-direct ratio RDR(k), pre-delay time tpre, and / or a virtual space geometry specification. Alternative to the RDR, the diffuse-to-source energy ratio (DSR) can be used. Each reverberator 200 in the reverberators 305 receives its own reverberation configuration specification 302 based on the acoustic characteristics of the AE which the reverberator models.

[0098] In some embodiments, the directional configuration specification 312 can indicate directions used to render the reverberation by a suitable rendering scheme that creates a perception of enveloping diffuse reverberation, such as ambisonics or amplitude panning rendering, or simply rendered directly to a surrounding (real or virtual) loudspeaker setup. As an example, the directional configuration may specify a spherical design such as a f-design, Lebedev grid, or other suitable (nearly) uniform spherical layout with D points representing encoding directions (and thus the number of reverberator output channels).

[0099] In some embodiments, the reverberator processing system 300 comprises a reverberator parameter determiner 303 configured to obtain or otherwise receive the reverberation configuration specification 302 and directional configuration specification 312. The reverberator parameter determiner 303 converts these specifications into suitable reverberator parameters 304 for each reverberator 200 within the reverberators 305.

[0100] Fig.4 shows schematically in further detail an example reverberator parameter determiner 303 as shown in Fig.3 according to some embodiments. The example reverberator parameter determiner 303 as described above is configured to generate suitable reverberator parameters 304, such as:

[0101] number of reverberator output channels;

[0102] feedback delay line lengths;

[0103] feedback attenuation filter coefficients;

[0104] reverberation ratio control filter coefficients; and

[0105] pre-delay line length.

[0106] In some embodiments the reverberator parameter determiner 303 comprises a feedback delay lengths determiner 401 configured to determine the feedback delay line length parameter values md. The feedback delay lengths mdmay be based on a virtual space (Acoustic Environment) geometry specification. For example, a bounding box that encloses or is aligned with the walls of the physical or virtual room can be defined with dimensions xDim,yDim,zDim. Alternatively, the dimensions can be obtained as three longest orthogonal dimensions in the non-shoebox shaped room, or by a mesh if the bounding box is provided as a mesh, or by another suitable method.

[0107] In some embodiments when the method is executed in a renderer then the enclosure vertices can be obtained from the bitstream (for VR acoustic environments) or the LSDF (for an AR acoustic environment) and the dimensions can be calculated.

[0108] The feedback delay lengths mdcan in some embodiments be set proportionally to standing wave resonance frequencies in the virtual room or physical room (the acoustic environment). The dimensions can further be converted to modified dimensions of a virtual room or enclosure by predetermined ratios which are suited for the generation of preferable room modes.

[0109] The feedback delay line lengths determiner 401 may set the delay line lengths mdto be made to be mutually prime integers. This choice minimizes coherent repetition in the impulse response of the FN. The sieve of the Sundaram algorithm can be used to find the prime numbers up to the maximum delay line length and each delay line length can then be mapped to the closest prime number in the obtained set of prime numbers.

[0110] In some embodiments the reverberator parameter determiner 303 further comprises a feedback attenuation filter parameter determiner 403 which is configured to determine attenuation filter coefficients for feedback attenuation filters GEQd253. The filter coefficients can be configured so that the rate of attenuation produced by the recirculation through the delay lines results in the desired reverberation time / ? T60(fc). This is done in a frequency-dependent manner to ensure the appropriate rate of decay of signal energy at specified frequencies. For a frequency bin k, the desired attenuation per signal sample is γsamp(k) = −60 / (fs* RT60(k)) dB, where fsis the sampling rate. The attenuation in decibels for a delay line pair of aggregate length md, where md= md,a+ md,b, is then

[0111]

[0112] which serves as a target command gain in the design procedure of cascade graphic equalizer filters as described in V. Välimäki and J. Liski, “Accurate cascade graphic equalizer,” IEEE Signal Process. Lett., vol. 24, no. 2, pp. 176-180, Feb. 2017, to produce the attenuation filter coefficients for GEQd. The cited design procedure operates in octave bands, although methods for similar graphic EQ structures can support third octave bands, increasing the number of biquad filters to 31 and providing a better match for detailed target responses, such as detailed in J. Rämö, J. Liski, and V. Välimäki, “Third-Octave and Bark Graphic-Equalizer Design with Symmetric Band Filters,” Applied Sciences, vol. 10, no. 4, p. 1222, Feb. 2020.

[0113] In some embodiments the reverberator parameter determiner 303 further comprises a pre-delay line length determiner 406. The pre-delay line length determiner 406 is configured to determine a predelay length mprein samples based on tprethat denotes an onset timing of the diffuse state of the reverberator, in which case mprecan be set such that the diffuse onset timing of the FN matches a desired tpre. The diffuse onset timing of the FN can be estimated by any mixing time or diffuseness estimator, or predicted from analytic methods using the virtual space geometry specification provided in the reverberation configuration specification 302.

[0114] In some embodiments the reverberator parameter determiner 303 further comprises reverberation ratio control filter parameter determiner 409. In some embodiments, the reverberation ratio control filter parameter determiner 409 can be configured to determine reverberation ratio control filter parameters GSQratio such that, when the filter GEQratio203 is applied to the input signal 201, the resultant reverberation will have the desired energy ratio defined by the RDR(k). The input to the design procedure can in some embodiments be the vector of reverberant-to-direct (RDR) energy ratio values RDR(k) obtained by the reverberation configuration specification 302. The generated coefficients of GEQratiois designed to match the reverberator spectrum energy to the target spectrum energy. To do this, an estimate of the RDR of the reverberator output is determined by the following procedure.

[0115] The RDR of the reverberator output can be obtained by first rendering a unit impulse through the reverberator that has been configured with the parameters produced by the delay line lengths determiner 401, feedback attenuation filter parameters determiner 403, and tpreset to 0. The input to the reverberator is a buffer of zeros of a sufficient length to capture the reverberation tail, such as the maximum RT60(k) among all frequency bands, with a unit impulse written to the head of the buffer. Once rendered, the energy of the reverberator output is measured, along with that of the unit impulse, and the ratio of these energies is calculated. The procedure for measuring signal energy is detailed in the following.

[0116] The monophonic output signal srev(t), which is a function of time t, can be obtained by summation of the outputs of the feedback network 250. A FFT of length NFFTis calculated over srev(t) and its magnitude spectrum can be obtained as

[0117]

[0118] abs(FFT(srev(t))).

[0119] Here, kbare the FFT bin indices. The positive half spectral energy density is

[0120]

[0121] where the energy from the negative frequency indices kbis added into the corresponding positive frequency indices kb. The energy of a unit impulse can be calculated or obtained analytical ly and is denoted as Sunit( / cb). In some embodiments the energy of each band at index k are calculated as the positive half spectral energy density of the reverberator Srev(fcb) and the positive half spectral energy density of the unit impulse Sunit(fcd). Band energies can be calculated as

[0122]

[0123] where btowand bhighare the lowest and highest bin indices belonging to band k, respectively. The band bin indices can be obtained by comparing the frequencies of the bins to the lower and upper frequencies of each band.

[0124] The reproduced RDR of the reverberator is obtained as

[0125] RDRrev(k) = Srev(k) / Sunit(k)

[0126] The target linear magnitude response for GEQraiiocan be obtained as gGEQ= sqrt(RDR(k)) / sqrt(RDRrev(k))

[0127] where RDR(k) is the target linear RDR value from reverberation configuration specification 302. The target response control gain is then

[0128]

[0129] The RDR target response control gain can also be obtained directly in the logarithmic domain as

[0130]

[0131] γGEQ(k) is then provided to the graphic equalizer design routine, previously described above with respect to the feedback attenuation filter parameter determ iner 403, to produce filter coefficients for GEQratio.

[0132] Fig.6 shows schematically the binaural renderer 309 as shown in Fig.3 in further detail. The input to the binaural renderer 309 is the combined reverberant audio signals 316d srev(t, d) and the directional configuration 312 indicating encoding directions for each combined reverberant audio signal. In the example shown in Fig.6 the binaural renderer 309 is organized on a channel-by-channel basis and there is one HRTF processor 701 d per reverberant audio channel. For example, a first channel a HRTF processor 7011 is configured to receive the combined reverberant audio signal 3161 (channel one) and the directional configuration 312i associated with channel one. A second channel a HRTF processor 7012 is configured to receive the combined reverberant audio signal 3162 (channel two) and the directional configuration 3122 associated with channel two. Also shown is a Dth channel HRTF processor 701 D configured to receive the combined reverberant audio signal 316D (channel D) and the directional configuration 312D associated with channel D. Each of the HRTF processors can comprise an HRTF filter pair hbin(m, i, d), where m is the time index of the filter coefficients, i - 1, 2 is the index of the binaural channel, and d is the reverberator output channel index.

[0133] The operation of the dth HRTF processor 701 d is as follows. Using the HRTF filter pairs hbin(m,i,d'), reverberant binaural audio signals sbin(t,i,d') 702d can be determined for each channel of the combined reverberant audio signals 316d by

[0134] $bin (T L ^0 hbin. > I, C0 ® comb > ^0

[0135] where ® denotes convolution (the filtering may also be performed in the frequency domain in some implementations instead of time-domain convolution) and scomb(t, d) is the combined reverberant signal.

[0136] The reverberant binaural audio signals sbin(t, i, d) 702a can then be passed to a binaural signal combiner 703.

[0137] The reverberant binaural audio signals sbln(t, i, d~) 702d can then be combined across channels d in the binaural signal combiner 703 by

[0138]

[0139] yielding the reverberant binaural signals

[0140]

[0141] 314 which is the output.

[0142] Fig.5 shows an example flow diagram of the operations of the Tenderer system shown in Fig.3 with respect to the reverberator and the associated binaural Tenderer.

[0143] First, the audio signal 201, reverberation configuration specification 302, and directional configuration specification 312 are obtained as shown by 601.

[0144] Then, the reverberator parameters 304 are determined from the reverberation configuration specification 302 and directional configuration specification 312 inputs as shown by 603.

[0145] Then, the reverberator 200 is configured using reverberator parameters 304 as shown by 605.

[0146] Then, the reverberant signal combiner 310 is configured using Listener position 900 and Room geometry 906 as shown by 607.

[0147] Then, the binaural renderer 309 is configured using the directional configuration specification 312 as shown by 609.

[0148] Then, reverberant audio signals 210 are generated by processing the audio signal with the configured reverberator 200 as shown by 611. Then, reverberant binaural signals 314 are rendered by processing the reverberated audio signals 210 with the configured reverberant signal combiner 310 and configured binaural renderer 309 as shown by 613.

[0149] Then, reverberant binaural signals 314 are output as shown by 615.

[0150] Fig.9 shows an example flow diagram of the operations of the system shown in Fig.3 an in particular the reverberator 200 in Fig.2 according to some embodiments in more detail.

[0151] First, in case there are several predelay lines the audio signal for the reverberator is downmixed across predelay lines as shown by 1001. Thus, referring to Fig.2, there can be several predelay lines 250 each with a different predelay mpre, and in this case a summation or other suitable combination is performed after the predelay lines 250 to create predelayed input signal. In some embodiments the downmixing can be a selection of one or more of the predelay lines 250 and a combination of the selection (where the selection is more than one) to generate (or create or determine) the predelayed input signal. In some embodiments an average, or mean value for the predelayed input signal is determined.

[0152] Then, there is a determination whether the input audio signal crosses a first threshold level by the first threshold determiner 277 as shown by 1003.

[0153] Then, there is determination whether the summed signal of delay lines (generated by the combiner 270 as shown in Fig.2) crosses a second threshold level by the second threshold determiner 271 as shown by 1005.

[0154] Then, as shown by 1007, if at least either determination is true as assessed by the cull determiner 280, (in other words as least one of the thresholds are not met or crossed) the audio signal is processed with the configured reverberator to produce reverberant audio signals as described in the operations of Fig.5.

[0155] Then, as shown by 1009, the reverberant signals are output.

[0156] Or, as shown by 1007, if both determinations are false as determined by the cull determiner 280, (in other words both of the thresholds are met or crossed) then there is performed the operation of zeroing (or substantially or otherwise supressing or squelching) the contents of the delay lines and therefore not producing reverberant audio signals.

[0157] In some embodiments, a first threshold level employed by the first threshold determiner 277 and a second threshold level employed by the second threshold determiner 271 is determined to be equal to 1e-6 * dist_to_AE, where dist_to_AE equals distance of the listener to the AE of this reverberator (measured in metres).

[0158] The first and second threshold values in some embodiments are not equal values (in other words the first and second threshold values are different values). In some embodiments the multiplying of the threshold with the distances causes a larger threshold to be used for reverberators which are further away from the listener, which enables culling them at already larger signal levels than reverberators closer to the listener. This is beneficial as the signals generated reverberators which are farther away will be reproduced at lesser level to the listener by the reverberant signal combiner 310 of Fig. 3 and can thus culled earlier. An example operation performed by the Reverberant signal combiner 310 can be to apply slight distance gain attenuation to a reverberant signal of an environment when the listener distance from that environment increases. As another example, the reverberant sound of an environment may be reproduced from an acoustic portal opening by the Reverberant signal combiner, and the level of the reverberant sound from the portal opening is subjected to distance gain attenuation as the listener distance from it increases.

[0159] The computational complexity of the system can be further reduced by setting the threshold level higher, such as using a higher constant value multiplied by the distance, for example using a constant of 1e-3. The higher threshold value can lead to earlier culling. In some embodiments the threshold can be obtained from the obtained bitstream based on a content creator parameter.

[0160] The method in some embodiments can have an initially fixed or determined first and second threshold which are later adjusted based on listening to different values and whether the effect of the culled reverberator would be audible. The thresholds in some embodiments can be adjusted to fixed values at which culling is not audible with a certain set of test signals.

[0161] In some embodiments, as discussed above, there is determined or obtained a dynamically adjusted first or second threshold (level or levels), where the adjustment can depend on the level of other signals in the system (for example the direct sound, early reflections). In such a manner the threshold level or levels can be adaptively increased as the level of other sounds increases (or decreased as the level of other sounds decreases).

[0162] This dynamic adjustment can involve calculating or otherwise determining the level of early reflection and direct sound levels, and then comparing the level of the reverberator input signal and / or output signal (content of the delay lines) to the level of the other signals. When it is determined with sufficient confidence that the other signal levels mask the reverberant signal levels, then reverberation culling can be enabled.

[0163] In some embodiments other signal levels can impact the first threshold and / or second threshold (level or levels) such that the thresholds are increased in value as the level of other signals increases, and correspondingly decreased in value as the level of other signals decreases.

[0164] In some embodiments the threshold level or levels and / or therefore the culling control or masking determination can be frequency dependent.

[0165] A schematic view of an example virtual audio scene rendering system is depicted in Fig.7. The virtual audio scene rendering system 800 can comprise a direct sound processor 861 configured to receive the audio signal 820 (for example 820i to 820n) and generate direct audio signal 860 which are passed to a direct audio binaural Tenderer 869. The direct sound processor 861 renders the sound that directly reaches the listener without reflecting or reverberating (this is shown for example by the reference 101 in Fig.

[0166] 1 ). The direct sound processor 861 can be configured to apply distance gain attenuation (e.g. attenuation proportional to 1 / r where r is the distance from the sound source to the listener) and air absorption filtering (which is a distance-dependent low-pass filter attenuating high frequencies).

[0167] The virtual audio scene rendering system 800 can furthermore comprise a direct audio binaural Tenderer 869 configured to receive the direct audio signals 860 and generate direct audio binaural audio signals 864 which are output.

[0168] The virtual audio scene rendering system 800 can comprise a reflection processor 851. With respect to the reverberator and the reflection processor these are configured to generate audio signals associated with echoes within the system. For example, the reflection processor is configured to produce a discrete number of echoes which are specular with regard to features and geometry of the modelled room and are correspondingly precise and independently varied in their arrival direction, intensity, and coloration, as characterizes early reflections in a room impulse response (this is shown for example by the reference 103 in Fig. 1). The echoes produced by the reflection processor are accordingly referred to as reflection echoes. The reflection processor is external to and running in parallel with the reverberator which is configured to produce late reverberation. The example reflection processor 851 is configured to receive the audio signal 820 (for example 820i to 820n) and generate reflection audio signal 850 which are passed to a reflection binaural Tenderer 859 to generate the reflection binaural audio signals 854 along with the reverberant binaural audio signals 314 described in the embodiments as discussed herein.

[0169] The virtual audio scene rendering system 800 can furthermore comprise the Tenderer system 300 shown in Fig.3 wherein the audio signal 201 input is the combination 870 of the audio signals from each source shown as audio signal 820i and 820nand the audio signal output is the reverberant binaural audio signals 314.

[0170] Fig.8, furthermore shows in further detail an example reflection processor 851 and the associated reflection binaural Tenderer 859 suitable for using along with the embodiments as discussed herein.

[0171] The Fig.8 example shows an example implementation and it would be understood that there are several ways to calculate or simulate early reflections which could be employed otherwise. For example an image source method can be employed such as detailed in J. B. Allen and D. A. Berkley, “Image method for efficiently simulating smallroom acoustic,” J. Acoust. Soc. Am., vol. 65, pp. 943-950, April 1979 and J. Borish. “Extension of the image model to arbitrary polyhedra.” The Journal of the Acoustical Society of America 75.6 (1984): 1827-1836.

[0172] In the example shown in Fig.8, a reflection parameter determiner 901 is configured to receive the inputs of room geometry 906, listener position 900, source position 902, and absorption coefficients 904 and generate control parameters such as delay 906, absorption 908, attenuation 910 and direction of arrival (DoA) 912 and pass these to the processors described hereafter.

[0173] In some embodiments the input audio signal 201 is first fed into a delay line 903 which buffers audio signal samples and enables picking segments of past samples of the audio signal 201.

[0174] The reflection signal obtainer 905 can receive the output of the delay line 903 and the delay 906 parameter. The reflection signal obtainer is configured to obtain a past signal sample based on the delay 906 to obtain a delayed signal.

[0175] A reflection absorption processor 907 then can filter the selected past signal sample to apply an equalizer filter to model the frequency-dependent absorption data for the reflection to obtain delayed and absorption-filtered signal. A reflection attenuation processor 909 can then attenuate the delayed and absorption-filtered signal by applying a l / r attenuation and optionally air absorption to obtain delayed and absorption-filtered and attenuated signal.

[0176] Finally, a reflection spatializer 911 can be configured to spatialize the delayed and absorption-filtered and attenuated signal by HRTF filtering with a left and right HRTF filter corresponding to the desired DoA for this reflection to obtain a reverberant binaural signal 912 containing the synthesized reflection portion. In some situations, the reflection spatializer can be the binauralizer.

[0177] There are various ways to determine the image source parameters within the reflection parameter determiner 901. In the image source method, the sound source position is mirrored with respect to each reflecting surface of the room geometry to obtain image sources.

[0178] In some circumstances the output of the determiner is a list of image source positions such

[0179]

[0180] the coordinates an image source that in each order of reflection has been reflected by the ith subsequent surface.

[0181] Fig.10 shows schematically an example system where the embodiments are implemented in an encoder device 1101 which performs part of the functionality; writes data into a bitstream 1121 and transmits that for a Tenderer device 1141, which decodes the bitstream, performs reverberator processing according to the embodiments and outputs audio for headphone listening.

[0182] The encoder side 1101 of Fig.10 can be performed on content creator computers and / or network server computers. The output of the encoder is the bitstream 1121 which is made available for downloading or streaming. The decoder / renderer 1141 functionality runs on an end-user-device, which can be a mobile device, personal computer, sound bar, tablet computer, car media system, home HiFi or theatre system, head mounted display for AR or VR, smart watch, or any suitable system for audio consumption.

[0183] The encoder 1101 is configured to receive the virtual scene description 1100 and the audio signals 1904. The virtual scene description 1100 can be provided in the MPEG-I encoder input format (EIF) or in another suitable format. Generally, the virtual scene description contains an acoustically relevant description of the contents of the virtual scene, and contains, for example, the scene geometry as a mesh or as voxels, acoustic materials, acoustic environments with reverberation parameters, positions of sound sources, and other audio element related parameters such as whether reverberation is to be rendered for an audio element or not. The encoder 1101 in some embodiments comprises a scene and reverberation payload encoder 1113 configured to generate reverberation parameters.

[0184] The encoder 1101 further comprises a MPEG-H 3D audio encoder 1114 configured to obtain the audio signals 1904 and MPEG-H encode them and pass them to a bitstream encoder 1115.

[0185] The encoder 1101 furthermore in some embodiments comprises a bitstream encoder 1115 which is configured to receive the output of the scene and reverberation payload encoder 1113 and the encoded audio signals from the MPEG-H encoder 1114 and generate the bitstream 1121 which can be passed to the bitstream decoder 1141. The bitstream 1121 in some embodiments can be streamed to end-user devices or made available for download or stored.

[0186] The decoder 1141 in some embodiments comprises a bitstream decoder 1141 configured to decode the bitstream.

[0187] The decoder 1141 further can comprise a scene payload decoder 1143 configured to obtain the encoded reverberation parameters and decode these in an opposite or inverse operation to the reverberation payload encoder 1113.

[0188] The reverberator parameter determiner 303 / 1142 is configured to receive the decoded reverberation configuration specification and room dimensions and spatial room impulse response (SRIR) 1140 information and generate the reverberator control parameters discussed herein. Note that in some embodiments no SRIR is received but reverberator parameters are obtained from the scene payload decoder 1143.

[0189] Furthermore, the head pose generator 1147 receives information from a head mounted device 1170 or similar and generates head pose information or parameters which can be passed to the binaural renderer 309 / 1159, the early reflection Tenderer 990 / 1162 and the direct sound binaural Tenderer 1163.

[0190] The decoder 1141 comprise MPEG-H 3D audio decoder 1144 which is configured to decode the audio signals and pass them to the reverberators 201 / 1161 and direct sound processing 1165.

[0191] The decoder 1141 furthermore comprises reverberators 201 / 1161 configured to implement a suitable reverberation of the audio signals from the MPEG-H 3D audio decoder 1144.

[0192] The output of the reverberator 201 / 1161 is configured to output reverberated audio based on the reverberator parameters to a binaural renderer 309 / 1159. The decoder furthermore comprises an early reflection Tenderer 990 / 1162 configured to obtain the output of the MPEG-H 3D audio decoder 1144 and generate early reflections as described above and pass these to an early reflection binaural Tenderer 1199.

[0193] The decoder further comprises a binaural renderer 309 / 1159 configured to generate binaural reverberant audio signals from the output of the reverberators 201 / 1161.

[0194] The decoder further comprises an early reflection (ER) binaural Tenderer 1199 configured to generate binaural early reflection audio signals from the output of the early reflection Tenderer 990 / 1162.

[0195] Additionally, the decoder / renderer 1141 comprises a direct sound processor 1165 which is configured to receive the decoded audio signals and configured to implement any direct sound processing such as air absorption and distance-gain attenuation and which can be passed to a direct sound binaural Tenderer 1163 which with the head orientation determination (from a suitable sensor) can generate the direct sound component which with the reverberant component is passed to a binaural signal combiner 1167. The binaural signal combiner 1167 is configured to combine the direct, early reflection, and reverberant parts to generate a suitable output (for example for headphone reproduction).

[0196] Furthermore, in some embodiments the decoder comprises a head orientation determiner which passes the head orientation information to the head pose generator 1147.

[0197] As an alternative to transmitting reverberation parameters from the encoder to the Tenderer it is possible in some embodiments to transmit reverberator parameters in the bitstream. Reverberator parameters refer to the FDN parameters such as delay line lengths, attenuation filters, reverberation ratio control filters, and so on.

[0198] In some embodiments the assignment of reverberator outputs to loudspeaker channels happens during configuration of the reverberator. The assignment can be stored during configuration and provided to the reverberant signal router.

[0199] In some embodiments, the output is a multichannel loudspeaker setup (such as 5.1 or 7.1+4 multichannel loudspeaker setup). In that case, the spatial processing proposed in Fig.8 can be modified by using the directions of the actual loudspeakers as the directional configuration and omitting the binaural Tenderers, and reproducing the reverberant audio signals from the corresponding loudspeakers of the loudspeaker setup. In the case of loudspeaker output, instead of binaural renderer 309 / 1159 in Fig.10 there will be a loudspeaker Tenderer (or panner) which in the simplest case will pass through the output signals to a loudspeaker signal combiner which will replace the binaural signal combiner 1167. Correspondingly, the direct sound part and early reflection part are spatialized with a panner such as vector-base amplitude panning instead of the binaural processors.

[0200] With respect to Fig.11 an example electronic device which may be used as any of the apparatus parts of the system as described above. The device may be any suitable electronics device or apparatus. For example, in some embodiments the device 2000 is a mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc. The device may for example be configured to implement the encoder or the Tenderer or any functional block as described above.

[0201] In some embodiments the device 2000 comprises at least one processor or central processing unit 2007. The processor 2007 can be configured to execute various program codes such as the methods described herein.

[0202] In some embodiments the device 2000 comprises a memory 2011. In some embodiments the at least one processor 2007 is coupled to the memory 2011. The memory 2011 can be any suitable storage means. In some embodiments the memory 2011 comprises a program code section for storing program codes implementable upon the processor 2007. Furthermore, in some embodiments the memory 2011 can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor 2007 whenever needed via the memory-processor coupling.

[0203] In some embodiments the device 2000 comprises a user interface 2005. The user interface 2005 can be coupled in some embodiments to the processor 2007. In some embodiments the processor 2007 can control the operation of the user interface 2005 and receive inputs from the user interface 2005. In some embodiments the user interface 2005 can enable a user to input commands to the device 2000, for example via a keypad. In some embodiments the user interface 2005 can enable the user to obtain information from the device 2000. For example, the user interface 2005 may comprise a display configured to display information from the device 2000 to the user. The user interface 2005 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device 2000 and further displaying information to the user of the device 2000. In some embodiments the user interface 2005 may be the user interface for communicating.

[0204] In some embodiments the device 2000 comprises an input / output port 2009. The input / output port 2009 in some embodiments comprises a transceiver. The transceiver in such embodiments can be coupled to the processor 2007 and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver or any suitable transceiver or transmitter and / or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.

[0205] The transceiver can communicate with further apparatus by any suitable known communications protocol. For example, in some embodiments the transceiver can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802. X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).

[0206] The input / output port 2009 may be configured to receive the signals.

[0207] In some embodiments the device 2000 may be employed as at least part of the renderer. The input / output port 2009 may be coupled to headphones (which may be a headtracked or a non-tracked headphones) or similar.

[0208] In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0209] The embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.

[0210] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.

[0211] Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0212] Programs, such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or “fab” for fabrication.

[0213] The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention as defined in the appended claims.

Claims

CLAIMS:

1. An apparatus for spatial rendering of reverberation, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the system at least to perform:obtaining at least one audio signal;configuring at least one reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal;determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level;obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; andcontrolling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

2. The apparatus as claimed in claim 1, wherein obtaining the at least one threshold causes the apparatus to perform:obtaining at least one first threshold associated with the at least one reverberation input signal level; andobtaining at least one further threshold associated with the at least one reverberation delay line signal level, wherein the apparatus caused to perform controlling the at least one reverberator is further caused to perform controlling the at least one reverberator at least based on:a comparison between the at least one reverberation input signal level against the at least one first threshold; anda comparison between the at least one reverberator delay line signal level against the at least one further threshold.

3. The apparatus as claimed in any of claim 1 or 2, wherein the at least one reverberator comprises at least two delay lines, and wherein the apparatus caused toperform determining the at least one reverberation delay line signal level is further caused to perform:obtaining outputs from the at least two delay lines; andgenerating at least one reverberation delay line signal level based on a combination of the outputs from the at least two delay lines.

4. The apparatus as claimed in any of claims 1 to 3, caused to perform obtaining the at least one audio signal is further caused to perform obtaining at least two audio signals, and the apparatus caused to perform determining the at least one reverberation input signal level is caused to perform combining the at least two audio signals.

5. The apparatus as claimed in claim 2 or any claim dependent on claim 2, wherein the at least one first threshold is equal to the at least one further threshold.

6. The apparatus as claimed in any of claims 1 to 5, caused to perform obtaining the at least one threshold is caused to perform:obtaining a listener distance, the listener distance defining a distance between a listener and an audio environment; andgenerating the at least one threshold based on the listener distance.

7. The apparatus as claimed in any of claims 1 to 6, caused to perform controlling the at least one reverberator is caused to perform at least one of:switching on or off the at least one reverberator based on the comparison; culling the at least one reverberator based on the comparison;reinstating the at least one reverberator based on the comparison; and attenuating at least one output of the at least one reverberator based on the comparison.

8. The apparatus as claimed in any of claims 1 to 7, wherein the at least one reverberator is configured to produce late reverberation.

9. The apparatus as claimed in any of claims 1 to 8, wherein the at least one reverberator is a digital reverberator.

10. The apparatus as claimed in any of claims 1 to 9, caused to perform configuring the at least one reverberator based on at least one reverberation parameter is further caused to perform at least one of:determining a number of reverberator output channels;determining feedback delay line lengths;determining feedback attenuation filter coefficients;determining reverberation ratio control filter coefficients; anddetermining a pre-delay line length.

11. A method for spatial rendering of reverberation, the method comprising:obtaining at least one audio signal;configuring at least one reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal;determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level;obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; andcontrolling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

12. The method as claimed in claim 11, wherein obtaining the at least one threshold further comprises:obtaining at least one first threshold associated with the at least one reverberation input signal level; andobtaining at least one further threshold associated with the at least one reverberation delay line signal level, wherein controlling the at least one reverberator further comprises controlling the at least one reverberator at least based on:a comparison between the at least one reverberation input signal level against the at least one first threshold; anda comparison between the at least one reverberator delay line signal level against the at least one further threshold.

13. The method as claimed in any of claims 11 or 12, wherein the at least one reverberator comprises at least two delay lines, and wherein determining at least one reverberation delay line signal level further comprises:obtaining outputs from the at least two delay lines; andgenerating at least one reverberation delay line signal level based on a combination of the outputs from the at least two delay lines.

14. The method as claimed in any of claims 11 to 13, wherein obtaining the at least one audio signal further comprises obtaining at least two audio signals, and determining the at least one reverberation input signal level comprises combining the at least two audio signals.

15. The method as claimed in claim 12 or any claim dependent on claim 12, wherein the at least one first threshold is equal to the at least one further threshold.

16. The method as claimed in any of claims 11 to 15, wherein obtaining the at least one threshold further comprises:obtaining a listener distance, the listener distance defining a distance between a listener and an audio environment; andgenerating the at least one threshold based on the listener distance.

17. The method as claimed in any of claims 11 to 16, wherein controlling the at least one reverberator further comprises at least one of:switching on or off the at least one reverberator based on the comparison; culling the at least one reverberator based on the comparison;reinstating the at least one reverberator based on the comparison; and attenuating at least one output of the at least one reverberator based on the comparison.

18. The method as claimed in any of claims 11 to 17, wherein the at least one reverberator is configured to produce late reverberation.

19. The method as claimed in any of claims 11 to 18, wherein the at least one reverberator is a digital reverberator.

20. The method as claimed in any of claims 11 to 19, wherein configuring the at least one reverberator based on at least one reverberation parameter comprises at least one of:determining a number of reverberator output channels;determining feedback delay line lengths;determining feedback attenuation filter coefficients;determining reverberation ratio control filter coefficients; anddetermining a pre-delay line length.

21. An apparatus for spatial rendering of reverberation, the apparatus comprising means configured to:obtain at least one audio signal;configure at least one digital reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal;determine at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level;obtain at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; andcontrol the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

22. A computer readable medium comprising instructions for causing an apparatus, for spatial rendering of reverberation, at least to perform:obtaining at least one audio signal;configuring at least one reverberator based on at least one reverberation parameter, the at least one reverberator for generating at least one reverberant audio signal based on the at least one audio signal;determining at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level;obtaining at least one threshold associated with at least one of: the at least one reverberation input signal level; and the at least one reverberation delay line signal level; andcontrolling the at least one reverberator at least based on a comparison between the at least one threshold and the at least one of: at least one reverberation input signal level; and at least one reverberation delay line signal level.

Images