Communication apparatus and signal generation method

Abstract

The present invention improves the signal quality of a virtual stereo microphone. This communication apparatus comprises a reception circuit for receiving encoded information, which is an encoded signal from microphones that constitute a stereo microphone, and a generation circuit for generating a signal of a virtual stereo microphone by performing an inverse Fourier transform and overlap-add processing on a phase spectrum which is obtained by multiplying, by a synthesis window, a signal that is generated by changing the phase difference between the channels in a decoded stereo signal obtained by decoding the encoded information, and on an amplitude spectrum which is obtained by multiplying the decoded stereo signal by the synthesis window without changing the phase difference.

Description

Communication device and signal generation method

[0001] This disclosure relates to a communication device and a signal generation method.

[0002] For example, there is a method that uses extrapolation to extend the distance between microphones to the distance between both ears and generates a binaural signal using a virtual microphone (hereinafter also referred to as "virtual microphone") (see, for example, Non-Patent Document 1).

[0003] Ryohei Jinzai et al., "Virtual Extension of Microphone Spacing by Extrapolation of Virtual Microphones," *Onkoron* 1-1-21, pp. 149-152, September 2018.

[0004] There is room for further investigation into methods for generating binaural signals using virtual microphones.

[0005] Non-limiting embodiments of this disclosure contribute to the provision of a communication device and a signal generation method that can improve the quality of binaural signals using a virtual microphone.

[0006] A communication device according to one embodiment of the present disclosure comprises a receiving circuit that receives encoded information obtained by encoding a signal from a stereo microphone, and a generating circuit that generates a virtual stereo microphone signal by performing an inverse Fourier transform and overlap-add processing on a phase spectrum obtained by applying a composite window to a signal generated by changing the phase difference between channels of a decoded stereo signal obtained by decoding the encoded information, and an amplitude spectrum obtained by applying a composite window to the decoded stereo signal without changing the phase difference.

[0007] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0008] According to one embodiment of this disclosure, the quality of binaural signals using a virtual microphone can be improved.

[0009] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features.

[0010] Diagram illustrating the principle of generating a virtual microphone Diagram illustrating the principle of generating a pseudo-binaural signal Block diagram showing an example configuration of a communication terminal Flowchart of the pseudo-binaural signal generation process Diagram illustrating the principle of determining the azimuth angle from the inter-channel phase difference and the distance between microphones Block diagram showing an example of a transmitting terminal Block diagram showing an example of a receiving terminal Block diagram showing an example of the configuration of the inter-channel phase difference changing unit Block diagram showing an example of the configuration of the inter-channel amplitude difference changing unit Flowchart of the pseudo-binaural signal generation process Flowchart of the amplitude generation process Block diagram showing an example of the configuration of the inter-channel phase difference changing unit and the inter-channel amplitude difference changing unit Flowchart of the amplitude generation process Block diagram showing an example of the configuration of the inter-channel phase difference changing unit and the inter-channel amplitude difference changing unit Block diagram showing an example of the configuration of the inter-channel phase difference changing unit and the inter-channel amplitude difference changing unit Block diagram showing an example of a transmitting terminal Block diagram showing an example of a receiving terminal

[0011] Embodiments of this disclosure will be described in detail below with reference to the drawings.

[0012] The method described in Non-Patent Document 1 applies an approximation (for example, a plane wave assumption) that assumes (or is assumed to be) that the sound source is sufficiently far away. Therefore, when applying this method to a nearby sound source (for example, a spherical wave), the error due to the plane wave assumption tends to be large.

[0013] In contrast, in one non-limiting embodiment of the present disclosure, for example, by performing virtual microphone extrapolation (e.g., arrangement of virtual microphones) on both sides of the actual microphone (hereinafter also referred to as "real microphone") instead of on one side of the actual microphone, the communication device (e.g., also referred to as a communication terminal, binaural communication terminal, or binaural audio communication device) becomes capable of generating virtual binaural microphone signals having a binaural phase difference. This reduces errors due to the approximation of the plane wave assumption (e.g., plane wave sound waves), even in the case of a near-field sound source (e.g., spherical wave sound waves).

[0014] Figure 1 shows an example of a method for generating a virtual microphone as described in Non-Patent Document 1.

[0015] In Figure 1, "x1" and "x2" represent real microphones, and "v" represents a virtual microphone. In the example in Figure 1, if the distance between x1 and x2 (also called the "microphone distance") is 1, then the distance between x1 and v is α. The value of α represents the degree or direction of extrapolation and will be called the "extrapolation coefficient" below.

[0016] The observed signal v(ω,t,α) in the virtual microphone shown in Figure 1 is expressed by the following equation (where ω is each frequency, t is time, and α is the extrapolation coefficient): v(ω, t, α) = A v food(jφ v ) (1)

[0017] As shown in Figure 1, the virtual microphone v generated by extrapolation (or interpolation) exists outside the real microphones x1-x2. For example, if the virtual microphone v is outside x2, then α > 1, and if the virtual microphone v is outside x1, then α < 0.

[0018] Here, A v =A1 (if α>1) or A v =A2 (when α<0), φ v =φ1+α(φ2-φ1)=(1-α)φ1+αφ2. Here, A1 and φ1 represent the amplitude and phase of the observed signal at microphone x1, respectively, and A2 and φ2 represent the amplitude and phase of the observed signal at microphone x2, respectively.

[0019] FIG. 2 shows an example of a method of arranging virtual microphones on both sides of real microphones (e.g., symmetrically arranged), instead of the method of arranging a virtual microphone on one side of real microphones as shown in FIG. 1. In FIG. 2, "L" and "R" each represent a real microphone, and "Lv" and "Rv" each represent a virtual microphone.

[0020] For example, when a sound source is located sufficiently far away and sound waves can be regarded as plane waves, there is no essential difference regardless of which of the virtual microphone arrangements in FIG. 1 and FIG. 2 is used. On the other hand, when a sound source exists at a short distance and sound waves are considered as spherical waves, the method of arranging a virtual microphone on one side of real microphones shown in FIG. 1 may cause the error of plane wave approximation with respect to actual sound waves to be non-uniform and large. In contrast, by arranging virtual microphones Lv and Rv symmetrically on both sides of real microphones L-R as shown in FIG. 2, even when sound waves are considered as spherical waves such as in the case of a short-distance sound source, compared with FIG. 1, the error of plane wave approximation with respect to actual sound waves (spherical waves) can be dispersed on both sides of the real microphones, so it is possible to make the error of plane wave approximation uniform and small (e.g., level equalization). In addition, in FIG. 2, it is sufficient to consider the case where α>1, and it is not necessary to consider the case where α≦1.

[0021] In FIG. 2, if the inter-microphone distance between real microphones L-R is defined as "r", the inter-microphone distance between virtual microphones Lv-Rv is represented by "α×r". At this time, the phase component φ for each time-frequency of the observation signals at the virtual microphones Rv and Lv Rv (t, f) and φ Lv (t, f) are expressed as follows in the same manner as in the case of FIG. 1. φ Rv (t,f)=φ L (t,f)+0.5×(α+1)×φ d1 (t,f) φ d1 (t,f)=φ R (t,f)-φ L (t,f), φ d1 (t,f)=φ R (t,f)-φ L (t,f)-2π, or φ d1 (t,f)=φ R (t,f)-φ L (t,f)+2π, -π<φ d1(t,f)<π (2) φ Lv (t,f)=φ R (t,f) + 0.5 × (α + 1) × φ d2 (t,f) φ d2 (t,f)=φ L (t,f)-φ R (t,f), φ d2 (t,f)=φ L (t,f)-φ R (t,f)-2π, or φ d2 (t,f)=φ L (t,f)-φ R (t,f)+2π, -π<φ d2 (t,f)<π (3)

[0022] By the way, the method for generating binaural signals when virtual microphones are placed on either side of physical microphones has not been sufficiently studied.

[0023] For example, in the binaural signal generation method described above, the inter-channel phase difference of the actual microphone (e.g., (φ2-φ1), φ d1 or φ d2 A virtual microphone (or binaural signal) is generated by extrapolating the above. In this case, for example, the higher the frequency component, the more likely the inter-channel phase difference is to exceed 2π, making it difficult to uniquely identify the correct phase difference. Thus, because the phase wrap-around processing has not been sufficiently considered, the accuracy of estimating the inter-channel phase difference by extrapolation may decrease, and the quality of the binaural signal using the virtual microphone may deteriorate.

[0024] Furthermore, the binaural signal generation method described above does not modify the amplitude difference between channels (hereinafter also referred to as "inter-channel amplitude difference"). Therefore, for example, when a nearby sound source is located behind the head (or auricle), amplitude attenuation due to occlusion (or diffraction) is not taken into consideration. Consequently, the difference between the binaural signal from a virtual microphone and the binaural signal from an actual microphone tends to become large, and the quality of the binaural signal using a virtual microphone may deteriorate.

[0025] In one non-limiting embodiment of this disclosure, a method for improving the quality of a binaural signal when virtual microphones are placed on either side of a physical microphone is described.

[0026] For example, a communication terminal may set a frequency band that can uniquely identify the phase difference between signals (inter-channel phase difference) based on the distance between the actual microphones constituting a stereo microphone as the target for phase difference calculation, and perform a phase difference calculation in that frequency band such that the phase difference is reliably within the range of 2π, thereby generating a virtual binaural microphone signal having an inter-aural phase difference.

[0027] Furthermore, the communication terminal may apply adjustment (e.g., modification) of the interaural amplitude difference to the virtual binaural microphone signal, taking into account the effects of shielding (or diffraction) by the head, etc., based on the estimated direction of arrival (angle of arrival) of the sound wave, in a frequency band in which the phase difference can be uniquely identified.

[0028] Furthermore, the communication terminal may generate a virtual binaural microphone signal by performing an inverse Fourier transform and overlap-add (OLA) processing on the phase spectrum of a signal obtained by applying a composite window to a signal with a modified inter-channel phase difference, and on the amplitude spectrum of a signal obtained by applying a composite window without modifying the inter-channel phase difference.

[0029] This allows communication terminals to improve the quality of binaural signals when virtual microphones are placed on either side of physical microphones.

[0030] [Overview of the transmission system] A transmission system for voice signals or acoustic signals (for example, referred to as voice-acoustic signals) according to one non-limiting embodiment of the present disclosure may include, for example, a plurality of communication terminals (for example, corresponding to communication devices).

[0031] Multiple communication terminals may include, for example, a transmitting device (hereinafter also referred to as a "transmitting terminal") that encodes an audio signal and transmits the encoded information, and a receiving device (hereinafter also referred to as a "receiving terminal") that receives the encoded information and decodes the audio signal. Furthermore, in one non-limiting embodiment of this disclosure, the process for improving the quality of the binaural signal described above (hereinafter also referred to as the "virtual binauralization process") may be performed in either the transmitting terminal or the receiving terminal.

[0032] Figure 3 is a block diagram showing some configuration examples of a communication terminal. In the communication terminal (transmitting terminal or receiving terminal) shown in Figure 3, the communication unit (corresponding to, for example, a transmitting circuit or a receiving circuit) transmits or receives encoded information (for example, stereo encoded information) obtained by encoding the signal between microphones constituting a stereo microphone (for example, an audio signal). The generation unit (corresponding to, for example, a generation circuit) performs virtual binaural processing on the stereo signal (for example, the stereo signal to be encoded, or the decoded stereo signal obtained by decoding the encoded information) to generate a virtual stereo signal (for example, a pseudo-binaural signal or a virtual binaural signal). An example of the configuration of a communication terminal in an audio signal transmission system will be described later.

[0033] Figure 4 is a diagram showing the process flow for generating signals observed by virtual microphones Lv and Rv in a communication terminal according to one embodiment of the present disclosure.

[0034] In Figure 4, the communication terminal calculates (generates) signals observed by virtual microphones Lv and Rv, taking as input the signals observed by real microphones L and R, the distance r between L and R, and the ratio α (extrapolation coefficient) of the distance between Lv and Rv and the distance between L and R.

[0035] <Step 0 (S0)> The communication terminal applies an analysis window (or window function) to the L channel signal (Lch signal) and R channel signal (Rch signal) included in the input stereo signal (performs windowing).

[0036] <Step 1 (S1)> The communication terminal converts the Lch signal and Rch signal after windowing into a frequency spectrum using FFT (Fast Fourier Transform), and obtains the amplitude spectrum a L (t, f), a R (t, f), and the phase spectrum φ L (t, f), φ R Calculate (t, f).

[0037] <Step 2 (S2)> The communication terminal calculates the phase spectrum φ using the frequency band where the frequency (f) is less than ss / r as the target of calculation. L (t, f) and φR phase difference (inter-channel phase difference) φ with respect to (t, f) d (t, f) is calculated. Here, ss represents the speed of sound. The phase difference φ d The calculation method for (t, f) may apply, for example, the method described with reference to Figure 2 (e.g., the calculation method for φ d1 (t, f) or φ d2 (t, f)).

[0038] Here, the reason why the calculation target of phase difference φ d (t, f) is set (e.g., limited) to a frequency band where the frequency is less than ss / r is to ensure that the phase difference φ d (t, f) falls within 2π.

[0039] For example, the time required for a sound wave to travel the inter-microphone distance r is r / ss. That is, the maximum value of the time difference between signals observed by the two real microphones L and R is r / ss. At this time, for a frequency component whose one period exceeds r / ss (that is, a frequency component whose frequency is less than ss / r), the phase difference between the signals observed by the two real microphones L and R falls within the range of 2π (e.g., ±π). On the other hand, for a frequency component whose one period is equal to or less than r / ss (that is, a frequency component whose frequency is equal to or higher than ss / r), the phase difference between the signals observed by the two real microphones L and R may be 2π (e.g., ±π) or more.

[0040] When the phase difference is 2π or more, it becomes difficult for the communication terminal to uniquely determine the phase difference, and there is a possibility that the true phase difference cannot be calculated. Extrapolation processing (generation of a virtual microphone) using an incorrect phase adversely affects the quality of a signal observed by the virtual microphone. Therefore, the communication terminal performs extrapolation processing on frequency components for which the true phase difference can be uniquely identified (e.g., a frequency band where the frequency is less than ss / r).

[0041] <Step 3 (S3)> The communication terminal calculates (or estimates) the azimuth angle θ(t, f) of the sound source using the phase difference φ d (t, f) for, for example, a frequency band where the frequency is less than ss / r (f<ss / r).

[0042] FIG. 5 is a diagram explaining an example of a method for calculating the azimuth angle θ of a sound wave arriving from a sound source, from the phase difference φ of signals observed by real microphones L and R with an inter-microphone distance r d .

[0043] As shown in FIG. 5, when a sound wave is assumed to be a plane wave, the propagation distance difference d between the sound wave arriving at microphone L and the sound wave arriving at microphone R is expressed as d=r×sinθ. Also, the phase difference φ of signals observed by microphone L and microphone R at frequency f d is d =2πf×d / ss. Therefore, from both equations, the azimuth angle θ can be calculated by θ=arcsin((ss×φ d ) / (2πf×r)).

[0044] Note that for calculating the azimuth angle θ, accurate calculation of the phase difference φ d is preferable. Therefore, as described above, the frequency band (target band) for calculating the azimuth angle θ may be set (for example, limited) to a frequency component that can specify the true phase difference (for example, a frequency band whose frequency is less than ss / r).

[0045] <Step 4 (S4)> The communication terminal calculates, for example, phase spectra φ Lv (t, f) and φ Rv (t, f) of signals observed by virtual microphone Lv and virtual microphone Rv for a frequency band with frequency less than ss / r (f<ss / r). For example, the communication terminal may calculate the phase spectra φ d and the extrapolation coefficient α by the extrapolation process described in FIG. 2 to obtain the phase spectra φ Lv and φ Rv .

[0046] <Step 5 (S5)> For a frequency band that is not a calculation target of the phase difference φ d (non-target band, f≧ss / r), the communication terminal copies, for example, the corresponding time and frequency components of the phase spectra φ L and φ R to obtain phase spectra φ of signals (virtual Lch signal and virtual Rch signal) observed by virtual microphones Lv and Rv Lv and φRv Generates.

[0047] Furthermore, the communication terminal, for example, the amplitude spectrum a L and a R Copy the following and view the amplitude spectrum of the signal observed at virtual microphone Lv and Rv. Lv and a Rv Generates.

[0048] <Step 6 (S6)> For frequency bands where the frequency is less than ss / r, the communication terminal uses the inter-channel amplitude difference (e.g., LR amplitude difference) of the actual microphone L and R, which is associated with the azimuth angle θ calculated in Step 3. d Determine the amplitude difference a d Based on this, the amplitude difference of the signals between virtual microphones is adjusted (or corrected, modified). For example, the communication terminal adjusts the amplitude spectrum a of the signal observed by the virtual microphone. Lv (t, f) and a Rv Of (t, f), the amplitude spectrum with the smallest amplitude is attenuated, and the amplitude difference is a d a Lv (t, f) and a Rv Adjust (t, f).

[0049] Azimuth angle θ and inter-channel amplitude difference a d In this correspondence, for example, amplitude attenuation due to shielding (or diffraction) may be considered when the sound source is located behind the head (or auricle) relative to a communication terminal (e.g., a smartphone).

[0050] Furthermore, for example, for each frequency (e.g., bin or subband) or for multiple frequencies, the azimuth angle θ and the inter-channel amplitude difference a d The two may be associated with each other. For example, the azimuth angle θ and the inter-channel amplitude difference a d A lookup table may be created with the data points associated with each other.

[0051] Alternatively, the azimuth angle θ may be associated with a uniform amplitude ratio (for example, the amplitude ratio of the actual microphone L and R) in part or all of the bandwidth targeted for amplitude difference adjustment. In this case, the relationship between the azimuth angle and the amplitude ratio may be approximated by a transformation function using a linear function, a quadratic function, or a combination thereof.

[0052] <Step 7 (S7)> The communication terminal generates the phase spectrum φ Lv and amplitude spectrum a Lv By performing an inverse Fourier transform such as IFFT (Inverse Fast Fourier Transform) on this, it is converted into a time signal L(t) (frequency-time conversion), and the generated phase spectrum φ is obtained. Rv and amplitude spectrum a Rv By performing an inverse Fourier transform on this, it is converted into a time signal R(t) (frequency-time conversion).

[0053] <Step 8 (S8)> The communication terminal applies a composite window to the time signals L(t) and R(t), performs overlap-add (OLA) processing using the preceding and succeeding frames, and generates signals observed by virtual microphones Lv and Rv. For example, if the distance between virtual microphones Lv and Rv, which constitute a virtual stereo microphone, is the interaural distance, the signals observed by virtual microphones Lv and Rv are called "pseudo-binaural signals" or "virtual binaural signals".

[0054] The above describes an example of the process for generating signals observed by virtual microphones Lv and Rv in a communication terminal.

[0055] Furthermore, in step 4 (phase extrapolation processing), the target frequency band for calculating the phase of a signal observed by a virtual microphone is not limited to a frequency band where the frequency is less than ss / r (f<ss / r). For example, the target frequency band for calculating the phase of a signal observed by a virtual microphone may be set to a frequency band lower than a frequency (e.g., approximately 1500 Hz to 1600 Hz) specified (calculated) from the interaural distance (e.g., approximately 22 cm to 23 cm). Further, the target frequency band for calculating the phase of a signal observed by a virtual microphone may be set to, for example, a frequency band where phase is more likely to affect auditory sound source localization (e.g., a frequency band of approximately 1500 Hz or lower). For example, the target frequency band for calculating the phase of a signal observed by a virtual microphone may be set to a certain frequency band (e.g., a frequency band of approximately 1500 Hz or lower) regardless of the inter-microphone distance r. This makes it possible to further narrow the frequency band to which the processing of calculating the phase of a signal observed by a virtual microphone through extrapolation is applied, thereby reducing the processing amount in a communication terminal.

[0056] Alternatively, for example, by setting a larger inter-microphone distance r (e.g., approximately 10 cm to 20 cm) in a region where actual microphones can be arranged, the frequency band with frequency less than ss / r can be further narrowed. This makes it possible to further narrow the frequency band to which the processing of calculating the phase of a signal observed by a virtual microphone through extrapolation is applied, thereby reducing the processing amount in a communication terminal.

[0057] Further, in step 2 (calculation of inter-channel phase difference of actual microphones), the communication terminal may perform wrapping processing by subtracting 2π from the calculated phase difference when the calculated phase difference is equal to or greater than π, and adding 2π to the calculated phase difference when the calculated phase difference is equal to or less than -π, in a frequency band where the frequency is less than ss / r. Accordingly, even when an error occurs in a phase difference that is assumed to be a value within the range of ±π (for example, when the value falls outside the range of ±π), the communication terminal can correct the calculated phase difference to fall within the range of ±π (that is, within a range where the phase difference can be uniquely determined).

[0058] Furthermore, in step 6 (amplitude difference adjustment), the frequency band targeted for adjusting the amplitude difference of a signal observed by a virtual microphone may be set (or limited) to frequencies higher than a boundary frequency calculated from the interaural distance (e.g., approximately 22 cm to 23 cm) (e.g., approximately 1500 Hz to 1600 Hz). Furthermore, the frequency band targeted for adjusting the amplitude difference of a signal observed by a virtual microphone may be set, for example, to a frequency band in which amplitude more easily affects auditory sound source localization (e.g., frequencies higher than approximately 1500 Hz to 1600 Hz). Note that, as described above, the upper limit of the frequency band targeted for adjusting the amplitude difference of a signal observed by a virtual microphone may be the frequency of ss / r. These settings make it possible to further narrow the frequency band to which the process of adjusting the amplitude of a signal observed by a virtual microphone is applied, thereby reducing the amount of processing in a communication terminal.

[0059] Furthermore, in step 6, the process of adjusting (correcting) the amplitude difference of a signal observed by a virtual microphone may be applied to a frequency band with a frequency equal to or higher than ss / r (e.g., a frequency band in which the phase difference exceeds 2π). In this case, the phase difference φ d For frequency components that are not the calculation target of (or the azimuth angle θ) (e.g., frequency components with a frequency equal to or higher than ss / r), the azimuth angle θ may be determined as follows. For example, in step 3, the communication terminal calculates the phase difference φ d For frequency components that are the calculation target of ), the communication terminal classifies the calculated azimuth angles θ into positive (e.g., direction on the R channel side) and negative (e.g., direction on the L channel side), and calculates the average values "θ+" and "θ-" of the respective azimuth angles. The communication terminal, for example, calculates the phase difference φ d For frequency components that are not the calculation target of , the azimuth angle θ(t, f) may be determined as follows. The communication terminal determines a L (t, f) > a R (t, f), the azimuth angle θ(t, f) is determined to be θ-, and a L (r, f) < a R (t, f), the azimuth angle θ(t, f) is determined to be θ+, and a L (r, f) = a RIn the case of (t, f), the azimuth angle θ(t, f) is determined to be 0. The communication terminal may apply the amplitude adjustment in step 6 based on the azimuth angle θ(t, f) determined in this way.

[0060] Furthermore, in the phase difference adjustment in steps 4 and 5, the extrapolation coefficient α may be gradually reduced to approach 1.0 as the frequency increases. For example, in the phase extrapolation process in step 5, in the frequency band where the frequency is ss / r or higher, the extrapolation coefficient α may be gradually reduced to approach 1.0 as the frequency increases (e.g., this corresponds to moving the extrapolation point closer to the actual microphone), and above a certain frequency, α may be set to 1 (e.g., no extrapolation). Alternatively, for example, α may be set to 1 in the frequency band where the frequency is ss / r or higher. Alternatively, for example, in the phase difference adjustment in steps 4 and 5, in the frequency band below the ss / r frequency, the extrapolation coefficient α may be gradually reduced to approach 1.0 as the frequency increases, and above the ss / r frequency (the boundary frequency of the extrapolation process), α may be set to 1.

[0061] Furthermore, in the amplitude difference adjustment in step 6, the amplitude difference may be gradually reduced to approach zero as the frequency increases.

[0062] Furthermore, in step 6, the amplitude difference of the signal observed by the virtual microphone is amplitude difference a d We have explained how to adjust the amplitude difference so that it becomes as follows, but the adjustment of the amplitude difference of the signal observed by the virtual microphone is not limited to this, and the amplitude difference a d An amplitude difference adjustment may be performed so that the value is calculated based on [the formula / condition].

[0063] Here, we will explain specific examples of the phase spectrum calculation process in Step 4 and Step 5.

[0064] As an example, we will explain the case where the distance between microphones is r = 4.25 cm, the speed of sound is ss = 340 m / s, the signal sampling frequency is 48 kHz, and the number of FFT points is 1024. The distance between microphones, sampling frequency, and number of FFT points are not limited to these values ​​and may be other values.

[0065] In this case, the frequency ss / r = 8000Hz corresponds to 1024 × 8000 / 48000 = 170.67 in the FFT frequency bins. Therefore, the communication terminal sets the frequency bins from the 170th onward, which correspond to frequency bands lower than 8000Hz, as targets for phase extrapolation (targets of step 4). Note that among the frequency bins from the 170th onward, the frequency bin corresponding to the DC component (the first frequency bin) does not need to be set as a target for phase extrapolation.

[0066] Furthermore, the communication terminal may gradually reduce α to 1.0 up to a certain frequency bin (for example, the 250th frequency bin) for the 171st frequency bin and above, which correspond to a frequency band of 8000 Hz or higher, and then set α=1 (no extrapolation or no phase manipulation) from the 251st frequency bin onward.

[0067] [Example of Communication Terminal Configuration] The following describes an example of the configuration of a communication terminal in a voice acoustic signal transmission system according to one non-limiting embodiment of the present disclosure.

[0068] As described above, the communication terminal performs pre-processing (e.g., processing at the transmitting terminal) or post-processing (e.g., processing at the receiving terminal) to generate an input signal (e.g., a pseudo-binaural signal) from a virtual binaural microphone (e.g., a virtual stereo microphone) based on signals input from two physical microphones (e.g., stereo microphones).

[0069] [Communication Terminal Configuration Example 1] In Communication Terminal Configuration Example 1, we will describe a communication terminal configuration example in which the pseudo-binaural signal generation process (virtual binauralization process) explained with reference to Figure 4 is performed as processing (for example, post-processing) at the receiving terminal. In Configuration Example 1, the transmission system includes, for example, a transmitting terminal 10 and a receiving terminal 20.

[0070] <Example Configuration of Transmitting Terminal 10> Figure 6 is a block diagram showing an example configuration of the transmitting terminal 10 in Configuration Example 1. The transmitting terminal 10 transmits, for example, stereo encoded information obtained by encoding the signal from a stereo microphone to the receiving terminal 20.

[0071] In Figure 6, the transmitting terminal 10 comprises a plurality of microphones 11, an input signal selection unit 12, a microphone distance information transmission unit 13, and an encoding unit 14.

[0072] Multiple microphones 11 supply the audio signals observed by each microphone to the input signal selection unit 12.

[0073] The input signal selection unit 12 selects two signals from the audio signals supplied by the multiple microphones 11 to be used for generating a pseudo-binaural signal, and outputs the two selected signals as a stereo signal (L channel (Lch signal) and R channel (Rch signal)) to the encoding unit 14.

[0074] Furthermore, the input signal selection unit 12 outputs mode information to the encoding unit 14 indicating that the two selected signals are stereo signals and that encoding should be performed in stereo signal encoding mode. In addition to the stereo signal encoding mode, the information that can be set in the mode information may also include other encoding modes applicable to the transmitting terminal 10.

[0075] Furthermore, the input signal selection unit 12 outputs information (for example, referred to as "selected microphone information") regarding the microphones that supply the two selected signals (for example, the microphones that make up a stereo microphone) to the microphone distance information transmission unit 13. The selected microphone information may include, for example, at least microphone distance information indicating the distance between the two selected microphones.

[0076] The microphone distance information transmission unit 13 encodes the selected microphone information (including at least the microphone distance information) input from the input signal selection unit 12 and sends the selected microphone information to the receiving terminal 20.

[0077] The encoding unit 14 encodes the signal input from the input signal selection unit 12 based on the mode information input from the input signal selection unit 12, and sends the encoded information to the receiving terminal 20. For example, if the mode information indicates stereo encoding mode, the encoding unit 14 encodes the stereo signal input from the input signal selection unit 12 in stereo encoding mode and sends the stereo encoded information to the receiving terminal 20.

[0078] For example, the transmission frequency of selected microphone information may be lower than the transmission frequency of stereo coding information. Selected microphone information may also be transmitted as signaling information at the start of a session or as control information when the selected microphone is changed.

[0079] Furthermore, the encoded selected microphone information (including, for example, microphone distance information) may be transmitted separately from the stereo encoded information, as shown in Figure 6, or it may be included in the bitstream as part of the stereo encoded information.

[0080] <Example Configuration of Receiving Terminal 20> Figure 7 is a block diagram showing an example configuration of the receiving terminal 20 in Configuration Example 1. The receiving terminal 20, for example, receives stereo encoding information transmitted from the transmitting terminal 10 and performs virtual binaural processing using the received stereo encoding information.

[0081] In Figure 7, the receiving terminal 20 includes a microphone distance information receiving unit 21, a decoding unit 22, and a virtual binaural processing unit 23. The virtual binaural processing unit 23 also includes, for example, an inter-channel phase difference changing unit 24 and an inter-channel amplitude difference changing unit 25.

[0082] The microphone distance information receiving unit 21 and decoding unit 22 shown in Figure 7 may be included in the communication unit shown in Figure 3, for example. Also, the virtual binaural processing unit 23 shown in Figure 7 may be included in the generation unit shown in Figure 3, for example.

[0083] The microphone distance information receiving unit 21 receives selected microphone information (for example, including microphone distance information) transmitted from the transmitting terminal 10 and outputs the microphone distance information included in the selected microphone information to the channel phase difference changing unit 24.

[0084] The decoding unit 22 receives stereo coding information transmitted from the transmitting terminal 10, decodes the stereo coding information, and outputs the decoded stereo signal (L channel (Lch signal) and R channel (Rch signal)) to the inter-channel phase difference changing unit 24.

[0085] The virtual binaural processing unit 23 uses, for example, the inter-microphone distance information input from the inter-microphone distance information receiving unit 21 to perform virtual binaural processing (for example, the processing shown in Figure 4) on the decoded stereo signal input from the decoding unit 22, thereby generating a signal from a virtual stereo microphone composed of virtual microphones (for example, a pseudo-binaural signal).

[0086] In the virtual binaural processing unit 23, the inter-channel phase difference changing unit 24 uses the inter-microphone distance information input from the inter-microphone distance information receiving unit 21 and the decoded stereo signal input from the decoding unit 22 to adjust (change) the phase difference of the phase spectrum in the virtual microphone, for example, by the method described using Figures 2 and 4 (steps 4 and 5) (phase extrapolation processing), and outputs the frequency spectrum (e.g., L and R) of the pseudo-binaural signal including the phase spectrum to the inter-channel amplitude difference changing unit 25. The inter-channel phase difference changing unit 24 also determines the azimuth angle θ of the sound source, for example, by the method described using Figures 4 (step 3) and 5 (estimation of the azimuth angle θ of the sound source), and outputs the information of the determined azimuth angle θ to the inter-channel amplitude difference changing unit 25.

[0087] In the virtual binaural processing unit 23, the inter-channel amplitude difference modification unit 25 determines (estimates) the inter-channel amplitude difference based on the azimuth angle θ input from the inter-channel phase difference modification unit 24. The inter-channel amplitude difference modification unit 25 adjusts (changes) the amplitude difference of the frequency spectrum L (amplitude and phase) and R (amplitude and phase) of the pseudo-binaural signal input from the inter-channel phase difference modification unit 24 using the determined inter-channel amplitude difference, for example, by the method (amplitude difference adjustment) described using Figure 4 (step 6). The inter-channel amplitude difference modification unit 25 outputs a pseudo-binaural signal with adjusted amplitude differences. The output pseudo-binaural signal is presented to the user's ears, for example, via headphones.

[0088] [Example 1 of Inter-Channel Phase Difference Changing Unit and Inter-Channel Amplitude Difference Changing Unit] <Example of the configuration of the Inter-Channel Phase Difference Changing Unit 24> Figure 8 is a block diagram showing an example of the internal configuration of the Inter-Channel Phase Difference Changing Unit 24 shown in Figure 7.

[0089] The inter-channel phase difference changing unit 24 generates a signal from a virtual stereo microphone (e.g., a pseudo-binaural signal) composed of virtual microphones placed on either side of the stereo microphone by extrapolating the phase spectrum (or phase component) of the decoded stereo signal obtained by decoding the stereo coding information in a frequency band where the phase difference between the signals of the microphones constituting the stereo microphone (e.g., microphone 11) is within 2π.

[0090] The inter-channel phase difference changing unit 24 shown in Figure 8 includes, for example, time / frequency conversion units 241-1 and 241-2, a phase difference calculation unit 242, phase changing units 243-1 and 243-2, and an azimuth angle estimation unit 244.

[0091] In the receiving terminal 20, the microphone distance information output from the microphone distance information receiving unit 21 is input to the phase change unit 243-1, the phase change unit 243-2, and the azimuth angle estimation unit 244.

[0092] Furthermore, in the receiving terminal 20, the L channel component (Lch signal) of the decoded stereo signal output from the decoding unit 22 is input to the L channel time / frequency conversion unit 241-1, and the R channel component (Rch signal) of the decoded stereo signal is input to the R channel time / frequency conversion unit 241-2.

[0093] The time / frequency conversion unit 241-1 converts the input L-channel signal into a frequency spectrum (amplitude component, phase component) using FFT or short-time Fourier transform, and the amplitude component (amplitude spectrum) (for example, a L ) is output to the inter-channel amplitude difference changing unit 25, and the phase component (phase spectrum) (for example, φ L The output is sent to the phase difference calculation unit 242 and the phase change unit 243-1.

[0094] The time / frequency conversion unit 241-2 converts the input R channel signal into a frequency spectrum (amplitude component, phase component) using FFT or short-time Fourier transform, and the amplitude component (amplitude spectrum) (for example, a R ) is output to the inter-channel amplitude difference changing unit 25, and the phase component (phase spectrum) (for example, φ ROutputs ) to a phase difference calculation unit 242 and a phase changing unit 243-2.

[0095] It should be noted that in the time-frequency conversion units 241-1 and 241-2, conversion such as FFT is generally performed on a signal to which an analysis window has been applied.

[0096] The phase difference calculation unit 242 uses the L-channel phase spectrum input from the time-frequency conversion unit 241-1 and the R-channel phase spectrum input from the time-frequency conversion unit 241-2 to calculate the phase difference between the L and R channels in a frequency band lower than a certain frequency (e.g., ss / r) (e.g., φ d ) and outputs information related to the calculated phase difference to the L-channel phase changing unit 243-1, the R-channel phase changing unit 243-2, and an azimuth angle estimation unit 244.

[0097] The phase changing unit 243-1 is based on inter-microphone distance information (e.g., inter-microphone distance r), the L-channel phase spectrum (e.g., φ L ), and phase difference information (e.g., φ d ), by using, for example, the method described with reference to FIG. 2 and FIG. 4 (step 4) (e.g., extrapolation processing), a phase spectrum (e.g., φ Lv ) of a signal observed by a virtual microphone (the L-channel of a pseudo-binaural signal) is generated (changed). The phase changing unit 243-1 outputs the generated phase difference spectrum to an inter-channel amplitude difference changing unit 25.

[0098] The phase changing unit 243-2 (R-channel) is based on inter-microphone distance information (e.g., inter-microphone distance r), the R-channel phase spectrum (e.g., φ R ), and phase difference information (e.g., φ d ), by using, for example, the method described with reference to FIG. 2 and FIG. 4 (step 4) (e.g., extrapolation processing), a phase spectrum (e.g., φ Rv ) of a signal observed by a virtual microphone (the R-channel of a pseudo-binaural signal) is generated (changed). The phase changing unit 243-2 outputs the generated phase difference spectrum to an inter-channel amplitude difference changing unit 25.

[0099] In this way, through the processing of the phase changing units 243-1 and 243-2, the phase difference between the L channel and the R channel is changed to the phase difference calculated by the phase difference calculation unit 242.

[0100] The azimuth angle estimation unit 244 receives microphone distance information (e.g., microphone distance r) and phase difference information (e.g., φ d Based on this, the azimuth angle θ of the sound source is estimated using the method described with reference to Figures 4 (Step 3) and 5. The azimuth angle estimation unit 244 outputs information regarding the estimated azimuth angle θ to the inter-channel amplitude difference changing unit 25.

[0101] The extrapolation coefficient α used in the phase extrapolation process in the phase changing units 243-1 and 243-2 may be determined, for example, by the ratio of the distance between the ears to the distance between the microphones r.

[0102] <Example of the configuration of the inter-channel amplitude difference changing unit 25> Figure 9 is a block diagram showing an example of the internal configuration of the inter-channel amplitude difference changing unit 25 shown in Figure 7.

[0103] The inter-channel amplitude difference changing unit 25 adjusts the distance r between the microphones constituting the stereo microphone and the phase difference φ between the microphones constituting the stereo microphone in a frequency band where the phase difference of the signals between the microphones (e.g., microphone 11) is within 2π. d Based on this, the azimuth angle θ of the sound source is estimated. Then, the inter-channel amplitude difference changing unit 25 converts the amplitude difference a of the signals between the microphones constituting the stereo microphone, which is associated with the azimuth angle θ. d Based on this, the amplitude difference between the signals from the virtual microphones that make up the virtual stereo microphone is adjusted. This generates a pseudo-binaural signal with adjusted amplitude.

[0104] The inter-channel amplitude difference changing unit 25 shown in Figure 9 comprises an amplitude difference estimation unit 251, an amplitude difference changing unit 252, and frequency / time conversion units 253-1 and 253-2.

[0105] The amplitude difference estimation unit 251 uses the azimuth angle θ input from the inter-channel phase difference changing unit 24 to determine the inter-channel amplitude difference (for example, a) using the method (azimuth angle θ estimation) described with reference to Figures 4 (step 6) and 5. dThe system estimates the inter-channel amplitude difference and outputs the estimated inter-channel amplitude difference information to the amplitude difference modification unit 252.

[0106] The amplitude difference modification unit 252 uses the inter-channel amplitude difference input from the amplitude difference estimation unit 251 and the amplitude spectrum (L channel, R channel) of the pseudo-binaural signal input from the inter-channel phase difference modification unit 24 to attenuate the amplitude spectrum of the channel with the smaller amplitude among the L channel and R channel, as explained in Figure 4 (step 6), and adjusts (modifies) the amplitude of the amplitude spectrum so that there is an inter-channel amplitude difference. The amplitude difference modification unit 252 outputs the amplitude spectra of the pseudo-binaural signal (L channel and R channel) after amplitude adjustment to the frequency / time conversion units 253-1 and 253-2, respectively.

[0107] The frequency / time conversion unit 253-1 uses the phase spectrum of the phase-shifted L channel input from the inter-channel phase-shifting unit 24 and the amplitude spectrum of the amplitude-shifted L channel input from the amplitude-shifting unit 252 to convert the frequency spectrum into a time signal using IFFT or the like, and outputs it as the L channel signal (Lch signal) of the pseudo-binaural signal.

[0108] The frequency / time conversion unit 253-2 uses the phase spectrum of the R channel with phase difference changed input from the inter-channel phase difference changing unit 24 and the amplitude spectrum of the R channel with amplitude difference changed input from the amplitude difference changing unit 252 to convert the frequency spectrum into a time signal using IFFT or the like, and outputs it as the R channel signal (Rch signal) of the pseudo-binaural signal.

[0109] In the frequency / time conversion units 253-1 and 253-2, the signal after conversion processing such as IFFT is generally subjected to a composite window to prevent discontinuities between frames, and then overlap-add processing (OLA processing) is performed on the preceding and succeeding frames to obtain the final composite signal.

[0110] [Example 2 of the inter-channel phase difference changing unit 24 and inter-channel amplitude difference changing unit 25] Figure 10 is a diagram showing the process flow for generating signals observed by the virtual microphone Lv and Rv in the receiving terminal 20 according to Example 2.

[0111] In Figure 10, the receiving terminal 20 calculates (generates) the signals observed by virtual microphones Lv and Rv, taking as input the signals observed by actual microphones L and R, the distance r between microphones L and R, and the ratio α (extrapolation coefficient) of the distance between Lv and Rv and the distance between L and R.

[0112] <Step 10 (S10)> The receiving terminal 20 applies an analysis window (or window function) to the Lch signal and Rch signal included in the input stereo signal (performs windowing).

[0113] The windowed Lch and Rch signals obtained in step 10 may be used in the amplitude spectrum generation process described later (for example, proceed to amplitude generation (1) shown in Figure 11).

[0114] <Step 11 (S11)> The receiving terminal 20 converts the Lch signal and Rch signal after windowing into a frequency spectrum using FFT and obtains the amplitude spectrum a L (t, f), a R (t, f), and the phase spectrum φ L (t, f), φ R Calculate (t, f).

[0115] <Step 12 (S12)> The receiving terminal 20 calculates the phase spectrum φ using the frequency band where the frequency (f) is less than ss / r as the target of calculation. L (t, f) and φ R Phase difference with (t, f) (inter-channel phase difference) φ d Calculate (t, f). Here, ss represents the speed of sound. Phase difference φ d The method for calculating (t, f) is the same as the process in Figure 4, for example, the method explained using Figure 2 (for example, φ d1 (t, f) or φ d2 The method for calculating (t, f) may be applied.

[0116] Note that the phase difference φ calculated in step 12 d (t, f) may be used in the amplitude generation process described later (for example, proceed to amplitude generation (2) in Figure 11).

[0117] <Step 13 (S13)> The receiving terminal 20, for example, obtains the phase spectrum φ of the signals observed by the virtual microphone Lv and the virtual microphone Rv for a frequency band whose frequency is less than ss / r (f<ss / r) Lv (t, f) and φ Rv (t, f) are calculated. For example, the receiving terminal 20 uses the phase difference φ calculated in step 12 d and the extrapolation coefficient α to calculate the phase spectrum φ by the extrapolation processing described with reference to Fig. 2 Lv and φ Rv may be calculated.

[0118] <Step 14 (S14)> The receiving terminal 20, for a frequency band that is not a calculation target of the phase difference φ (non-target band, f≧ss / r), obtains, for example, the phase spectrum φ d and φ L by copying the corresponding time and frequency components of R to generate the phase spectra φ of the signals observed by the virtual microphones Lv and Rv (virtual Lch signal and virtual Rch signal) Lv and φ Rv are generated.

[0119] <Step 15 (S15)> The receiving terminal 20 performs inverse Fourier transform such as IFFT on the amplitude spectrum a L and the generated (for example, phase difference modified) phase spectrum φ Lv to convert the result into a time signal L v (t) (pseudo binaural signal) (frequency-time conversion). Further, the receiving terminal 20 performs inverse Fourier transform on the amplitude spectrum a R and the generated (for example, phase difference modified) phase spectrum φ Rv to convert the result into a time signal R v (t) (pseudo binaural signal) (frequency-time conversion).

[0120] <Step 16 (S16)> The receiving terminal 20 applies a synthesis window (or OLA window) to the time signals L v (t) and R v (t) to generate pseudo binaural synthesized signals L vs (t) and R vs (t).

[0121] <Step 17 (S17)> The receiving terminal 20 receives the pseudo-binaural synthesized signal L vs (t) and R vs (t) is converted to a frequency spectrum using FFT, and the amplitude spectrum a of the pseudo-binaural synthesized signal is obtained. Lvs (t, f), a Rvs (t, f), and the phase spectrum φ Lvs (t, f), φ Rvs Calculate (t, f).

[0122] <Step 18 (S18)> The receiving terminal 20 receives the amplitude spectrum a generated by the amplitude generation process described later. Lg (t, f), a Rg (t, f), and the phase spectrum φ of the pseudo-binaural synthesis signal. Lvs (t, f), φ Rvs By applying an inverse Fourier transform such as IFFT and overlap-add processing to (t, f), the final pseudo-binaural signal L is obtained. vb (t) and R vb Generate (t).

[0123] In other words, in step 18, the amplitude spectrum a of the pseudo-binaural synthesis signal (e.g., a phase-shifted signal) calculated in step 17 is obtained. Lvs (t, f), a Rvs (t, f) is not used, and the amplitude spectrum a of the signal generated by the amplitude generation process described later (for example, a signal whose phase difference is not changed) is used. Lg (t, f), a Rg (t, f) is used.

[0124] Next, we will explain an example of amplitude generation processing.

[0125] Figure 11 shows the flow of the amplitude spectrum generation process at the receiving terminal 20.

[0126] <Step 20 (S20)> The receiving terminal 20 applies a composite window (or OLA window) to the Lch signal and Rch signal after windowing by the analysis window in step 11 of Figure 10.

[0127] <Step 21 (S21)> Receiving terminal 20 converts the windowed Lch signal and Rch signal into a frequency spectrum using FFT to obtain an amplitude spectrum a Lw (t, f), a Rw (t, f), and a phase spectrum φ Lw (t, f), φ Rw (t, f) are calculated.

[0128] <Step 22 (S22)> Receiving terminal 20 calculates (or estimates) the azimuth angle θ(t, f) of the sound source using the phase difference φ calculated in step 12 of FIG. 10, for example, for a frequency band where the frequency is less than ss / r (f<ss / r) d (t, f). The calculation method for the azimuth angle θ(t, f) of the sound source may be, for example, the method described with reference to FIG. 5.

[0129] <Step 23 (S23)> For a frequency band where the frequency is less than ss / r, receiving terminal 20 determines the inter-channel amplitude difference of the real microphones L-R (e.g., L-R amplitude difference) a according to the azimuth angle θ calculated in step 22 d , and adjusts (or corrects or changes) the amplitude difference of the signal between the microphones based on the determined amplitude difference a d . For example, receiving terminal 20 adjusts at least one of a Lw (t, f) and a Rw (t, f) such that the amplitude difference between the amplitude spectrum a d (t, f) and a Lw (t, f) is a Rw , and generates adjusted amplitude spectrum a Lg (t, f) and a Rg (t, f).

[0130] <Step 24 (S24)> For a frequency band that is not a calculation target of phase difference φ (non-target band: f≧ss / r), receiving terminal 20 copies the corresponding time and frequency components of the amplitude spectrum a d and a Lw of the Lch signal and Rch signal after combined windowing to obtain amplitude spectrum a Rw and a Lg and a RgGenerates.

[0131] <Step 25 (S25)> The receiving terminal 20 receives the amplitude spectrum a generated in step 23. Lg (t, f) and a Rg (t, f), and the amplitude spectrum a generated in step 24. Lg (t, f) and a Rg Using (t, f), the amplitude spectrum a of each frequency band. Lg (t, f) and a Rg Obtain (t, f). Obtain the amplitude spectrum a Lg (t, f) and a Rg (t, f) is, for example, the final pseudo-binaural signal L in step 18 of Figure 10. vb (t) and R vb Used to generate (t).

[0132] The above describes an example of the process for generating a pseudo-binaural signal in the receiving terminal 20.

[0133] Figure 12 is a block diagram showing an example of the internal configuration of the inter-channel phase difference changing unit 24 and the inter-channel amplitude difference changing unit 25 shown in Figure 7.

[0134] <Example of the configuration of the inter-channel phase difference changing unit 24> The inter-channel phase difference changing unit 24 generates a signal from a virtual stereo microphone (for example, a pseudo-binaural signal) composed of virtual microphones placed on both sides of the stereo microphone by extrapolating the phase spectrum (or phase component) of the decoded stereo signal obtained by decoding the stereo coding information in a frequency band where the phase difference of the signals between the microphones (for example, microphone 11) constituting the stereo microphone is within 2π.

[0135] The inter-channel phase difference changing unit 24 shown in Figure 12 includes, for example, analysis windows 261-1 and 261-2, FFT units 262-1 and 262-2, a phase difference changing unit 263, phase changing units 264-1 and 264-2, and an azimuth angle estimation unit 265.

[0136] In the receiving terminal 20, the microphone distance information output from the microphone distance information receiving unit 21 is input to the phase difference changing unit 263 and the azimuth angle estimation unit 265.

[0137] The analysis window 261-1 performs a windowing process using an analysis window (window function) on the L channel component (Lch signal) of the decoded stereo signal output from the decoding unit 22, and outputs the windowed Lch signal to the FFT unit 262-1 and the inter-channel amplitude difference modification unit 25 (combination window 271-1).

[0138] The analysis window 261-2 applies a windowing process using an analysis window (window function) to the R channel component (Rch signal) of the decoded stereo signal output from the decoding unit 22, and outputs the windowed Rch signal to the FFT unit 262-2 and the inter-channel amplitude difference modification unit 25 (combination window 271-2).

[0139] The FFT unit 262-1 converts the Lch signal input from the analysis window 261-1 into a Fourier spectrum (amplitude component, phase component) using FFT, and the amplitude component (amplitude spectrum) (for example, a L ) is output to the inter-channel amplitude difference changing unit 25 (IFFT unit 272-1) and the azimuth angle estimation unit 265, and the phase component (phase spectrum) (for example, φ L The output is sent to the phase difference changing unit 263 and the phase changing unit 264-1.

[0140] The FFT unit 262-2 converts the Rch signal input from the analysis window 261-2 into a Fourier spectrum (amplitude component, phase component) using FFT, and the amplitude component (amplitude spectrum) (for example, a R ) is output to the inter-channel amplitude difference changing unit 25 (IFFT unit 272-2) and the azimuth angle estimation unit 265, and the phase component (phase spectrum) (for example, φ R The output is sent to the phase difference changing unit 263 and the phase changing unit 264-2.

[0141] The phase difference changing unit 263 uses the phase spectrum of the L channel input from the FFT unit 262-1 and the phase spectrum of the R channel input from the FFT unit 262-2 to change the phase difference between the L and R channels (for example, φ) in a frequency band below a certain frequency (for example, ss / r). dThe phase difference changing unit 263 then calculates the distance between microphones (for example, the distance between microphones r) and the calculated phase difference information (for example, φ). d Based on this, for example, the phase spectrum (e.g., φ) of the signal observed by the virtual microphone (the L channel and R channel of the pseudo-binaural signal) is obtained by the method described using Figure 10 (steps 13 and 14) (e.g., extrapolation). Lv , φ Rv The phase difference conversion unit 263 calculates the phase spectrum of the L channel and the phase spectrum of the R channel, respectively, and outputs the calculated phase difference information to the phase difference conversion unit 264-1 for the L channel and the phase difference conversion unit 264-2 for the R channel, and outputs the information regarding the calculated phase difference to the azimuth angle estimation unit 265.

[0142] The phase change unit 264-1 (for the L channel) changes the phase spectrum input from the FFT unit 262-1 to the phase spectrum of the L channel input from the phase difference change unit 263, and outputs the changed phase spectrum to the inter-channel amplitude difference change unit 25 (IFFT unit 272-1).

[0143] The phase change unit 264-2 (for the R channel) changes the phase spectrum input from the FFT unit 262-2 to the phase spectrum of the R channel input from the phase difference change unit 263, and outputs the changed phase spectrum to the inter-channel amplitude difference change unit 25 (IFFT unit 272-2).

[0144] In this way, through the processing of phase changing units 264-1 and 264-2, the phase difference between the L channel and the R channel is changed to the phase difference calculated by the phase difference changing unit 263.

[0145] The azimuth angle estimation unit 265 receives microphone distance information (e.g., microphone distance r) and phase difference information (e.g., φ) input from the phase difference modification unit 263. d Based on the amplitude spectrum information input from the FFT units 262-1 and 262-2, the azimuth angle θ of the sound source is estimated using the method described with reference to Figure 11 (step 22).

[0146] For example, the azimuth angle estimation unit 265 may estimate the azimuth angle using microphone distance information and phase difference information. Furthermore, if it is difficult to uniquely determine the azimuth angle from the microphone distance information and phase difference information, the azimuth angle estimation unit 265 may further estimate the azimuth angle based on the amplitude spectrum. For example, the azimuth angle estimation unit 265 may use the amplitude spectrum information to determine whether the amplitude of the L channel or the R channel is larger for the azimuth angle estimated using microphone distance information and phase difference information.

[0147] The azimuth angle estimation unit 265 outputs information regarding the estimated azimuth angle θ to the inter-channel amplitude difference modification unit 25 (amplitude difference estimation unit 276).

[0148] The extrapolation coefficient α used in the phase extrapolation process for phase difference modification in the phase difference modification of the phase difference modification unit 263 and phase modification units 264-1 and 264-2 may be determined, for example, by the ratio of the distance between both ears to the distance between microphones r.

[0149] <Example of configuration of the inter-channel amplitude difference changing unit 25> The inter-channel amplitude difference changing unit 25 adjusts the distance r between the microphones constituting the stereo microphone and the phase difference φ between the microphones constituting the stereo microphone in a frequency band where the phase difference of the signals between the microphones (for example, microphone 11) is within 2π. d The azimuth angle θ is estimated based on this. Then, the inter-channel amplitude difference changing unit 25 adjusts the amplitude difference a of the signals between the microphones constituting the stereo microphone according to the estimated azimuth angle θ. d Based on this, the amplitude difference of the signals between the virtual microphones that make up the virtual stereo microphone is adjusted (modified).

[0150] In the inter-channel amplitude difference modification unit 25 shown in Figure 12, the amplitude spectrum used to generate the final pseudo-binaural signal is not the amplitude spectrum of the signal obtained by applying a synthesis window to the phase difference modified signal in the inter-channel phase difference modification unit 24, but rather the amplitude spectrum of the signal obtained by applying a synthesis window to the signal that is not phase difference modified in the inter-channel phase difference modification unit 24. Furthermore, as shown in Figure 12, the inter-channel amplitude difference modification unit 25, for example, applies the output of analysis windows 261-1 and 261-2 (decoded stereo signals that are not phase difference modified) to the synthesis window, and then performs amplitude difference modification (amplitude modification of each channel). This generates a pseudo-binaural signal using the amplitude spectrum after amplitude difference adjustment.

[0151] The inter-channel amplitude difference changing unit 25 shown in Figure 12 comprises synthesis windows 271-1, 271-2, IFFT units 272-1, 272-2, synthesis windows 273-1, 273-2, FFT units 274-1, 274-2, FFT units 275-1, 275-2, amplitude difference estimation unit 276, amplitude difference changing unit 277, amplitude changing units 278-1, 278-2, IFFT units 279-1, 279-2, and OLA units 280-1, 280-2.

[0152] The synthesis window 271-1 applies a synthesis window to the Lch signal input from the analysis window 261-1 after analysis windowing, and outputs the synthesized signal to the FFT unit 274-1.

[0153] The synthesis window 271-2 applies a synthesis window to the Rch signal input from the analysis window 261-2 after analysis windowing, and outputs the signal with the synthesis window applied to the FFT unit 274-2.

[0154] The IFFT unit 272-1 processes the amplitude spectrum of the L channel input from the inter-channel phase difference changing unit 24 (FFT unit 262-1) (for example, a L ), and the phase spectrum of the L channel input from the inter-channel phase difference changing unit 24 (phase changing unit 264-1) (for example, φ Lv ) performs IFFT on the time signal (for example, L v ) is output to the composite window 273-1.

[0155] The IFFT unit 272-2 processes the amplitude spectrum of the R channel input from the inter-channel phase difference changing unit 24 (FFT unit 262-2) (for example, a R ), and the phase spectrum of the R channel input from the inter-channel phase difference changing unit 24 (phase changing unit 264-2) (for example, φ Rv ) performs IFFT on the time signal (for example, R v ) is output to the composite window 273-2.

[0156] The composite window 273-1 applies the composite window to the signal input from the IFFT unit 272-1, and the resulting signal (for example, L) is obtained by applying the composite window. vs The output is sent to the FFT unit 275-1.

[0157] The composite window 273-2 applies the composite window to the signal input from the IFFT unit 272-2, and the resulting signal (for example, R) vs The output is sent to the FFT section 275-2.

[0158] The FFT unit 274-1 performs an FFT on the signal input from the composite window 271-1 and obtains the amplitude spectrum (for example, a Lw The output is sent to the amplitude changing unit 278-1.

[0159] The FFT unit 274-2 performs an FFT on the signal input from the composite window 271-2 and obtains the amplitude spectrum (for example, a Rw The output is sent to the amplitude changing unit 278-2.

[0160] The FFT unit 275-1 performs an FFT on the signal input from the composite window 273-1 and obtains the phase spectrum (for example, φ Lvs The output is sent to the IFFT unit 279-1.

[0161] The FFT unit 275-2 performs an FFT on the signal input from the composite window 273-2 and obtains the phase spectrum (for example, φ Rvs The output is sent to the IFFT unit 279-2.

[0162] The amplitude difference estimation unit 276 uses the azimuth angle θ input from the inter-channel phase difference changing unit 24 (azimuth angle estimation unit 265) to determine the inter-channel amplitude difference (for example, a) using the method (azimuth angle θ estimation) described with reference to Figures 11 (step 23) and 5. dThe system estimates the inter-channel amplitude difference and outputs the estimated information regarding the inter-channel amplitude difference to the amplitude difference modification unit 277.

[0163] The amplitude difference modification unit 277 uses the inter-channel amplitude difference input from the amplitude difference estimation unit 276 and the amplitude spectra (L channel, R channel) of the input signals after windowing by the analysis window and the synthesis window, input from the FFT units 274-1 and 274-2, to control (or determine, set) the amplitudes of the L channel amplitude spectrum and the R channel amplitude spectrum so that they become an inter-channel amplitude difference, as explained in Figure 11 (step 23). For example, the amplitude difference modification unit 277 may attenuate the amplitude spectrum of the channel with the smaller amplitude among the L channel and R channel, and determine the amplitude of the amplitude spectrum so that there is an inter-channel amplitude difference.

[0164] The amplitude changing unit 278-1 changes (adjusts) the amplitude of the amplitude spectrum of the L channel according to the control of the amplitude difference changing unit 277, and the amplitude spectrum after amplitude adjustment (for example, a Lg The output is sent to the IFFT unit 279-1.

[0165] The amplitude changing unit 278-2 changes (adjusts) the amplitude of the amplitude spectrum of the R channel according to the control of the amplitude difference changing unit 277, and the amplitude spectrum after amplitude adjustment (for example, a Rg The output is sent to the IFFT unit 279-2.

[0166] The IFFT unit 279-1 processes the phase spectrum of the L channel input from the FFT unit 275-1 (for example, φ Lvs ), and the amplitude spectrum of the amplitude-modified L channel input from the amplitude modification unit 278-1 (for example, a Lg Using this, the frequency spectrum is converted into a time signal by IFFT and output to the OLA unit 280-1.

[0167] The OLA unit 280-1 performs overlap-add processing on the time signal input from the IFFT unit 279-1 using the preceding and succeeding frames to create the L channel signal of the pseudo-binaural signal (for example, L vb ) Output.

[0168] The IFFT unit 279-2 processes the phase spectrum of the R channel input from the FFT unit 275-2 (for example, φ Rvs ), and the amplitude spectrum of the amplitude-modified R channel input from the amplitude modification unit 278-2 (for example, a Rg Using this, the frequency spectrum is converted into a time signal by IFFT and output to the OLA unit 280-2.

[0169] The OLA unit 280-2 performs overlap-add processing on the time signal input from the IFFT unit 279-2 using the preceding and succeeding frames to obtain the R channel signal of the pseudo-binaural signal (for example, R vb ) Output.

[0170] In this way, the receiving terminal 20 performs IFFT processing and overlap-add processing using the phase spectrum (e.g., the output of FFT units 275-1 and 275-2) calculated by performing FFT processing again on the signal synthesized by changing the phase difference between channels (e.g., the output of synthesis windows 273-1 and 273-2), and the amplitude spectrum (e.g., the output of amplitude modification units 278-1 and 278-2) calculated by performing FFT processing on the signal synthesized without changing the phase difference between channels (e.g., the output of synthesis windows 271-1 and 271-2), to synthesize a time signal. In other words, the receiving terminal 20 generates a pseudo-binaural signal without using the amplitude spectrum calculated by performing FFT processing on the signal synthesized by changing the phase difference between channels (e.g., the output of synthesis windows 273-1 and 273-2). This mitigates the effect of attenuation of the amplitude in the signal after applying the synthesis window due to the change in phase difference, thereby improving the accuracy of pseudo-binaural signal generation.

[0171] [Example 3 of the Inter-Channel Phase Difference Changing Unit 24 and Inter-Channel Amplitude Difference Changing Unit 25] In Example 3, the processing is basically the same as in Example 2 (for example, the processing in Figure 10), but the amplitude generation processing differs from that of Example 2. In the processing of Figure 10, in Example 2, the case in which the windowed Lch signal and Rch signal obtained in step 10 are used in the amplitude spectrum generation processing was explained, but in Example 3, in step 11, the amplitude spectrum a obtained by converting the windowed Lch signal and Rch signal into a frequency spectrum using FFT is used.L (t, f), a R (t, f) is used in the amplitude spectrum generation processing (for example, the process proceeds to amplitude generation (1)' shown in FIG. 13).

[0172] FIG. 13 is a diagram showing the flow of amplitude spectrum generation processing in the receiving terminal 20.

[0173] <Step 30 (S30)> For a frequency band whose frequency is less than ss / r (f<ss / r), the receiving terminal 20 uses the phase difference φ calculated in step 12 of FIG. 10 d (t, f) to calculate (or estimate) the azimuth θ(t, f) of the sound source. The method for calculating the azimuth θ(t, f) of the sound source may be, for example, the method described in FIG. 5.

[0174] <Step 31 (S31)> For a frequency band whose frequency is less than ss / r, the receiving terminal 20 determines the inter-channel amplitude difference of real microphones L-R (for example, L-R amplitude difference) a according to the azimuth θ calculated in step 30 d and adjusts (or corrects, changes) the amplitude difference of signals between microphones based on the determined amplitude difference a d . For example, the receiving terminal 20 adjusts at least one of a L (t, f) and a R (t, f) such that the amplitude difference between the input signal amplitude spectrum a d is a L (t, f) and a R (t, f), and generates the adjusted amplitude spectra a Lm (t, f) and a Rm (t, f).

[0175] <Step 32 (S32)> For a frequency band that is not a calculation target of the phase difference φ (non-target band: f≧ss / r), the receiving terminal 20, for example, copies the corresponding time and frequency components of the amplitude spectra a d (t, f) and a L (t, f) of the Lch signal and Rch signal of the input signal to generate amplitude spectra a R (t, f) and a Lm (t, f) and a Rm (t, f).

[0176] <Step 33 (S33)> The receiving terminal 20 displays the corrected amplitude spectrum a of the Lch signal and the Rch signal. Lm (t, f) and a Rm (t, f), and the phase spectra φ of the Lch signal and Rch signal. L (t, f) and φ R (t, f) is converted into a time signal using IFFT, and the time signal (amplitude corrected signal) L m (t), R m We obtain (t).

[0177] <Step 34 (S34)> The receiving terminal 20 applies a composite window (or OLA window) to the Lch signal and the Rch signal, for example.

[0178] <Step 35 (S35)> The receiving terminal 20 converts the Lch signal and Rch signal after windowing into a frequency spectrum using FFT and obtains the amplitude spectrum a Lg (t, f), a Rg (t, f), and the phase spectrum φ Lg (t, f), φ Rg Calculate (t, f).

[0179] <Step 36 (S36)> The receiving terminal 20 receives the amplitude spectrum a generated in step 35. Lg (t, f) and a Rg (t, f) is, for example, in step 18 of Figure 10, the final pseudo-binaural signal L vb (t) and R vb Used to generate (t).

[0180] The above describes an example of the process for generating a pseudo-binaural signal in the receiving terminal 20.

[0181] Figure 14 is a block diagram showing an example of the internal configuration of the inter-channel phase difference changing unit 24 and the inter-channel amplitude difference changing unit 25 shown in Figure 7.

[0182] In Figure 14, the configuration of the inter-channel phase difference changing unit 24 is the same as that shown in Figure 12.

[0183] In the inter-channel amplitude difference changing unit 25 shown in Figure 14, the amplitude spectrum used to generate the final pseudo-binaural signal is not the amplitude spectrum of the signal obtained by applying a synthesis window to the signal whose phase difference has been changed in the inter-channel phase difference changing unit 24, as in Figure 12. Instead, it is the amplitude spectrum of the signal obtained by applying a synthesis window to the signal whose phase difference has not been changed in the inter-channel phase difference changing unit 24.

[0184] In the inter-channel amplitude difference changing unit 25 shown in Figure 14, the amplitude changing units 278-1 and 278-2 are positioned before the synthesis windows 271-1 and 271-2, which is different from the configuration shown in Figure 12. That is, as shown in Figure 14, the inter-channel amplitude difference changing unit 25 performs amplitude difference modification (amplitude modification of each channel) before applying the output of the FFT units 262-1 and 262-2 (amplitude spectrum of the decoded stereo signal without phase difference modification) to the synthesis window. As a result, a pseudo-binaural signal is generated using the amplitude spectrum after amplitude difference adjustment.

[0185] In the case of Figure 14, the amplitude difference change (amplitude modification) is performed in the frequency domain. Therefore, as shown in Figure 14, when the amplitude difference change (amplitude modification) is performed before the composite windows 271-1 and 271-2, the inter-channel amplitude difference change unit 25 takes the signal after the FFT as input and performs the amplitude difference change processing, and then applies the composite window after the IFFT is performed in the IFFT units 281-1 and 281-2.

[0186] In this way, the receiving terminal 20 performs IFFT processing and overlap-add processing using the phase spectrum (e.g., the output of FFT units 275-1 and 275-2) calculated by performing FFT processing again on the signal synthesized by changing the phase difference between channels (e.g., the output of synthesis windows 273-1 and 273-2), and the amplitude spectrum (e.g., the output of FFT units 274-1 and 274-2) calculated by performing FFT processing on the signal synthesized without changing the phase difference between channels (e.g., the output of synthesis windows 271-1 and 271-2), to synthesize a time signal. In other words, the receiving terminal 20 generates a pseudo-binaural signal without using the amplitude spectrum calculated by performing FFT processing on the signal synthesized by changing the phase difference between channels. This mitigates the effect of attenuation of the amplitude in the signal after applying the synthesis window due to the change in phase difference, thereby improving the accuracy of pseudo-binaural signal generation.

[0187] [Example 4 of the inter-channel phase difference changing unit 24 and inter-channel amplitude difference changing unit 25] Figure 15 is a block diagram showing an example of the internal configuration of the inter-channel phase difference changing unit 24 and inter-channel amplitude difference changing unit 25 shown in Figure 7.

[0188] In the inter-channel amplitude difference changing unit 25 shown in Figure 15, the amplitude spectrum used to generate the final pseudo-binaural signal is not the amplitude spectrum of the signal obtained by applying a synthesis window to the signal whose phase difference has been changed in the inter-channel phase difference changing unit 24, as in Figure 12. Instead, it is the amplitude spectrum of the signal obtained by applying a synthesis window to the signal whose phase difference has not been changed in the inter-channel phase difference changing unit 24.

[0189] The configuration shown in Figure 15 is basically the same as the configuration shown in Figure 12, but compared to the configuration in Figure 12, it does not include the azimuth angle estimation unit 265, the amplitude difference estimation unit 276, the amplitude difference modification unit 277, and the amplitude modification units 278-1 and 278-2. In other words, as shown in Figure 15, the inter-channel phase difference modification unit 24 does not need to perform azimuth angle estimation, and the inter-channel amplitude difference modification unit 25 does not need to perform amplitude difference modification (amplitude modification of each channel).

[0190] The receiving terminal 20 shown in Figure 15 performs IFFT processing and overlap-add processing using the phase spectrum (e.g., output of FFT units 275-1, 275-2) calculated by performing FFT processing again on the signal synthesized by changing the phase difference between channels (e.g., output of synthesis windows 273-1, 273-2), and the amplitude spectrum (e.g., output of FFT units 274-1, 274-2) calculated by performing FFT processing on the signal synthesized without changing the phase difference between channels (e.g., output of synthesis windows 271-1, 271-2), to synthesize a time signal. In other words, the receiving terminal 20 generates a pseudo-binaural signal without using the amplitude spectrum calculated by performing FFT processing on the signal synthesized by changing the phase difference between channels. This mitigates the effect of attenuation of the amplitude in the signal after applying the synthesis window due to the change in phase difference, thereby improving the accuracy of pseudo-binaural signal generation.

[0191] If the amplitude difference is not changed, the perceived azimuth angle of the sound may be smaller than the actual azimuth angle (it may tend to approach the 0-degree direction in front). To compensate for this, the receiving terminal 20 may, for example, set the amount of phase difference change in the phase difference change unit 263 to be larger. For example, the amount of phase difference change in the phase difference change unit 263 may be set to be larger when the amplitude difference is not changed than when the amplitude difference is changed (for example, when the amplitude difference between channels of the decoded stereo signal is changed).

[0192] For example, the equivalent level difference Δ(φ) with respect to the sound source direction φ is expressed by the following formula, as described in the 3GPP standard TS 26.260: Δ(φ) = ICLD(φ) + ICTD(φ) * 17.3 [dB] (4)

[0193] Here, ICLD represents the inter-channel level difference [dB], and ICTD represents the inter-channel time difference [ms]. The phase difference changing unit 263 may, for example, set a large phase difference (e.g., ICTD in equation (4)) instead of changing the amplitude difference (e.g., eliminating ICLD in equation (4)) so that the equivalent level difference Δ(φ) does not decrease (or remains unchanged) compared to the case where the amplitude difference is changed. This makes it possible to obtain the perceived direction of the sound source as accurately as when ICLD is not zero (e.g., when the amplitude difference is changed).

[0194] For example, when amplitude difference modification is performed, the distance between the ears is set to approximately 22 cm to 23 cm, whereas in the case of the receiving terminal 20 shown in Figure 15 (when amplitude difference modification is not performed), the distance between the ears may be set to about 45 cm. For example, if the distance between microphones is 15 cm, the extrapolation coefficient α defined in Figure 2 may be set to 3. In this way, by setting a larger distance between the ears when amplitude difference modification is not performed than when amplitude difference modification is performed, the amount of phase difference change (e.g., the extrapolation coefficient α) when amplitude difference modification is not performed can be increased, making it possible to achieve the above-mentioned settings.

[0195] Furthermore, while the value of the extrapolation coefficient α in the extrapolation process can be a decimal, it may be set (or limited) to an integer. Setting the value of α to an integer can improve the quality of the virtual microphone signal obtained by the extrapolation process. This is because, when the true phase of the observed signal's phase φ is φ + 2π (in the case of an observation error of 2π), if α is a decimal, the extrapolated observation error of 2π, 2πα, appears as an error. In contrast, when α is an integer, even if there is an observation error of 2π, 2πα only changes the number of phase rotations and does not affect the apparent phase φ. Note that in the case of α as defined in Figure 1, any value of α is acceptable as long as it is an integer. Also, in the case of α as defined in Figure 2, since 0.5 × (α + 1) is an integer, it is preferable to set (limit) the value of α to an odd integer.

[0196] Furthermore, since the observation error of phase φ is also extrapolated by the extrapolation coefficient α, the larger α is, the greater the observation error tends to be, degrading the quality of the resulting virtual microphone signal. For example, if the distance between microphones is about 5 cm, to extrapolate to the typical inter-ear distance of 23 cm, a value of α such as 4 or 5 is set, and if the inter-channel level difference is not manipulated, an even larger α may be set. For this reason, it is important to set the distance between microphones that make up a stereo microphone so that the value of α does not become too large. For example, the distance between microphones may be set to a threshold corresponding to the set inter-ear distance. For example, as mentioned above, if the inter-ear distance is set to about 45 cm, the distance between microphones may be set to 15 cm or more (threshold: 15 cm). When the distance between microphones is about 15 cm, even if the inter-channel level difference is not manipulated, a small value of α such as 3 is set, which can suppress the increase in the value of α. On the other hand, the longer the distance between microphones, the more frequency components there tend to be where the inter-channel phase difference exceeds 2π (the phase rotates more than once), so the target bandwidth for phase manipulation tends to become narrower. To broaden the bandwidth affected by phase manipulation, it is desirable to minimize the distance between microphones as much as possible.

[0197] The above-mentioned interaural distance, microphone distance, and extrapolation coefficient α settings are examples only; other values ​​may be used.

[0198] [Communication Terminal Configuration Example 2] In Communication Terminal Configuration Example 2, we will describe a communication terminal configuration example in which the pseudo-binaural signal generation process (virtual binauralization process) explained using Figure 4 is performed as processing (for example, pre-processing) at the transmitting terminal. In Configuration Example 2, the transmission system includes, for example, a transmitting terminal 30 and a receiving terminal 40.

[0199] <Example Configuration of Transmitting Terminal 30> Figure 16 is a block diagram showing an example configuration of the transmitting terminal 30 in Configuration Example 2. The transmitting terminal 30, for example, performs virtual binaural processing using a stereo signal and transmits encoded information that encodes the generated pseudo-binaural signal (virtual stereo signal).

[0200] In Figure 16, the transmitting terminal 30 comprises a plurality of microphones 31, an input signal selection unit 32, a virtual binaural processing unit 33, and an encoding unit 36. The virtual binaural processing unit 33 includes, for example, an inter-channel phase difference changing unit 34 and an inter-channel amplitude difference changing unit 35. The configuration of the inter-channel phase difference changing unit 34 may be the same as the inter-channel phase difference changing unit 24 shown in Figures 8, 12, 14, or 15. The configuration of the inter-channel amplitude difference changing unit 35 may be the same as the inter-channel amplitude difference changing unit 25 shown in Figures 9, 12, 14, or 15.

[0201] The input signal selection unit 32 and the virtual binaural processing unit 33 shown in Figure 16 may be included in the generation unit shown in Figure 3, for example. The transmitting terminal 30 shown in Figure 16 may include, for example, a transmitting unit (corresponding to the communication unit shown in Figure 3) that transmits a signal (e.g., encoded information) to the receiving terminal 40, or a receiving unit (corresponding to the communication unit shown in Figure 3) that receives a signal (e.g., signaling information) from the receiving terminal 40 (e.g., signaling information) (not shown).

[0202] Multiple microphones 31 supply the audio signals observed by each microphone to the input signal selection unit 32.

[0203] The input signal selection unit 32 selects two signals from the audio signals supplied by multiple microphones 31 to be used for generating a pseudo-binaural signal, and outputs the two selected signals as a stereo signal (L channel (Lch signal) and R channel (Rch signal)) to the virtual binaural processing unit 33.

[0204] Furthermore, the input signal selection unit 32 outputs information (e.g., selected microphone information) regarding the microphones that supply the two selected signals (e.g., microphones constituting a stereo microphone) to the virtual binaural processing unit 33. The selected microphone information may include, for example, at least microphone distance information indicating the distance between the two selected microphones.

[0205] The virtual binaural processing unit 33 uses, for example, the microphone distance information and stereo signals (L channel, R channel) input from the input signal selection unit 32 to perform virtual binaural processing (for example, the processing shown in Figure 4) and generate a signal from a virtual stereo microphone composed of virtual microphones (for example, a pseudo-binaural signal).

[0206] Furthermore, the virtual binaural processing unit 33 outputs mode information to the encoding unit 36, for example, indicating that the signal output to the encoding unit 36 ​​is a binaural signal and that it should be encoded in binaural encoding mode.

[0207] In the virtual binaural processing unit 33, the inter-channel phase difference changing unit 34 generates a virtual stereo microphone signal (pseudo-binaural signal) composed of virtual microphones placed on either side of the stereo microphone by extrapolating the phase spectrum of the stereo microphone signal in a frequency band where the phase difference between the signals of the microphones constituting the stereo microphone (e.g., microphone 31) is within 2π.

[0208] For example, the inter-channel phase difference changing unit 34 uses the distance information between microphones and the stereo signal to adjust (change) the phase difference of the phase spectrum in the virtual microphone (e.g., the phase spectrum of the pseudo-binaural signal) using the method described using, for example, Figures 2, 4 (steps 4, 5) or 10 (steps 13, 14) (phase extrapolation processing), and outputs the frequency spectrum (e.g., L, R) of the pseudo-binaural signal including the phase spectrum to the inter-channel amplitude difference changing unit 35. The inter-channel phase difference changing unit 34 also determines the azimuth angle θ of the sound source using the method described using, for example, Figure 4 (step 3), Figure 11 (step 22) or Figure 13 (step 30), and Figure 5 (estimation of the azimuth angle θ of the sound source), and outputs the information of the determined azimuth angle θ to the inter-channel amplitude difference changing unit 35. Note that the inter-channel phase difference changing unit 34 does not need to determine (estimate) the azimuth angle θ of the sound source, similar to the inter-channel phase difference changing unit 24 shown in Figure 15.

[0209] The transmitting terminal 30 may, for example, receive interaural distance information from the receiving terminal 40 (for example, the dashed line shown in Figure 16). When interaural distance information is received from the receiving terminal 40, the inter-channel phase difference changing unit 34 may, for example, use the ratio of the interaural distance to the microphone distance as an extrapolation coefficient α. This allows the transmitting terminal 30 to generate a pseudo-binaural signal suitable for the receiving user.

[0210] Furthermore, the inter-channel phase difference changing unit 34 may determine the frequency band to be subject to inter-channel phase difference changing processing based on the interaural distance information.

[0211] Furthermore, the inter-channel phase difference conversion unit 34 outputs the amplitude spectra of the input stereo signals (L channel, R channel) as the amplitude spectra of the pseudo-binaural signal to the inter-channel amplitude difference conversion unit 35.

[0212] In the virtual binaural processing unit 33, the inter-channel amplitude difference changing unit 35 adjusts the distance r between the microphones constituting the stereo microphone and the phase difference φ between the microphones constituting the stereo microphone in a frequency band where the phase difference of the signals between the microphones constituting the stereo microphone is within 2π. d Based on this, the azimuth angle θ of the sound source is estimated, and the amplitude difference a of the signals between the microphones constituting the stereo microphone, which is associated with the azimuth angle θ, is calculated. d Based on this, the amplitude difference between the signals from the virtual microphones that make up the virtual stereo microphone is adjusted. This generates a pseudo-binaural signal with adjusted amplitude.

[0213] For example, the inter-channel amplitude difference changing unit 35 determines (estimates) the inter-channel amplitude difference based on the azimuth angle θ input from the inter-channel phase difference changing unit 34. Using the determined inter-channel amplitude difference, the inter-channel amplitude difference changing unit 35 adjusts (changes) the amplitude difference of the frequency spectrum L (amplitude and phase) and R (amplitude and phase) of the pseudo-binaural signal input from the inter-channel phase difference changing unit 34.

[0214] Furthermore, the inter-channel amplitude difference changing unit 35 does not necessarily have to change the amplitude difference, similar to the inter-channel amplitude difference changing unit 25 shown in Figure 15.

[0215] Furthermore, the inter-channel amplitude difference changing unit 35 uses the amplitude-adjusted amplitude spectra (L channel, R channel) and the phase spectra (L channel, R channel) whose phase difference has been changed by the inter-channel phase difference changing unit 34 to convert the frequency spectrum into a time signal using IFFT or the like to generate a pseudo-binaural signal (L channel, R channel), which is then output to the encoding unit 36.

[0216] Furthermore, the inter-channel amplitude difference changing unit 35 may, for example, determine the frequency band to be subjected to amplitude adjustment processing based on the inter-aural distance information received from the receiving terminal 40 (for example, the dashed line shown in Figure 16).

[0217] In Figure 16, if the mode information input from the virtual binauralization processing unit 33 is in binaural encoding mode, the encoding unit 36 ​​encodes the pseudo-binaural signals (L channel, R channel) input from the virtual binauralization processing unit 33 in binaural encoding mode and sends the encoded binaural signals (encoded information) to the receiving terminal 40.

[0218] <Example Configuration of Receiving Terminal 40> Figure 17 is a block diagram showing an example configuration of the receiving terminal 40 in Configuration Example 2. The receiving terminal 40 receives, for example, encoded information transmitted from the transmitting terminal 30, decodes the received encoded information, and obtains a pseudo-binaural signal.

[0219] In Figure 17, the receiving terminal 40 includes a decoding unit 41. The receiving terminal 40 may also include, for example, a binaural distance information analysis and transmission unit 42.

[0220] The decoding unit 41 receives the encoded binaural signal transmitted from the transmitting terminal 30, decodes it, and outputs a decoded pseudo-binaural signal. The output pseudo-binaural signal is presented to the user's ears, for example, via headphones.

[0221] The binaural distance information analysis and transmission unit 42 extracts the user's binaural distance information using, for example, a position sensor provided on an output device such as headphones worn by the user, and sends it to the transmission terminal 30 (for example, the dashed line shown in Figure 17).

[0222] The above describes an example of the configuration of a communication terminal according to one non-limiting embodiment of the present disclosure.

[0223] Thus, in this embodiment, the communication terminal generates a virtual stereo microphone signal (e.g., a pseudo-binaural signal) composed of virtual microphones placed on both sides of the stereo microphone by performing virtual binaural processing (e.g., phase difference modification or amplitude difference modification) on the signal of the actual microphones constituting the stereo microphone.

[0224] For example, in phase difference shifting, the communication terminal generates a pseudo-binaural signal by performing phase extrapolation in the frequency band where the phase difference between the signals from the actual microphones falls within a range of 2π, and by not performing phase extrapolation in frequency bands other than the frequency band where the phase difference falls within a range of 2π. In this way, since extrapolation is not performed in high-frequency components where the phase difference between the signals from the actual microphones tends to exceed 2π (for example, frequency components where it is difficult to accurately identify the phase difference), the communication terminal can improve the accuracy of estimating the inter-channel phase difference by extrapolation.

[0225] Furthermore, for example, in amplitude difference modification, the communication terminal estimates the azimuth angle of the direction of arrival of the sound source signal by phase difference analysis, and applies the interaural amplitude difference corresponding to the estimated azimuth angle to the pseudo-binaural signal. This allows the communication terminal to generate a pseudo-binaural signal that takes into account amplitude attenuation due to shielding (or diffraction) that may occur, for example, when a nearby sound source is located in the shadow of the head (or auricle).

[0226] This allows for the reduction of the difference between the binaural signal obtained using an actual binaural microphone and the binaural signal obtained using a virtual binaural microphone when, for example, extrapolation is used to extend the distance between microphones to the distance between both ears to generate a virtual binaural microphone, thereby improving the quality of the pseudo-binaural signal. Therefore, according to one embodiment of this disclosure, the quality of the binaural signal can be improved when virtual microphones are placed on both sides of the actual microphones.

[0227] For example, even when the distance between microphones is short compared to the distance between both ears, such as when a mobile device (e.g., a smartphone) is equipped with a stereo microphone, the user can obtain a realistic sense of three-dimensionality by listening to the pseudo-binaural signal generated using the stereo signal input from the stereo microphone through headphones. Therefore, according to one embodiment of this disclosure, binaural audio communication using a smartphone's built-in microphone becomes possible, for example.

[0228] Furthermore, in this embodiment, the communication terminal applies a composite window to the signal generated by changing the phase difference between channels of the decoded stereo signal obtained by decoding the stereo encoding information, and obtains a phase spectrum (for example, φ Lv , φ Rv ) and the amplitude spectrum obtained by applying a synthesis window to the decoded stereo signal without changing the phase difference (for example, a Lg a Rg ) is subjected to IFFT and overlap-add processing to generate a pseudo-binaural signal.

[0229] As a result, even if the energy (e.g., amplitude spectrum) of the composite signal may be attenuated when a composite window or overlap-add processing is applied to the signal after phase difference modification, the communication terminal can generate a pseudo-binaural signal using the amplitude spectrum of the signal obtained by applying a composite window to a signal that has not undergone phase difference modification. This suppresses the effects of amplitude attenuation due to phase difference modification and improves the accuracy of pseudo-binaural signal generation.

[0230] An embodiment of the present disclosure has been described above.

[0231] The above example describes a case where an audio signal (e.g., a pseudo-binaural signal) is presented to the user's ears via headphones or the like in a communication terminal, but it is not limited to this. For example, the audio signal may be played back by a speaker instead of headphones. When playing back the audio signal by a speaker, the communication terminal may play back the output signal of the stereo encoding unit as is without performing virtual binaural processing, or it may apply other processing different from virtual binaural processing (e.g., conversion processing to an arbitrary virtual microphone distance different from the inter-ear distance).

[0232] Furthermore, while the above example described a case where the communication terminal performs both phase difference modification and amplitude difference modification during virtual binaural processing, it is not limited to this case. The communication terminal may perform either phase difference modification or amplitude difference modification, but not the other. Even in this case, the quality of the binaural signal can be improved.

[0233] Furthermore, the virtual microphone is not limited to being positioned symmetrically with respect to the physical microphone; it may also be positioned asymmetrically on both sides of the physical microphone.

[0234] Furthermore, while the above example described the generation of binaural signals using a virtual microphone, the signals virtually generated using a virtual microphone are not limited to binaural signals; other signals may also be used. For example, the distance between virtual microphones that constitute a virtual stereo microphone is not limited to the distance between the ears.

[0235] Furthermore, for operations related to parts of the communication terminal configuration described above that are not explicitly stated, operations based on other existing encoding schemes (e.g., stereo encoding schemes) may be performed.

[0236] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. Furthermore, the components in each of the embodiments described above may be combined in any way.

[0237] Furthermore, the notation "...part" in the above-described embodiment may be replaced with other notations such as "...circuit," "...device," "...unit," or "...module."

[0238] This disclosure can be implemented in software, hardware, or software in conjunction with hardware. Each functional block used in the description of the above embodiments may be implemented in part or in whole as an integrated circuit (LSI), and each process described in the above embodiments may be controlled in part or in whole by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0239] The integrated circuit implementation method is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, a Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0240] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0241] This disclosure is applicable to all types of devices, systems, and equipment having communication capabilities (collectively referred to as communication equipment). Communication equipment may include a radio transceiver and a processing / control circuit. The radio transceiver may include a receiver and a transmitter, or both as functions. The radio transceiver (transmitter, receiver) may include an RF (Radio Frequency) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or similar. Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned devices.

[0242] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting fixtures, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0243] Communication includes data communication via cellular systems, wireless LAN (Local Area Network) systems, and communication satellite systems, as well as data communication using combinations of these.

[0244] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0245] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0246] A communication device according to one embodiment of the present disclosure comprises a receiving circuit that receives encoded information obtained by encoding a signal from a stereo microphone, and a generating circuit that generates a virtual stereo microphone signal by performing an inverse Fourier transform and overlap-add processing on a phase spectrum obtained by applying a composite window to a signal generated by changing the phase difference between channels of a decoded stereo signal obtained by decoding the encoded information, and an amplitude spectrum obtained by applying a composite window to the decoded stereo signal without changing the phase difference.

[0247] In one embodiment of the present disclosure, the generation circuit modifies the amplitude difference between channels of the decoded stereo signal and generates the amplitude spectrum using the decoded stereo signal after the amplitude difference has been modified.

[0248] In one embodiment of the present disclosure, the generation circuit estimates the azimuth angle of the sound source based on the distance between the microphones constituting the stereo microphone and the phase difference, and modifies the amplitude of the decoded stereo signal based on the amplitude difference corresponding to the azimuth angle.

[0249] In one embodiment of the present disclosure, the generation circuit estimates the azimuth angle based on the distance between the microphones, the phase difference, and the amplitude spectrum.

[0250] In one embodiment of the present disclosure, the generation circuit modifies the amplitude of the decoded stereo signal before applying the decoded stereo signal to the synthesis window.

[0251] In one embodiment of the present disclosure, the generation circuit modifies the amplitude of the decoded stereo signal after applying the decoded stereo signal to the synthesis window.

[0252] In one embodiment of the present disclosure, the generation circuit modifies the phase difference by extrapolating the phase spectrum of the decoded stereo signal.

[0253] In one embodiment of the present disclosure, the amount of change in the phase difference is greater when the amplitude difference between channels of the decoded stereo signal is not changed than when the amplitude difference between channels is changed.

[0254] In one embodiment of the present disclosure, the set interaural distance is greater when the amplitude difference between channels of the decoded stereo signal is not changed than when the amplitude difference between channels is changed.

[0255] In one embodiment of the present disclosure, the distance between the microphones constituting the stereo microphone is greater than or equal to a threshold corresponding to the distance between the two ears.

[0256] In one embodiment of the present disclosure, the extrapolation coefficient in the extrapolation process is an integer value.

[0257] In a signal generation method according to one embodiment of the present disclosure, the communication device receives encoded information obtained by encoding a signal from a stereo microphone, and generates a virtual stereo microphone signal by performing an inverse Fourier transform and overlap-add processing on the phase spectrum obtained by applying a composite window to a signal generated by changing the phase difference between channels of a decoded stereo signal obtained by decoding the encoded information, and on the amplitude spectrum obtained by applying a composite window to the decoded stereo signal without changing the phase difference.

[0258] All disclosures in the specification, drawings, and abstract contained in the Japanese application No. 2025-052253, filed on March 26, 2025, are incorporated herein by reference.

[0259] One embodiment of this disclosure is useful for audio signal transmission systems and the like.

[0260] 10, 30 Transmitting terminal 11, 31 Microphone 12, 32 Input signal selection unit 13 Microphone distance information transmission unit 14, 36 Encoding unit 20, 40 Receiving terminal 21 Microphone distance information receiving unit 22, 41 Decoding unit 23, 33 Virtual binaural processing unit 24, 34 Inter-channel phase difference modification unit 25, 35 Inter-channel amplitude difference modification unit 42 Inter-ear distance information analysis and transmission unit 241 Time / frequency conversion unit 242 Phase difference calculation unit 243, 263 Phase difference modification unit 244, 265 Azimuth angle estimation unit 251, 276 Amplitude difference estimation unit 252, 277 Amplitude difference modification unit 253 Frequency / time conversion unit 261 Analysis window 262, 274, 275 FFT unit 264 Phase modification unit 271, 273 Composite window 272, 279, 281 IFFT section 278 Amplitude change section 280 OLA section

Claims

A receiving circuit that receives encoded information obtained by encoding the signal from a stereo microphone, A generation circuit that generates a virtual stereo microphone signal by performing an inverse Fourier transform and overlap-add processing on a phase spectrum obtained by applying a composite window to a signal generated by changing the phase difference between channels of the decoded stereo signal obtained by decoding the aforementioned encoded information, and on an amplitude spectrum obtained by applying a composite window to the decoded stereo signal without changing the phase difference. A communication device equipped with the following. 　 The generation circuit modifies the amplitude difference between channels of the decoded stereo signal and generates the amplitude spectrum using the decoded stereo signal after the amplitude difference has been modified. The communication device according to claim 1. 　 The generation circuit estimates the azimuth angle of the sound source based on the distance between the microphones constituting the stereo microphone and the phase difference, and modifies the amplitude of the decoded stereo signal based on the amplitude difference corresponding to the azimuth angle. The communication device according to claim 2. 　 The generation circuit estimates the azimuth angle based on the distance between the microphones, the phase difference, and the amplitude spectrum. The communication device according to claim 3. 　 The generation circuit modifies the amplitude of the decoded stereo signal before applying the decoded stereo signal to the synthesis window. The communication device according to claim 2. 　 The generation circuit modifies the amplitude of the decoded stereo signal after applying the decoded stereo signal to the synthesis window. The communication device according to claim 2. 　 The generation circuit modifies the phase difference by extrapolating the phase spectrum of the decoded stereo signal. The communication device according to claim 1. 　 The amount of change in the phase difference is greater when the amplitude difference between channels of the decoded stereo signal is not changed than when the amplitude difference is not changed. The communication device according to claim 7. 　 The set interaural distance is greater when the amplitude difference between channels of the decoded stereo signal is not changed than when the amplitude difference is changed. The communication device according to claim 7. 　 The distance between the microphones constituting the stereo microphone is greater than or equal to a threshold corresponding to the distance between the two ears. The communication device according to claim 9. 　 The extrapolation coefficients in the above extrapolation process are integer values. The communication device according to claim 7. 　 Communication equipment, Receive encoded information obtained by encoding the signal from a stereo microphone. The following steps are performed to generate a virtual stereo microphone signal: an inverse Fourier transform and overlap-add processing are applied to the phase spectrum obtained by applying a composite window to a signal generated by changing the phase difference between channels of the decoded stereo signal obtained by decoding the aforementioned encoded information, and an amplitude spectrum obtained by applying a composite window to the decoded stereo signal without changing the phase difference. Signal generation method.

Images