Methods, apparatus and systems for estimation and correction of acoustic properties in a flexibly rendered system

WO2026187597A1PCT designated stage Publication Date: 2026-09-10DOLBY LABORATORIES LICENSING CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/017014
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2026-01-29
Filing Date
2026-02-27
Publication Date
2026-09-10

Smart Images

  • Figure US2026017014_10092026_PF_FP_ABST
    Figure US2026017014_10092026_PF_FP_ABST
Patent Text Reader

Abstract

Some audio processing methods may involve determining, based at least in part on microphone signals obtained by a microphone of a hand-held sensing device, level data corresponding to both direct acoustic paths and indirect acoustic paths of sound corresponding to one or more audio calibration signals emitted by each loudspeaker of a plurality of loudspeakers, and calibrating an audio data rendering process for audio played back by the plurality of loudspeakers. The audio data rendering process may be based, at least in part, on the level data. The calibrating may involve determining one or more loudness correction gains and performing a correction for audio reproduced by at least one loudspeaker of the plurality of loudspeakers, the correction being based, at least in part, on the loudness correction gains.
Need to check novelty before this filing date? Find Prior Art

Description

D25030WO01 METHODS, APPARATUS AND SYSTEMS FOR ESTIMATION AND CORRECTION OF ACOUSTIC PROPERTIES IN A FLEXIBLY RENDERED SYSTEMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U. S. Provisional Patent Applications 63 / 971,065 filed January 29, 2026 and 63 / 767,227 filed March 5, 2025, each of which is hereby incorporated by reference in their entireties.TECHNICAL FIELD

[0002] This disclosure pertains to audio processing systems and methods.BACKGROUND

[0003] Audio devices and systems are widely deployed. Although existing systems and methods for loudspeaker mapping and flexible rendering are known, improved systems and methods would be desirable.NOTATION AND NOMENCLATURE

[0004] Throughout this disclosure, including in the claims, the terms “speaker,” “loudspeaker” and “audio reproduction transducer” are used synonymously to denote any sound-emitting transducer (or set of transducers). A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single, common speaker feed or by multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuitry branches coupled to the different transducers.

[0005] Throughout this disclosure, including in the claims, the expression performing an operation “on” a signal or data (e.g., filtering, scaling, transforming, or applying gain to, the signal or data) is used in a broad sense to denote performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or pre-processing prior to performance of the operation thereon).

[0006] Throughout this disclosure including in the claims, the expression “system” is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, inD25030W001 which the subsystem generates M of the inputs and the other X - M inputs are received from an external source) may also be referred to as a decoder system.

[0007] Throughout this disclosure including in the claims, the term “processor” is used in a broad sense to denote a system or device programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chip set), a digital signal processor programmed and / or otherwise configured to perform pipelined processing on audio or other sound data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chip set.

[0008] Throughout this disclosure including in the claims, the term “couples” or “coupled” is used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections.

[0009] As used herein, a “smart device” is an electronic device, generally configured for communication with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near-field communication, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., that can operate to some extent interactively and / or autonomously. Several notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smartwatches, smart bands, smart key chains and smart audio devices. The term “smart device” may also refer to a device that exhibits some properties of ubiquitous computing, such as artificial intelligence.

[0010] Herein, we use the expression “smart audio device” to denote a smart device which is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of virtual assistant functionality). A singlepurpose audio device is a device (e.g., a television (TV)) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker and / or at least one camera), and which is designed largely or primarily to achieve a single purpose. For example, although a TV typically can play (and is thought of as being capable of playing) audio from program material, in most instances a modern TV runs some operating system on which applications run locally, including the application of watching television. In this sense, a single-purpose audio device having speaker(s) and microphone(s) is often configured to run a local application and / or service to use the speaker(s) and microphone(s) directly. Some singlepurpose audio devices may be configured to group together to achieve playing of audio over a zone or user configured area.D25030W001

[0011] One common type of multi-purpose audio device is an audio device that implements at least some aspects of virtual assistant functionality, although other aspects of virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multi-purpose audio device is configured for communication. Such a multi-purpose audio device may be referred to herein as a “virtual assistant.” A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide an ability to utilize multiple devices (distinct from the virtual assistant) for applications that are in a sense cloud-enabled or otherwise not completely implemented in or on the virtual assistant itself. In other words, at least some aspects of virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which a virtual assistant may communication via a network, such as the Internet. Virtual assistants may sometimes work together, e.g., in a discrete and conditionally defined way. For example, two or more virtual assistants may work together in the sense that one of them, e.g., the one which is most confident that it has heard a wakeword, responds to the wakeword. The connected virtual assistants may, in some implementations, form a sort of constellation, which may be managed by one main application which may be (or implement) a virtual assistant.

[0012] Herein, “wakeword” is used in a broad sense to denote any sound (e.g., a word uttered by a human, or some other sound), where a smart audio device is configured to awake in response to detection of (“hearing”) the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, to “awake” denotes that the device enters a state in which it awaits (in other words, is listening for) a sound command. In some instances, what may be referred to herein as a “wakeword” may include more than one word, e.g., a phrase.

[0013] Herein, the expression “wakeword detector” denotes a device configured (or software that includes instructions for configuring a device) to search continuously for alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wakeword event is triggered whenever it is determined by a wakeword detector that the probability that a wakeword has been detected exceeds a predefined threshold. For example, the threshold may be a predetermined threshold which is tuned to give a reasonable compromise between rates of false acceptance and false rejection. Following a wakeword event, a device might enter a state (which may be referred to as an “awakened” state or a stateD25030W001 of “attentiveness”) in which it listens for a command and passes on a received command to a larger, more computationally-intensive recognizer.

[0014] As used herein, the terms “program stream” and “content stream” refer to a collection of one or more audio signals, and in some instances video signals, at least portions of which are meant to be heard together. Examples include a selection of music, a movie soundtrack, a movie, a television program, the audio portion of a television program, a podcast, a live voice call, a synthesized voice response from a smart assistant, etc. In some instances, the content stream may include multiple versions of at least a portion of the audio signals, e.g., the same dialogue in more than one language. In such instances, only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is intended to be reproduced at one time.

[0015] As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “or” is to be read as “and / or” unless the context clearly indicates otherwise. The term “based on” is to be read as “based at least in part on.” The term “one example implementation” and “an example implementation” are to be read as “at least one example implementation.” The term “another implementation” is to be read as “at least one other implementation.” The terms “determined,” “determines,” or “determining” are to be read as obtaining, receiving, computing, calculating, estimating, predicting, or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0016] Moreover, as used herein the following acronyms have the following meanings:• dB: Decibel• dBSPL: Sound Pressure Level in Decibel units• DSSS: direct sequence spread spectrum• DW: delay waveform• SPL: Sound Pressure LevelSUMMARY

[0017] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some instances, the method(s) may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some audio processing methods may involve determining, based at least in part on microphone signals obtained by a microphone of a hand-held sensing device, levelD25030W001 data corresponding to both direct acoustic paths and indirect acoustic paths of sound corresponding to one or more audio calibration signals emitted by each loudspeaker of a plurality of loudspeakers, and calibrating an audio data rendering process for audio played back by the plurality of loudspeakers. The audio data rendering process may be based, at least in part, on the level data. The calibrating may involve determining one or more loudness correction gains and performing a correction for audio reproduced by at least one loudspeaker of the plurality of loudspeakers, the correction being based, at least in part, on the loudness correction gains.

[0018] In some examples, the one or more audio calibration signals may not be audible to humans. According to some examples, the one or more audio calibration signals may utilize a Code Division Multiple Access scheme. In some examples, the one or more audio calibration signals may include direct sequence spread spectrum (DSSS) signals utilizing orthogonal spreading codes.

[0019] According to some examples, the level data may be determined from a delay waveform from a matched filter. In some examples, determining one or more loudness correction gains may be based, in part, on one or more stored loudspeaker device descriptors indicating measured loudspeaker response.

[0020] In some examples, the correction may be, or may include, a single broadband gain per speaker. According to some examples, the correction may be, or may include, a frequency-dependent gain per speaker. In some examples, the correction may be constrained to a loudness correction threshold.

[0021] Some methods may involve determining, by the control system, a rendering quality factor based at least in part on a comparison of the level data of a peak corresponding to one of the direct acoustic paths with level data of a peak corresponding to one or more of the indirect acoustic paths, a comparison of the level data of the peak corresponding to one of the direct acoustic paths with noise level data, or both. According to some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) level data of one or more peaks corresponding to one or more of the indirect acoustic paths of sound from the loudspeaker. In some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker. According to some examples, for each loudspeaker, determining the renderingD25030W001 quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of the level data of the peak corresponding to the direct acoustic path and level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker. In some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a noise level data. According to some examples, the comparison may be based on level data in a delay waveform.

[0022] Some methods may involve using, by the control system, the rendering quality factor to alter the audio data rendering process. Some methods may involve using, by the control system, the rendering quality factor for configuring a dialog enhancer. Some methods may involve using, by the control system, the rendering quality factor for modifying an audio object render. Some methods may involve using, by the control system, the rendering quality factor for selecting a different configuration within a codec bitstream. In some examples, selecting a different configuration within a codec bitstream may involve one or more of selecting a dialog enhancement presentation or selecting a channel-based render instead of an audio object-based render. Some methods may involve controlling, by the control system, a user interface to provide a user prompt based, at least in part, on the rendering quality factor.

[0023] In some examples, the one or more calibration signals may be simultaneously emitted by two or more loudspeakers of the plurality of loudspeakers. According to some examples, the microphone signals may be obtained while the hand-held sensing device is being pointed at each loudspeaker of the plurality of loudspeakers. In some examples, a frequency range of the audio calibration signals may vary over time. According to some examples, each loudness correction gain of the one or more loudness correction gains may be based on a relative loudness estimate. The relative loudness estimate may be a loudness estimate relative to the loudness of a reference loudspeaker. The relative loudness estimate may be based, at least in part, on a signal level estimate of a direct arrival peak. The relative loudness estimate may be based, at least in part, on the signal level estimate minus a noise estimate. Some methods may involve generating the audio calibration signals or audio calibration signal parameters and providing the audio calibration signals or the audio calibration signal parameters to each loudspeaker of the plurality of loudspeakers.

[0024] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus is, or includes, an audioD25030W001 processing system having an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.

[0025] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable, non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more non-transitory media having software stored thereon.

[0026] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A shows an example of an audio environment.

[0028] Figure 1B is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0029] Figure 2A shows examples of direct and indirect acoustic paths in a playback environment.

[0030] Figure 2B shows examples of an audio DSSS signal and an audio playback signal.

[0031] Figure 2C is a graph that shows examples of the powers of two DSSS signals with different bandwidths but located at the same central frequency.

[0032] Figure 3A shows an example of a delay waveform (DW).

[0033] Figure 3B shows an example of another DW.

[0034] Figure 4 shows an illustrative autocorrelation of an order 11 Gold Code.

[0035] Figure 5 shows example components of system that is configured to generate a full set of coefficients that may be used by a flexible Tenderer during a rendering process.D25030W001

[0036] Figure 6 illustrates example components of a system that includes a Tenderer for rendering loudness-corrected speaker feeds using the rendering coefficients generated by the rendering coefficient generator of Figure 5.

[0037] Figure 7 illustrates an example of a system that is configured to compensate for the orientation of individual drivers within a loudspeaker enclosure based upon data in the loudspeaker device descriptor.

[0038] Figure 8A is a flow diagram that outlines blocks of a disclosed method of determining a rendering quality factor according to some examples.

[0039] Figure 8B shows details of block 803 of Figure 8A according to some examples.

[0040] Figure 9 shows an example of a normalized delay waveform corresponding to the measurement of a speaker in ideal conditions.

[0041] Figure 10 shows an example of a normalized delay waveform corresponding to the measurement of a speaker that is pointing away from a listening position.

[0042] Figure 11 illustrates an example of an effect of furniture on delay waveforms.

[0043] Figure 12 illustrates an example of an effect of loudspeaker occlusion on delay waveforms.

[0044] Figure 13 shows an example of the effects on delay waveforms when a speaker is placed in a room that is different from the room in which the microphone (or listener) is.

[0045] Figure 14 is a flow diagram that outlines another example of a disclosed method.DETAILED DESCRIPTION OF EMBODIMENTS

[0046] Flexible rendering is a playback solution which delivers immersive audio experiences from a constellation of speakers that can be flexibly placed around the room, not necessarily conforming to a canonical surround sound layout such as Dolby 5.1 or 7.1. A larger number of speakers allows for greater immersion, because the spatiality of the media presentation may be leveraged. To configure a flexible Tenderer, a map needs to be created that describes the layout of the speakers and optionally the position of a listener.

[0047] Some previously-deployed acoustic mapping solutions for flexible rendering have focused on the use of microphones within the speakers, such as in the case of smart speakers, to build a map of a loudspeaker constellation instead of being listener-centric. Such solutions are constrained in their scope of support to loudspeakers with in-built microphones, such as smart speakers.

[0048] This disclosure provides listener-centric acoustic mapping solutions for flexible rendering. The methods disclosed herein do not require any sound capturing apparatus withinD25030W001 the loudspeakers, thereby expanding the scope of mapping beyond smart speakers to all loudspeakers in general.

[0049] One requirement of accurate flexible rendering for varying loudspeaker types is loudness matching between different speaker types. Relative loudness of speakers can be defined as the Sound Pressure Level (SPL) measurement at a fixed distance with a particular signal being input into the signal chain. SPL measurements, and data from which SPL may be estimated, may be referred to herein as examples of “level data.” As used herein, the term “level data” includes, for example, the amplitude (also referred to herein as magnitude), power, and / or energy of the peaks of delay waveforms that are based on audio calibration signals, such as DSSS audio calibration signals. In some examples, “level data” may include the amplitude, power and / or energy of samples in the noise region, direct response peaks, or peaks in the indirect response region of a delay waveform.

[0050] Loudness matching at the listener’s location is generally thought of as adjusting signals that speakers play to the listener such that the effect of factors such as the gain of a speaker’s amplifier and the speaker’s sensitivity are minimized. This can be done by applying a loudness correction gain to each speaker feed. The loudness correction gain may be a multi frequency band adjustment.

[0051] It should be noted that in the rest of this document “sensitivity” is used to mean the combination of factors that make up the combination of a speaker driver, amplifier and in some case an implicit fixed distance and not be limited to “speaker sensitivity” in its traditional sense. In other words, the term “sensitivity” can be used interchangeably to relative loudness.

[0052] There are numerous methods for calculating a loudness correction gain adjustment. These methods vary between “online” and “offline” methods as these terms as used herein. Online loudness correction gain adjustment methods involve performing measurements in the playback environment, such as in a listener’s home.

[0053] Offline loudness correction gain adjustment methods involve measuring the relative loudness of different speaker types at “tuning time” and storing the resulting relative loudness measurements in what may be referred to herein as a “device descriptor” or a “loudspeaker device descriptor.” The loudspeaker device descriptor may, for example, be stored on a memory of each loudspeaker — or in another memory — after the loudspeaker’s relative loudness has been determined (e.g., measured). “Tuning time” in this context may be during the tuning of a representative device, the performance of which is deemed to be of the desired quality for the product, around which an expected variance in performance may alsoD25030W001 be acceptable. In some examples, “tuning time” may be a time of a factory calibration process at manufacturing time. One example of measuring the relative loudness is that a speaker is placed in an anechoic room and a known pink noise signal of a particular amplitude is played out of the speaker. In some such examples, the loudness of the speaker may be measured with an A- weighted SPL meter at Im directly in front of the speaker. The measured SPL may be stored in the loudspeaker device descriptor. This measured speaker loudness in the rest of the document is referred to as the device sensitivity.

[0054] In the playback environment, at mapping time, the loudspeaker device descriptors are obtained by a control system for evaluation. In some examples, the device descriptors may be exchanged between the speakers in a constellation. Based on the device descriptors, the control system may calculate relative gains as the differences between the gains for a reference speaker in the constellation and the gains for that particular speaker. This process may be repeated for all speakers. The reference speaker is in some implementations chosen to be the speaker with the largest SPL value. This ensures that adding a speaker to the constellation does not result in the constellation becoming quieter. The calculated relative gains may be used to determine the speaker feed signals for each loudspeaker during a subsequent rendering process. The calculated relative gains may be modified as disclosed herein to determine the final speaker feed signals for each loudspeaker.

[0055] The offline speaker loudness measurement requires controlled acoustic conditions within a specially treated room. The effectiveness of loudness matching of speakers can be affected by differences in tuning measurements between different specially treated rooms.

[0056] Furthermore, since the offline loudness matching is done at tuning time, the orientation of the devices, differences in manufacturing tolerances, furniture in the playback environment, and the reflectivity of surfaces in the playback environment all have an impact on the loudness perceived at the listener’s location.

[0057] In some implementations the device descriptor also contains a target curve that specifies the approximate frequency response of the loudspeaker. In some implementations this target curve can also be matched between devices. However, the target curve does not take into account the effects of the playback environment on the loudness perceived at the listener’s location.

[0058] Further online methods are described herein for adjusting the rendering based upon measurements in a playback environment.D25030W001

[0059] To configure a flexible renderer a map needs to be created that describes the layout of the speakers and optionally the position of the listener(s). The combination of the device descriptors described above, and the map are used for configuring the flexible renderer.

[0060] Figure 1A shows an example of an audio environment. As with other figures provided herein, the types and numbers of elements shown in Figure 1 A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, other audio environments 115 may include more than three loudspeakers.

[0061] Figure 1A shows an example of a listener-centric, semi-automatic calibration system in an audio environment 115. In this example, a user 105 — also referred to herein as a person or a listener — is holding a mobile device 100 that includes a directi onally-sensitive sensor system 130 and a microphone system 140. The mobile device 100 is an example of what may be referred to herein as a “hand-held sensing device.” The mobile device 100 may be cellular telephone, a remote control device, a tablet device, or another type of mobile device. The sensor system 130 may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more accelerometers — which may collectively be referred to an inertial measurement unit (IMU) — one or more cameras, one or more radios or combinations thereof. In some examples, the sensor system 130 may be configured for light detection and ranging (LiDAR).

[0062] According to this example, the user 105 provides input 101 to the mobile device 100 to initiate a calibration process. Calibration signals 120A-C then play out from all loudspeakers 110A-C in the constellation. In some examples, the calibration signals 120A-C may be audio signals that are included with rendered audio content that is played back by the loudspeakers 110A-C. Simultaneously, sound data — including the calibration signals 120A-C — is collected by the microphone system 140. In this example, the user 105 is required to reposition (rotate and / or translate, as indicated by the arrow 102) the mobile device 100 for the collection of azimuth data. According to some examples, the mobile device 100 may provide a prompt — such as an audio prompt and / or a visual prompt — to the user 105 indicating when to point the mobile device 100 to each of the loudspeakers during the calibration process. In some examples, the mobile device 100 may indicate the progress of the calibration process, e.g., by displaying a graphical user interface (GUI) that indicates the estimated time remaining for obtaining calibration signals from a particular loudspeaker, the percentage of the calibration process that has been completed for a particular loudspeaker, etc.D25030W001

[0063] Collecting the azimuth data may involve obtaining azimuth angles for each of the loudspeakers 110A-C in a loudspeaker plane, in a listener plane, etc. In the example shown in Figure 1 A, the azimuth angle ©A corresponding to the loudspeaker l00A is measured relative to the y axis of a coordinate system having x and y axes parallel to the floor of the audio environment 115 and having its origin inside of the user 105. In other examples, the azimuth angles may be measured relative to the x axis or to another axis. In some such examples, the mobile device 100 may include a user interface system. A control system of the mobile device 100 may be configured to prompt the user 105, via the user interface system, to position and re-position the mobile device 100 for the collection of azimuth data.

[0064] Various examples of calibration processes are disclosed in detail herein. In some examples, a control system of the mobile device 100 may be configured to perform some or all of the calibration process(es). Alternatively, or additionally, one or more other devices may be configured to perform some or all of the calibration process(es). In some such examples, one or more servers or one or more other devices, such as a television, a laptop computer or a smart home hub, may be configured to perform some or all of the calibration process(es), based at least in part on data obtained by the mobile device 100.

[0065] Figure 1B is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1B are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 100 may be configured for performing at least some of the methods disclosed herein. In some implementations, the apparatus 100 may be, or may include, one or more components of an audio system. For example, the apparatus 100 may be an audio device, such as a smart audio device, in some implementations. In other examples, the apparatus 100 may be a mobile device (such as a cellular telephone or a remote control), a laptop computer, a tablet device, a television or another type of device. The mobile device 100 of Figure 1A is an instance of the apparatus 100 of Figure 1B.

[0066] According to some examples, the audio environment 100 of Figure 1A may include an orchestrating device, such as what may be referred to herein as a smart home hub. The smart home hub (or other orchestrating device) may be an instance of the apparatus 100. In some implementations, one or more of the loudspeakers 110A-110C may be capable of functioning as an orchestrating device.D25030W001

[0067] According to some alternative implementations the apparatus 100 may be, or may include, a server. In some such examples, the apparatus 100 may be, or may include, an encoder. Accordingly, in some instances the apparatus 100 may be a device that is configured for use within an audio environment, such as a home audio environment, whereas in other instances the apparatus 100 may be a device that is configured for use in “the cloud,” e.g., a server.

[0068] In this example, the apparatus 100 includes a sensor system 130, a microphone system 140 and a control system 150. The microphone system 140 includes one or more microphones. According to some examples, the microphone system 140 includes an array of microphones. The array of microphones may, in some instances, be configured for receiveside beamforming, e.g., according to instructions from the control system 150. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to instructions from the control system 150. Alternatively, or additionally, the control system 150 may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to microphone signals received from the microphone system 140.

[0069] In some implementations, the control system 150 may be configured for performing, at least in part, the methods disclosed herein. The control system 150 may, for example, include a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0070] In some implementations, the control system 150 may reside in more than one device. For example, in some implementations a portion of the control system 150 may reside in a device within one of the environments depicted herein and another portion of the control system 150 may reside in a device that is outside the environment, such as a server, a mobile device (e.g., a smartphone or a tablet computer), etc. In other examples, a portion of the control system 150 may reside in a device within one of the environments depicted herein and another portion of the control system 150 may reside in one or more other devices of the environment. For example, control system functionality may be distributed across multiple smart audio devices of an environment, or may be shared by an orchestrating device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 150 may reside in a device that is implementing a cloud-based service, such as a server, and another portion of the control system 150 mayD25030W001 reside in another device that is implementing the cloud-based service, such as another server, a memory device, etc. The interface system 145 also may, in some examples, reside in more than one device.

[0071] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 155 shown in Figure 1B and / or in the control system 150. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may, for example, be executable by one or more components of a control system such as the control system 150 of Figure 1B.

[0072] The sensor system 130 may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more cameras, one or more radios or combinations thereof. In some implementations, the sensor system 130 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some implementations, the sensor system 130 may include one or more cameras. In some implementations, the cameras may be free-standing cameras. In some examples, one or more cameras of the sensor system 130 may reside in a smart audio device, which may be a single purpose audio device or a virtual assistant. In some such examples, one or more cameras of the sensor system 130 may reside in a television, a mobile phone or a smart speaker. In some examples, the apparatus 100 may be configured to receive sensor data for one or more sensors residing in one or more other devices in an audio environment via the interface system 145.

[0073] The interface system 145 — when present — may, in some implementations, include a wired or wireless interface that is configured for communication with one or more other devices of an audio environment. The audio environment may, in some examples, be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. The interface system 145 may, in some implementations, be configured for exchanging control information and associated data with audio devices of the audio environment. The control information and associated data may,D25030W001 in some examples, pertain to one or more software applications that the apparatus 100 is executing.

[0074] The interface system 145 may, in some implementations, be configured for receiving, or for providing, a content stream. The content stream may include audio data. The audio data may include, but may not be limited to, audio signals. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. Metadata may, for example, have been provided by what may be referred to herein as an “encoder.” In some examples, the content stream may include video data and audio data corresponding to the video data.

[0075] The interface system 145 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 145 may include one or more wireless interfaces, e.g., configured for Wi-Fi or Bluetooth™ communication.

[0076] The interface system 145 may, in some examples, include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 145 may include one or more interfaces between the control system 150 and a memory system, such as the optional memory system 155 shown in Figure 1B. However, the control system 150 may include a memory system in some instances. The interface system 145 may, in some implementations, be configured for receiving input from one or more microphones in an environment.

[0077] In some implementations, the apparatus 100 may include the display system 160 shown in Figure 1B. The display system 160 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the display system 160 may include one or more organic light-emitting diode (OLED) displays. In some examples, the display system 160 may include one or more displays of a smart audio device. In other examples, the display system 160 may include a television display, a laptop display, a mobile device display, or another type of display. In some examples wherein the apparatus 100 includes the display system 160, the sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 160. According to some such implementations, the control system 150 may be configured for controlling the display system 160 to present one or more graphical user interfaces (GUIs). In some examples, a user interface system of the apparatus 100 may include the display system 160, a touch sensor system and / or a gesture sensor system proximate one or more displays ofD25030W001 the display system 160, the microphone system 140, the loudspeaker system 110, or combinations thereof.

[0078] According to some implementations, the apparatus 100 may include the optional loudspeaker system 110 shown in Figure 1B. The optional loudspeaker system 110 may include one or more loudspeakers, which also may be referred to herein as “speakers” or, more generally, as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), the apparatus 100 may not include a loudspeaker system 110.

[0079] According to some such examples the apparatus 100 may be, or may include, a smart audio device. In some such implementations the apparatus 100 may be, or may include, a wakeword detector. For example, the apparatus 100 may be, or may include, a virtual assistant.

[0080] Direct and Indirect Acoustic Paths

[0081] Figure 2A shows examples of direct and indirect acoustic paths in a playback environment. As with other figures provided herein, the types, numbers and arrangement of elements shown in Figure 2A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. In the example shown in Figure 2A, the audio travels from the loudspeaker 210B to the listener 205 via direct acoustic path 215i and indirect acoustic path 215ii.

[0082] Figure 2A includes the following elements:• 200: A room containing 4 wireless speakers in a Flexible Rendering system;• 205: A listener at a listening position;• 210A, 210B, 210C and 210D: 4 wireless speakers;• 215: the acoustic path between speaker 210B (arbitrary chosen) and the listener 205;• 215i: the direct component of 215; and• 215 ii: the indirect or reverberant component of 215.

[0083] In the acoustic space 200, a listener 205 at a listening position hears the speaker 210B by way of the direct acoustic path 215i and the reverberant path 215ii that compose the total acoustic response 215. The relative sensitivities of the individual audio devices (loudspeakers 210A through 210D) can be estimated by estimating the direct acoustic path of each and comparing them. The relative sensitivities of the individual audio devices may also be estimated by using a combination of direct acoustic path signals and indirect acoustic path signals for each audio device and comparing the resulting combinations.

[0084] The direct acoustic path is the shortest acoustic path. There are multiple acoustic paths which have the same length as any indirect path. The knowledge that the direct path isD25030W001 the shortest is exploited to separate the two in measurements made by the system described in this disclosure.

[0085] One may observe that the sound emitted by loudspeaker 210D is being directed towards a wall of the acoustic space 200, not towards the listener 205. If the sound emitted by loudspeaker 210D were directed towards the listener 205, the perceived loudness at the listening position would be relatively greater. Accordingly, the orientation of a loudspeaker is related to relative loudness at the listening position. The orientation of a loudspeaker is also related to what is referred to herein as a “rendering quality factor,” which is described below.Audio Calibration Signals and Audio Calibration Signal Parameters

[0086] Various types of audio calibration signals, also referred to herein as “calibration signals,” may be used in the methods disclosed in this document. Under sequential playback conditions, any calibration signal with a spectrally-rich autocorrelation in the bandwidth of concern could be used to measure a DW-like response. Examples include time-stretched-pulse (TSP) calibration signals, maximum length sequence (MLS) calibration signals, sine sweep calibration signals, etc. In some implementations, the calibration signals may be Direct Sequence Spread Spectrum (DSSS) signals. DSSS calibration signals are particularly beneficial for implementations involving the simultaneous playback of calibration signals in overlapping bandwidths from multiple loudspeakers of a playback environment.

[0087] DSSS signals have previously been deployed in the context of telecommunications. When DSSS signals are used in the context of telecommunications, DSSS signals are used to spread out the transmitted data over a wider frequency range before it is sent over a channel to a receiver. Most or all of the disclosed implementations, by contrast, do not involve using DSSS signals to modify or transmit data. Instead, such disclosed implementations involve sending DSSS signals between audio devices of an audio environment. What happens to the transmitted DSSS signals between transmission and reception is, in itself, the transmitted information. That is one significant difference between how DSSS signals are used in the context of telecommunications and how DSSS signals are used in the disclosed implementations.

[0088] Moreover, the disclosed implementations involve sending and receiving acoustic DSSS signals, not sending and receiving electromagnetic DSSS signals. In many disclosed implementations, the acoustic DSSS signals are inserted into a content stream that has been rendered for playback, such that the acoustic DSSS signals are included in played-back audio. According to some such implementations, the acoustic DSSS signals are notD25030W001 audible to humans, so that a person in the audio environment would not perceive the acoustic DSSS signals, but would only detect the played-back audio content.

[0089] Another difference between the use of acoustic DSSS signals as disclosed herein and how DSSS signals are used in the context of telecommunications involves what may be referred to herein as the “near / far problem.” In some instances, the acoustic DSSS signals disclosed herein may be transmitted by, and received by, many audio devices in an audio environment. The acoustic DSSS signals may potentially overlap in time and frequency. Some disclosed implementations rely on how the DSSS spreading codes are generated to separate the acoustic DSSS signals. In some instances, the audio devices may be so close to one another that the signal levels may encroach on the acoustic DSSS signal separation, so it may be difficult to separate the signals. That is one manifestation of the near / far problem, some solutions for which are disclosed herein.

[0090] A DSSS signal typically includes data, a carrier signal and a spreading code. To implement DSSS, in some examples binary phase shift keying (BPSK) modulation may be utilized. If the need to transmit data over the channel is omitted, then one can express the BPSK signal, s(t), as follows:s[n] = Ac[n] cos(ω₀n) (1)

[0091] In the foregoing equation, A represents the amplitude of the DSSS signal, c[n] represents the spreading code, and cos() represents a sinusoidal carrier with a digital frequency of ω₀ derived from a physical carrier frequency of f0. These terms of Equation 1 are examples of audio calibration signal parameters.

[0092] Figure 2B shows examples of an audio DSSS signal and an audio playback signal. The curve 230 in Figure 2B corresponds to an example of s[n] in the equation above. The DSSS signal 230 includes a main lobe 235, in which the highest-level portions of the DSSS signal 230 can be found. One of the potential advantages of some disclosed implementations involving acoustic DSSS signals is that by spreading the signal one can reduce the perceivability of the DSSS signal component of audio device playback sound, because the amplitude of the DSSS signal component is reduced for a given amount of energy in the acoustic DSSS signal.

[0093] This allows us to place the DSSS signal component of audio device playback sound (e.g., as represented by the curve 230 of Figure 2B) at a level sufficiently below the levels of the content stream component of the audio device playback sound (e.g., as represented by the curve 225 of Figure 2B) such that the DSSS signal component is not perceivable to a listener (is not audible to a human listener). Some disclosed implementations exploit theD25030W001 masking properties of the human auditory system to optimize the parameters of the DSSS signal in a way that maximizes the signal-to-noise ratio (SNR) of the derived DSSS signal observations and / or reduces the probability of perception of the DSSS signal component. Some disclosed examples involve applying a weight to the levels of the content stream component and / or applying a weight to the levels of the DSSS signal component. Some such examples apply noise compensation methods, wherein the acoustic DSSS signal component is treated as the signal and the content stream component is treated as noise. Some such examples involve applying one or more weights according to (e.g., proportionally to) a play / listen objective metric.DSSS Spreading Codes

[0094] As noted elsewhere herein, in some examples calibration signals may be, or may include, one or more DSSS signals based on DSSS spreading codes, which may be referred to herein simply as “spreading codes.” Parameters of these spreading codes are examples of audio calibration signal parameters. The spreading codes used to spread the carrier wave in order to create the DSSS signal(s) are extremely important. The set of DSSS spreading codes is preferably selected so that the corresponding DSSS signals have the following properties:1. A sharp main lobe in the autocorrelation waveform;2. Low sidelobes at non-zero delays in the autocorrelation waveform;3. Low cross-correlation between any two spreading codes within the set of spreading codes to be used if multiple devices are to access the medium simultaneously (e.g., to simultaneously play back modified audio playback signals that include a DSSS signal component). Some spreading codes may be orthogonal spreading codes, meaning that there is no correlation between any two different spreading codes; and4. The DSSS signals are unbiased, (have zero DC component).

[0095] The family of spreading codes (e.g., Gold codes, which are commonly used in the GPS context) typically characterizes the above four points. If multiple audio devices are all playing back modified audio playback signals that include a DSSS signal component simultaneously and each audio device uses a different spreading code (with good crosscorrelation properties, e.g., low cross-correlation), then a receiving audio device should be able to receive and process all of the acoustic DSSS signals simultaneously by using a code domain multiple access (CDMA) method. By using a CDMA method, multiple audio devices can send acoustic DSSS signals simultaneously, in some instances using a single frequency band. Spreading codes may be generated during run time and / or generated in advance and stored in a memory, e.g., in a data structure such as a lookup table.D25030W001

[0096] Choosing a particular set of codes, c(t), that have good auto- and crosscorrelation properties allows for employing a code division multiple access (CDMA) scheme. As noted above, one such set of codes that is widely used is the Gold Code family. CDMA schemes, including those utilizing Gold codes, are widely used for telecommunications, positioning, navigation and timing applications. This results in the simultaneous use of set of signals that can be expressed as follows:5f[n] = AiCi[n] cos(ωin) (2)In Equation 2, the i subscript indicates the ithsignal.

[0097] In the embodiments disclosed in this document, DSSS refers to a set of signals which are, in general, synthesized using Gold Codes as the spreading code c[n]. While only BPSK is described in the examples provided in this document, quadrature phase shift keying (QPSK) can also be used.

[0098] A Gold Code of order K, is a binary sequence of length L = 2K— 1 This is then sampled at a rate fc: the chipping rate of the sequence. This specifies how fast each chip of the binary sequence is transmitted. Note, the term chip, not bit, is used for the Gold Code as it must be known by both the transmitter and receiver ahead of time and no data is transmitted in the Gold Code itself. If the audio sampling rate is fsthen (pc= fc / fsis the digital frequency of the code. A typical DSSS calibration signal for the purposes of acoustic calibration may have fc= 6000, o = 12000, K = 13 which results in a code periodicity of approximately 1.365 seconds or M = 65528 samples.

[0099] Figure 2C is a graph that shows examples of the powers of two DSSS signals with different bandwidths but located at the same central frequency. In these examples, Figure 2C shows the spectra of two DSSS signals 235A and 235B that are both centered on the same center frequency 237. In some examples, the DSSS signal 235A may be produced by one audio device of an audio environment (e.g., by the audio device 100A) and the DSSS signal 235B may be produced by another audio device of the audio environment (e.g., by the audio device 100B).

[0100] According to this example, the DSSS signal 235B is chipped at a higher rate (in other words, a greater number of bits per second are used in the spreading signal) than the DSSS signal 235A, resulting in the bandwidth 240B of the DSSS signal 235B being larger than the bandwidth 240A of the DSSS signal 235A. For a given amount of energy for each DSSS signal, the larger bandwidth of the DSSS signal 235B results in the amplitude and perceivability of the DSSS signal 235B being relatively lower than those of the DSSS signal 235A. A higher-bandwidth DSSS signal also results in higher delay-resolution of the baseband data products,D25030W001 leading to higher-resolution estimates of acoustic scene metrics that are based on the DSSS signal (such as time of flight estimates, time of arrival (ToA) estimates, range estimates, direction of arrival (DoA) estimates, etc.). However, a higher-bandwidth DSSS signal also increases the noise-bandwidth of the receiver, thereby reducing the SNR of the extracted acoustic scene metrics. Moreover, if the bandwidth of a DSSS signal is too large, coherence and fading issues associated with the DSSS signal may become present.

[0101] The length of the spreading code used to generate a DSSS signal limits the amount of cross-correlation rejection. For example, a 10 bit Gold code has just -26dB rejection of an adjacent code. This may give rise to an instance of the above-described near / far problem, in which a relatively low-amplitude signal may be obscured by the cross correlation noise of another louder signal. Some of the novelty of the systems and methods described in this disclosure involves orchestration schemes that are designed to mitigate or avoid such problems.Receiving Calibration Signals

[0102] In the CDMA scheme, receiving individual signals can be done using a matched filter, which is optimal in the sense of maximizing the output signal-to-noise ratio (SNR) of the filter. During a calibration process, a microphone of a hand-held sensing device will produce a microphone signal d[n] containing a mixture of noise and a plurality of calibration signals. If multiple loudspeakers are simultaneously transmitting calibration signals, this microphone signal will contain the calibration signals it hears from multiple speakers after travelling through the playback environment to the microphone. After travelling through the playback environment, the calibration signals are altered (lower amplitude, delayed, etc.). The microphone signal also contains noise signals, etc. Moreover, in addition to noise, the calibration signals may be masked by audio content that is also being played back by the loudspeakers. This audio content is not used for the calibration measurement itself, but to improve the user experience.

[0103] According to some examples, as a modulated signal is received, a complex replica r;[n] of the calibration signal for reception such as the following may be used:[n] = AfC n] cos(o in) + jAici[n] sin(On) (3)In Equation 3, A represents the amplitude of the DSSS signal, c represents the spreading code, < >o represents the digital frequency of the carrier wave and i represents the index of the code and the index of a loudspeaker that is emitting calibration signals based on that code. A control system may use the complex replica rt[n] of the calibration signal to produce, the complex matched filter output for the ith calibration signalD25030W001 yi M = Sm=o n [T - ] d [tn] (4)

[0104] Accordingly, j / j represents the circular convolution of rtand d. In Equation 4, T represents the discrete time sample of the circular convolution signal, / W represents the length of the code in samples and m represents the convolution summation variable. In some implementations, the control system may be configured to implement fast versions of this algorithm that utilize the Fourier transform and exploit the convolution theorem.D25030W001 Delay waveforms

[0105] The matched filter output ytis a complex signal. For the purposes of peak picking, level data estimation, and range finding, the delay waveform (DW) may be computed by either taking the magnitude or power of the complex matched filter output, yf. The power delay waveform may be computed as follows:DWilr] = / ?eaZ(yf[T])2+ Imag(_yi[z]')2(5)

[0106] Figure 3A shows an example of a delay waveform (DW).

[0107] In the graph shown in Figure 3 A, the horizontal axis indicates samples in a time interval and the vertical axis indicates power. The direct response 305 of the DW 300 corresponds to the earliest peak that is above a noise threshold (e.g., above 1000 units, above 2000 units, etc.). The noise threshold may vary according to characteristics of the particular playback environment. Often, the direct response 305 peak is also the strongest (highest-power) peak. The direct response 305 corresponds to sound waves traversing a direct acoustic path from a loudspeaker to a listening position (or to a proxy for a listening position such as the microphone of the hand-held sensing device 100). Accordingly, the direct response 305 corresponds to the direct acoustic path 215i shown in Figure 2A.

[0108] The indirect response 310 is the remainder of the response. It is comprised of the reflections in the acoustic space. The indirect response 310 can, in some instances, carry on for a significant amount of time (such as hundreds of milliseconds).

[0109] The noise region 315 corresponds with samples obtained prior to time of the direct response 305. Any samples occurring before the direct response 305 do not correspond to a physically possible acoustic path. Accordingly, samples in the noise region 315 are suitable for noise level estimation.

[0110] Figure 3B shows an example of another DW. In Figure 3B, the peak that has been selected as the “direct response 305” corresponds with a power peak that is smaller than the peak 320 corresponding to a reflection within the indirect response 310. In this particular example, there is no peak that corresponds to a direct acoustic path. The weak “direct response 305” is actually the earliest peak of the indirect response 310, which the control system selected because the power level of the peak is greater than a noise threshold. This is an example of a measurement where there is no detectable peak in the DW corresponding to the direct acoustic path from the speaker to the microphone. For example, a back- or downfiring television (TV) could produce such a response.D25030W001

[0111] Unlike methods which utilize the Wiener filter to estimate an impulse response, some methods disclosed in this document use a matched filter, which produces an impulse response convolved with the autocorrelation of the calibration code, e.g., as follows:

[0112] yf[r] = h [T] ® / ?[T] (6)

[0113] In Equation 6, ® represents the circular convolution operator and R [T] represents the autocorrelation of the calibration signal. It may be assumed that there is insignificant carrier frequency mismatch and there is only a need to consider the delay component.

[0114] Figure 4 shows an illustrative autocorrelation of an order 11 Gold Code. In particular, Figure 4 illustrates that there is a significant amount of noise in the autocorrelation waveform at all lags (delays). Practically, this results in an upper bound on the SNR of the output of the matched filter process regardless of the input SNR. This upper bound, T, expressed in Decibels, is a function of the Gold Code order K:

[0115] T = 6 floor (7)

[0116] In Equation 7, the floor() function rounds down to the nearest integer.

[0117] Estimating Parameters from the Delay Waveform

[0118] The lag, also referred to as a delay, of the direct response corresponds to the Time of Arrival (ToA) of the calibration signal. Because the sensing device and the playback system are not synchronized, the delay of the direct response does not correspond to the Time of Flight (ToF) that is linearly related to the distance between the phone and speaker. The ToA of the ith calibration signal, Tj, in the DW may be estimated using a peak picking algorithm, the most naive of these being the max peak picker:

[0119] fj = argmax(J)W) (8)

[0120] More elaborate peak picking schemes can deal with strong reflections and obstructions in the playback environment.

[0121] The level of the direct response is a function of the relative loudness of the speaker, Gt, and the distance between the phone and the speaker, di. The relative loudness corresponds to how loud the speaker is at a fixed distance playing the DSSS signal. It is assumed that the speakers in the constellation have roughly the same target curve (the approximate frequency response of a loudspeaker).

[0122] The direct response power levelcan be modelled as follows:

[0123] Si =k-D25030W001

[0124] In Equation 9, k represents a scaling factor associated with the gain of the calibration device. Here, it may be assumed that the calibration device is itself not calibrated and that an unknown scaling factor k means that the absolute level of any signal in the microphone cannot be measured.

[0125] The direct response level can be estimated by taking the value of the earliest peak in the delay waveform. There are many ways of estimating what the earliest peak is. One such approach is to find the absolute peak and then, within a short window (e.g., in the range of 80 to 120 milliseconds) earlier than this, to look for all peaks which are either 1) significantly above the noise floor (e.g., more than twice the power of the noise floor), or 2) not too far down from the absolute peak level (e.g., above 15% of the argmax, above 20% of the argmax, above 25% of the argmax, etc.). Once the earliest peak is found, the maximum of the earliest peak can be taken as the level estimate Sj.

[0126] Alternatively, knowledge of the autocorrelation waveform (R [T], the autocorrelation of the calibration signal) may be exploited. The shape of the autocorrelation waveform can be used to interpolate the samples surrounding the peak to refine the estimate of the signal level. This interpolation addresses the fact that the peak value corresponding to the direct response is likely to be located at some delay between samples. In some embodiments, the noise in the autocorrelation signal is ignored. The peak response of the DW is then modelled as a triangle. Recalling that DW [f is the maximum value of the DW, then the estimate of the direct peak level, St, using the triangle signal model can be made as follows:

[0127] Si= (DlV[fi] + dL + MeZ) / 2 (10)

[0128] In Equation 10, Meirepresents the next largest sample corresponding to the peak and may be expressed as follows:

[0129] Mel= max (DW[ti - l], DW[ti + 1]). (11)

[0130] In Equation 10, dL represents the level change of the delay waveform between the samples and may be expressed as follows:

[0131] dL = - min (DW[ti - l], DW[ti + 1]) (12)

[0132] This process comprises the autocorrelation shape based interpolation.

[0133] The noise level, N, can be estimated by processing the samples in the noise region 315 of the DW. As yt [T] can be modelled as a complex Gaussian process, the noise samples in the power DW will follow a chi-squared distribution. If, instead of taking the power DW, the magnitude DW (MDW) is computed, the MDW may be used synonymously with DW (except where differentiation between them is required):D25030W001

[0134] MDWilr] = y / Real(yi[z]')2+ Imag(yi[T]')2(13)

[0135] Then the noise-region follows a Rician distribution, and a Gaussian approximation can be sufficient in typical operating conditions. One way to estimate the noise level, aN, is to take the standard deviation of the noise only region Tw

[0136] aN= std (MDI / ! / ;[r]) VT G TW(14)

[0137] Estimating Relative Device (Loudspeaker) Sensitivity Levels

[0138] Because the sensing device is not synchronized with the playback system, the absolute distances between the sensing device and the speakers cannot be measured directly. Consequentially, the absolute device sensitivity levels also cannot be measured directly. Even if synchronization were achieved, the sensing device itself is not assumed to be calibrated, meaning the absolute levels cannot be measured in the microphone feed of the sensing device. As a result, the relative level estimates of the loudspeakers is used to estimate the relative sensitivities of the loudspeakers. The Signal Minus Noise (SMN) metric can be used to estimate these relative sensitivities. The estimated SMN in dB can be expressed as follows:

[0139] S = 101og10(S - a ) (15)

[0140] The SMN metric is a function of both the device (loudspeaker) sensitivity and the direct acoustic path attenuation.

[0141] In the DW the signal level estimate, S, can be taken by estimating the level of the direct response 305 peak in the DW. The signal level estimate can also be taken by accumulating all of the power in the indirect response 310, or by accumulating all of the power in both the direct response 305 and the indirect response 310, known as the total response. The combination of direct and total response level estimates can be used to apply direct sound compensation, including by methods described in VIRTUAL SOUND SOURCES AND RENDERING TECHNIQUES, WIPO Patent Publication No. WO 2024 / 238368, contents of which are incorporated by reference therein. In any of these cases, the SMN estimates can be used as a proxy for the device sensitivity estimate that incorporates the acoustic attenuation component (—^ ). If some reference device, r, is chosen, then arelative sensitivity estimate Fj, in decibels, for the ithdevice can be produced as follows:

[0142] Ff= SMNt- SMNr(16)

[0143] In Equation 16, SMNtrepresents the SMN estimate of the ithloudspeaker and SMNrrepresents the SMN estimate of the reference loudspeaker r. Because the foregoing SMN estimates incorporate the acoustic attenuation component, these estimates are relativeD25030W001 sensitivity estimates. These SMN estimates are useful, however, as they are the actual direct level at the listening position.

[0144] Estimating Absolute Device Sensitivity Levels

[0145] Alternatively, sensitivity may be estimated in a manner that the sensitivity estimate does not incorporate the acoustic attenuation component. To estimate the relative loudness of the loudspeaker, G in this manner some reference distance drcan be assumed for the reference device. Then, from the location of the direct peak in each DW, the absolute distances between the user and speaker can be estimated, e.g., as follows:

[0146] dt= dr+ 8n(ji — fr) (17)

[0147] In Equation 17, 8nrepresents a conversion factor between samples and distance, which may be expressed as follows:

[0148] 8n= (18)Is

[0149] In Equation 18, c represents the speed of sound in meters / second. As noted above, SMN estimates can be used as a proxy for device sensitivity estimates. According to some alternative methods involving Equations 17 and 18, an alternative SMN’ may be determined as follows:

[0150] SMN' = 101og10(S - - 101og10(df) (19)

[0151] Estimating Absolute Scale

[0152] As noted elsewhere in this document, the calibration device will not normally be synchronized with the playback system. Thus, in previous sections, an assumed distance to one reference device can be used along with the differences in ToA from this device to all other devices in order to compute an estimated distance. In this section, methods for estimating the absolute scale of the playback environment are described, including using the set of asynchronous measurements described earlier.

[0153] Specifically, if Ndrepresents the number of loudspeakers, then there are NdToA measurements and Ndsignal level measurements. These are the asynchronous measurements obtained by the calibration device. The linear signal power level, Lj, may be represented as follows:

[0154] Li = - 8?N(20)

[0155] This power level estimate is inversely proportional to the distance between the calibration device and the speaker squared. That is,

[0156] oc k-^ (21)D25030W001

[0157] Thus, the Ndlevel measurements contain information about the absolute distances from the calibration device position to the loudspeakers, which may be used as a proxy for the absolute distances from the user position to the loudspeakers. When the playback speakers are all synchronized amongst themselves, there is only one unknown to solve for in order to obtain the absolute scale of the solution, which is the clock offset. With the 2Ndmeasurements, there is a sufficient number of measurements in order to solve this problem.

[0158] An example embodiment of using the estimated signal levels to resolve the absolute scale of the room follows. Based on Equation 21, the ratio of the ith and jth device (loudspeaker) signal level estimates Ltand Lj may be expressed as follows:Li Gi dl[(H59] (22)

[0160] In a typical listener-centric acoustic mapping (LCAM) system, there may be some speakers which are not firing directly at the listener position. If possible, the calibration data obtained from loudspeakers that are not firing directly at the listener position (e.g., obtained from loudspeakers that are emitting sound in a direction that is plus or minus 90 degrees of the direction directly away from a listening location, plus or minus 100 degrees, plus or minus 110 degrees, plus or minus 120 degrees, etc.) are omitted when implementing the methods of this section. If the ith and jth device descriptor gains Gtand Gy, respectively are known, then the absolute scale of the playback environment can be estimated. The distance between loudspeakers i and j, ij, may be defined as follows:

[0161] = 8n(Ti - tj) (23)

[0162] Using Equation (17), Equation (22) may be rewritten as follows:

[0163] | (24)

[0164] Using the quadratic formula, this allows us to solve for d;-:\1:hKGi /

[0165] - L (25)

[0166] Equation 25 may be simplified as follows:

[0167] di = (26)LjG -w1

[0168] After di, the absolute distance to loudspeaker z, is determined, then Equation (26) can be used to solve for all the other absolute loudspeaker distances, e.g., as follows:D25030W001

[0169] dj = dL+ btj Vj E Na (27)

[0170] Frequency Hopping

[0171] The spectrum of the DSSS signal can be changed dynamically over time to obtain a multi-band set of calibration data. The DSSS signal spectrum can be approximated by just the main lobe of its representation in the frequency domain (e.g., the DSSS main lobe 235 in Figure 2B). It is this main lobe that is moved by both altering the center frequency and the width (by changing the chipping frequency). These changes may be done in discrete steps such that the DSSS signal will stay at a particular center frequency bandwidth for at least one sequence period.

[0172] In some embodiments, the range within which the center frequency and bandwidth changes can be chosen based upon the device descriptor (the parameters of a particular loudspeaker) whereby the device descriptor specifies a valid set of frequencies that the speaker can reproduce. The above methods for calculating sensitivity levels can then result in a frequency-dependent set of sensitivity levels. These sensitivity levels can then be used for performing an EQ on each speaker feed to correct for playback environment effects, such as the acoustic reflectivities of walls and ceilings, furniture effects and speaker manufacturing variations.

[0173] Combining Online and Offline Loudness Measurements to Render Audio

[0174] After an “online” estimate of the relative loudness or sensitivity of the loudspeakers has been made according to one or more of the above-described methods, this estimate can be used to configure the Flexible Renderer. This online estimate of the relative loudness can either be used in conjunction with “offline” loudness measurements, such as those indicated by device descriptor sensitivity information, or alone to calculate a loudness correction gain for each speaker feed signal.

[0175] Figure 5 shows example components of system that is configured to generate a full set of coefficients that may be used by a flexible renderer during a rendering process. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more elements, fewer elements and / or one or more different types of elements. For example, in some alternative implementations rendering coefficients may be generated based on the “online” data provided by block 502, without reference to the “offline” data provided by block 503 or 504.

[0176] According to this example, the elements of Figure 5 include the following:

[0177] 500: A system for the generation of coefficients to use in rendering;D25030W001

[0178] 501: The relative map data derived (for example, by the control system 150 of Figure 1B) from azimuth data obtained from a sensing device during a calibration process and the analysis of delay waveforms corresponding to audio calibration signals emitted by each loudspeaker of a plurality of loudspeakers and detected by a microphone system of the sensing device during the calibration process;

[0179] 502: Relative sensitivity data indicating the relative sensitivities of each loudspeaker derived “online,” in a playback environment (for example, by the control system 150 of Figure 1B), from the delay waveforms, e.g., according to one of the methods disclosed in this document;

[0180] 503: The loudspeaker device descriptor data that configures the other parts of the device processing contained within the offline created device descriptor that is stored on (or for) each loudspeaker at tuning time. The device descriptor data 503 may be received at mapping time from storage on the speaker or from an external database. The device descriptor data 503 may contain a measured response of a speaker with the tuning applied. The device descriptor data 503 may include the equalization (EQ) that should be applied for the speaker to meet a particular target curve, multiband limiter limits, filter coefficients that describe how to create crossovers, etc. The device descriptor data 503 may include loudspeaker sensitivity data corresponding to the sensitivities indicated by the device descriptors created “offline” at tuning time. In other words, the loudspeaker sensitivity data may be stored as part of each of the device descriptors;

[0181] 505: The rendering coefficients produced by the rendering coefficient generator 506;

[0182] 506: The rendering coefficient generator, which may also be implemented by the control system 150 of Figure 1B. In some relatively simpler examples, the rendering coefficient generator 506 may calculate the loudness correction gains based upon the map data 501 and the provided relative sensitivities (502).

[0183] The rendering coefficient generator 506 may, in some implementations, calculate the loudness correction gains based upon the map data 501 and both the offline and online measurements of the relative sensitivities (502 & 503). The map data 501 is optional for the loudness correction gains, but is not optional if used for creating the rendering coefficients (which include the loudness correction gains). In some examples, the rendering coefficient generator 506 may be configured to use the offline sensitivities indicated by the device descriptor data 503 to provide a gain that should be applied to each loudspeaker such that the loudness at the listener’s location (or assumed location) is the same from eachD25030W001 loudspeaker. In some such examples, the rendering coefficient generator 506 may be configured to augment these offline-determined gains with the “online” relative sensitivity data 502, e.g., such that the online gains are used instead of, or in conjunction with, the offline gains. The offline-determined gains may, for example, be used to by the rendering coefficient generator 506, e.g., to constrain the amount of adjustment that can occur by implementing the “online” relative sensitivity data 502. Such constraints may be desirable because one cannot trust that the online measurements have all been made accurately.Implementing the “online” relative sensitivity data 502 without such constraints could, for example, cause a situation in which one loudspeaker is adjusted to be very loud compared to the rest. Furthermore, if the rendering quality factor were derived, then those loudspeakers may be penalized if their corresponding rendering quality factor is low.

[0184] However, the offline-determined gains can also occasionally be erroneous. The reasons for such possible errors include the following:1. The vendor or manufacturer indicated inaccurate and / or misleading device descriptor data, e.g., to make a loudspeaker appear to have superior characteristics. (This can be prevented by certification testing.)2. The offline measurement included too much of the room response of the room in which the loudspeaker was measured (in other words, the loudspeaker was not measured in an anechoic room).

[0185] The combination of the above two problems means that in some circumstances the online measurement is actually more trustworthy. In some examples, the offline measurement-based constraints applied by the rendering coefficient generator 506 may be looser or ignored entirely if the values from the offline measurement are improbable (e.g., if one or more values are not in an expected range).

[0186] The rendering coefficients 505 produced by the rendering coefficient generator 506 may, in some examples, include coefficients of a cost function. According to some such examples, the cost function may be expressed as follows:C(g) ^spatial (g< {■$ / <:}) + Cproximity (g< +Sz Cl (& {{»}, (28)

[0187] In Equation 28, C(g) represents the cost C as a function of g, which represents an -dimensional vector of speaker activations. In Equation 28, the set {sfc} represents the positions of a set of Ns loudspeakers and 6 denotes the desired perceived spatial position of the audio signal. In this example, the first term of Equation 28, Cspatiai, models how closelyD25030W001 a desired spatial impression is achieved as a function of mapping an audio object into loudspeaker signals, and the second term Cproximity assigns a cost to activating each of the loudspeakers. One purpose of the term Cproximity is creating a sparse solution in which only loudspeakers in close proximity to the intended spatial position of the object signal are activated. According to this example, the cost function includes one or more additional dynamically configurable terms to the activation penalty, allowing the spatial rendering to be modified in response to numerous other controls. In Equation 28, the termsg, {{o}, {s / c}< {e]}zrepresent these additional cost terms, with {0} representing a set of one or more properties of the audio signals (e.g., of an object-based audio program) being rendered, {sfc} representing a set of one or more properties of the speakers over which the audio is being rendered, and {e} representing one or more additional external inputs. Each term Ctg, {{0}, {s, {e}] returns a cost as a function of activations g in relation to a combination of one or more properties of the audio signals, speakers, and / or external inputs, represented generically by the set {{0}, {sk},. In some examples, the set {{0}, {sk}, {e]jjcontains at a minimum only one element from any of {0}, {sfc}, or {e}.

[0188] Examples of {0} include but are not limited to:• Desired perceived spatial position of the audio signal;• Level (possible time-varying) of the audio signal; and / or• Spectrum (possibly time-varying) of the audio signal.Examples of {sfc} include but are not limited to:• Locations of the loudspeakers in the listening space;• Frequency response of the loudspeakers;• Playback level limits of the loudspeakers;• Parameters of dynamics processing algorithms within the speakers, such as limiter gains;• A measurement or estimate of acoustic transmission from each speaker to the others;• A measure of echo canceller performance on the speakers; and / or • Relative synchronization of the speakers with respect to each other. Examples of {e} include but are not limited to:• Locations of one or more listeners or talkers in the playback space;• A measurement or estimate of acoustic transmission from each loudspeaker to the listening location;D25030W001 • A measurement or estimate of the acoustic transmission from a talker to the set of loudspeakers;• Location of some other landmark in the playback space; and / or• A measurement or estimate of acoustic transmission from each speaker to some other landmark in the playback space.

[0189] Figure 6 illustrates example components of a system that includes a Tenderer for rendering loudness-corrected speaker feeds using the rendering coefficients generated by the rendering coefficient generator of Figure 5. The loudness-corrected speaker feeds 604 produced by the Tenderer 603 can be sent to the individual loudspeakers of a playback environment. As with other figures provided herein, the types and numbers of elements shown in Figure 6 are merely provided by way of example. Other implementations may include more elements, fewer elements and / or one or more different types of elements.

[0190] According to this example, the elements of Figure 6 include the following:

[0191] 600: A flexible rendering system;

[0192] 601: The audio data that is input to the flexible Tenderer 603;

[0193] 505: Rendering coefficients that are generated by the rendering coefficient generator 506 of Figure 5. The rendering coefficients 505 contain the loudness correction gains;

[0194] 603: The flexible Tenderer. In some examples, the flexible Tenderer 603 may be configured to implement center of mass amplitude panning (CMAP), flexible virtualization (FV), vector-based amplitude panning (VBAP), another flexible rendering method, or combinations thereof;

[0195] 604: Loudness-corrected speaker feeds generated by the flexible Tenderer 603 from the audio data 601, based on the rendering coefficients 505.

[0196] In some implementations, after the relative sensitivities 502 are derived using the delay waveforms, the relative sensitivities 502 can be combined with the relative map data 501 to generate rendering coefficients that can be used with a flexible Tenderer to render audio, without the need to include loudspeaker device descriptor data obtained “offline.” This may be done as described in the “Estimating Parameters from the Delay Waveform” section. Such implementations negate the need for careful measurement of the relative loudness of different speakers at tuning time and removes the need for careful co-ordination between the manufacturers of different speakers. In some examples, some but not all of the loudspeaker device descriptor data may be used to generate rendering coefficients.D25030W001

[0197] In some cases, the process of generating rendering coefficients with relative sensitivities 502 but without “offline” data (such as that provided by block 503 or block 504 of Figure 5) can be improved using a simple speaker model based upon the approximate dimensions of the speaker. In some such examples, a magnitude response and frequencydependent radiation pattern may be estimated based upon loudspeaker size, based on the principles of wave diffraction through an opening. The "opening" in this case is the speaker driver itself. The radiation pattern becomes more omnidirectional for wavelengths of sound that are relatively longer as compared to the opening size.

[0198] Furthermore, this simple speaker model may be used for adjusting the frequency range of the DSSS signals to ensure the signals occur in a frequency bands in which the speaker is most likely to have a response.

[0199] In some implementations, when using the online relative sensitivities, distance models may be disabled when compared to a case where the relative sensitivity is not provided. Disabling the distance models may ensure that the loudness that is perceived at the listener is accurate, and that the panning works as expected. If the position of one of the loudspeakers (a reference loudspeaker) is assumed, the relative map of block 501 can be converted into an absolute map.

[0200] In some implementations a different distance model can be derived to decide the level of correction that is permissible. For example, in a case where the listener is close to the loudspeaker, a large amount of correction may want to be avoided because the level of error of the listener position may have a larger impact than instances in which a loudspeaker is relatively farther from the listener.

[0201] In some implementations the relative map and relative sensitivities derived in the playback environment may be combined with manufacturer-provided tuning coefficients and sensitivity measurements. In some such examples, the tuning time sensitivities (e.g., from block 504) may be used to loudness match the different loudspeakers. This tuning-based loudness matching can be used as a ground truth to constrain the amount of loudness correction that occurs based on the relative sensitivities 502. For example, the loudness correction based on user measurements and the resulting relative sensitivities 502 may be clipped to not deviate by more than 3dB from the tuning based implementation. In other implementations the amount of correction may be constrained to a particular threshold, such as 4dB, 5dB, 6dB, 7dB, 8dB, etc., irrespective of the tuning-based correction and may in some instances wholly override the tuning based correction. In some examples, the amountD25030W001 of correction may be constrained in a smoothed manner, e.g., by performing soft clipping (e.g., by applying a sigmoid function) instead of hard clipping.

[0202] In some implementations the tuning time sensitivity measurement may be a calibrated measurement, for example measured in dBSPL. This tuning time absolute sensitivity measurement can be used in conjunction with the user relative measurements (relative delay and relative sensitivity) to derive an absolute map. This derivation can be used for solving the near and far problem in DSSS mapping, changing the rendering (for example steering a direct sound compensation system as described in VIRTUAL SOUND SOURCES AND RENDERING TECHNIQUES, WIPO Patent Publication No. WO 2024 / 238368, the contents of which are hereby incorporated by reference). Some such methods involve penalizing close by speakers to improve the performance of the Tenderer. In some implementations the user may be prompted to move the speakers into a different location based upon these measurements.

[0203] Based on loudspeaker frequency response data such as that included in the device descriptor, in some implementations the center frequency of the DSSS signal may be adjusted in order to ensure that the DSSS signal — or at least the main lobe of the DSSS signal — lies within a frequency range for which the loudspeaker has an adequate response. This information may be stored within the loudspeaker device descriptor that is created “offline” at tuning time.D25030W001

[0204] Speakers Not Pointing Directly at the Listener

[0205] There is a class of speakers for which, during normal operation, the speaker is not pointed directly at the user. Some such loudspeakers are, or include, one or more “upfiring” speakers that are designed to cause sound produced by the speaker(s) to reflect from a ceiling, e.g., in order to simulate “height” effects that may be produced by a speaker that is actually mounted on the ceiling, on the upper part of a wall, etc. Alternatively, or additionally, some loudspeakers may be, or may include, downfiring speakers.

[0206] Figure 7 illustrates an example of a system that is configured to compensate for the orientation of individual drivers within a loudspeaker enclosure based upon data in the loudspeaker device descriptor. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other implementations may include more elements, fewer elements and / or one or more different types of elements.

[0207] According to this example, the elements of Figure 7 include the following:

[0208] 700: A system in which device descriptors may be used in conjunction with delay waveforms to derive a relative delay and relative loudness of an individual driver;

[0209] 701: Delay waveform data from one or more captured audio calibration signals;

[0210] 702: The device descriptor data that contains the individual driver orientations;

[0211] 703: A driver orientation compensation module configured to convert a delay waveform into a set of relative delays and relative loudnesses based on the delay waveform data 701 — or information derived from delay waveform data 701 — and the device descriptor data 702. The driver orientation compensation module 703 may, for example, be implemented by the control system 150 of Figure 1B. In some implementations, the driver orientation compensation module 703 may be configured to change the selection of the peak for individual delay waveforms;

[0212] 704: The relative delays derived for each speaker by the driver orientation compensation module 703;

[0213] 705: The relative loudnesses for each speaker derived by the driver orientation compensation module 703;

[0214] The first class of speakers that can be corrected via the system 700 is downfiring drivers. In the downfiring case, the reflecting surface is normally either quite close to the speaker (in the case where the speaker is on a table or another piece of furniture) or quite far away, e.g., in the case of a TV mounted on a wall.D25030W001

[0215] The second class of speakers that can be corrected via the system 700 is upfiring speakers. Upfiring speakers generally assume a particular reflectivity and ceiling height to give a perceived height effect. If these assumptions are not true, the height effect can be sub-optimal.

[0216] In some implementations the user may be asked to program the expected reflection surface for sound emitted from an upfiring speaker or a downfiring speaker, for example the distance from a downfiring speaker of a TV that is mounted on the wall or the distance from an upfiring speaker to the height of a ceiling.

[0217] In some implementations the device descriptor may be used for changing the calculation of the relative delay and sensitivity. The device descriptor indicates the orientation of the drivers in the enclosure.

[0218] In the case of upfiring speaker(s), the direct path is not the path that should be used for calculating the level adjustment. The relative delays and sensitivity adjustments of the upfiring drivers should be used independently of the other drivers within the enclosure. In other words, if a loudspeaker enclosure includes one speaker that is directed towards a listener position and an upfiring speaker, different relative delays and sensitivity adjustments should be determined for the direct speaker and the upfiring speaker.

[0219] In the case of the downfiring speaker the relative loudness can be used to further change the rendering. The difference in time for the peaks of the delay wave form will be different for a device that is sitting on a table vs one that is reflecting off the floor. For loudspeakers that emit sound which is reflected off the floor, a different equalization (EQ) may be chosen to compensate for the sound absorption of the flooring material. The flooring material may result in a few different EQ types. The flooring material may be derived by looking at the relative level differences between what is predicted by what has been previously measured and stored in the device descriptor and what is derived from the user measurement.

[0220] Suboptimal Speaker Placement and Other Room Effects - Derivation of a Rendering Quality Factor

[0221] The discussion so far has assumed that a user has placed the speakers pointing at the listener — or at least towards a zone in which the listener expects to be during playback — and relatively far away from any walls or other sound-reflecting surfaces.However, many users will not be able to — or may not think to — control loudspeaker locations and orientations of the playback environment in order to ensure that these assumptions are in accordance with reality.D25030W001

[0222] Therefore, some implementations may implement a rendering quality factor derivation from the delay waveforms. Setups with a large number of speakers with a low rendering quality factor will have worse virtualization than those with speakers with a high rendering quality factor.

[0223] The rendering quality factor can be used to detect suboptimal rendering aspects such as the speakers being in doorways, pointing in the wrong direction, having furniture or walls too close to the speaker enclosure, the listener being too close or too far away to the speakers.

[0224] This rendering quality factor may, in some implementations, be used to instruct the user to change aspects of their loudspeaker setup in order to improve their loudspeaker setup. In other cases, the user may be presented with a score indicating that their loudspeaker setup is likely to provide a good listening experience. A control system may provide such feedback via one or more loudspeakers or displays of a user interface system, such as by the display of such instructions on a mobile phone screen or a television display, via audio prompts from one or more loudspeakers, etc.

[0225] In other implementations, the rendering quality factor may be used for changing the render itself, e.g., by adjusting the rendering coefficients. For example, the rendering quality factor may be used by implementing a penalty factor that penalizes (reduces) the amount of energy that comes out of a particular speaker when its rendering quality factor is low and instead prioritizes speakers that have a good rendering quality factor. In some examples, the rendering quality factor may be used to modify an audio object render, e.g., by rendering the audio object in a different location so that it will be reproduced by one or more loudspeakers having a higher rendering quality factor. In some instances, the rendering quality factor may be used for selecting a different configuration within a codec bitstream, e.g., for selecting a dialog enhancement presentation or selecting a channel-based render (e.g., a Dolby 7.1 render, a Dolby 5.1 render or a stereo-based render) instead of an audio object-based render.

[0226] Examples of Estimating the Rendering Quality Factor

[0227] According to some examples, the rendering quality factor may be estimated based, at least in part, upon the device descriptor’s assumed path to the listener. However, for speakers that are intended to be pointed at the listener in ideal circumstances, some implementations may implement the method of Figure 8 A to estimate the rendering quality factor.D25030W001

[0228] Figure 8A is a flow diagram that outlines blocks of a disclosed method of determining a rendering quality factor according to some examples. The blocks of method 800, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and / or described. The method 800 may be performed by an apparatus or system, such as by the apparatus 100 that is shown in Figure 1B and described above. More specifically, method 800 may be performed by the control system 150 of Figure 1B.

[0229] Some aspects of method 800 may involve calculating the ratio of magnitude, energy or power information between different peaks or weighted combinations of the magnitudes, energies or powers of the peaks in the delay waveforms for determining a rendering quality factor. The numerator and denominator of the ratio may each be either: (1) the magnitude, energy or power information of a singular peak; or (2) the weighted combination of magnitudes, energies or powers from 2 or more peaks.

[0230] In this example, block 801 involves a peak picking process. A “peak” may be defined as an instance of the largest amplitude in a series of time samples, e.g., a value that is larger than the last M samples and larger than the next M samples, where M may be 1, 2, 3, 4, 5, etc. Here, block 801 involves selecting significant peaks in the delay waveform. A “significant” peak may, for example, be a peak that is 1) significantly above the noise floor (e.g., more than twice the power of the noise floor), 2) not too far below the absolute peak level (e.g., above 15% of the “argmax,” above 20% of the argmax, above 25% of the argmax, etc.), or 3) the argmax peak having the highest peak level.

[0231] According to this example, block 802 involves a peak attribution process. Here, block 802 involves categorizing the significant peaks found in block 801 according to a component of a room / playback environment response and / or a component of the delay waveform. Alternatively, or additionally, block 802 may involve categorizing the peaks based on acoustic or psychoacoustic phenomena. In some examples, block 802 may involve categorizing peaks into the “signal” and “noise” portions of the delay waveform. According to some examples, block 802 may involve categorizing peaks based on a corresponding portion of the room acoustic response, such as direct response, specular reflections, and late responses. The peak attribution process of block 802 may be used to indicate what component goes into what part of the rendering quality factor equations. This categorization may be based upon the order of the peaks in the delay waveform. For example, the first significant peak may be assumed to correspond with the direct response and the subsequent peaks are assumed to be the reflections within the indirect response.D25030W001

[0232] In this example, block 803 involves using peak values and attributions to compute a rendering quality factor. According to some examples, block 803 may involve calculating the ratio of the maximum magnitude, energy or power value of the highest-energy peak and the maximum magnitude, energy or power values of one or more other peaks. In some examples, block 803 may involve calculating the ratio of the magnitude, energy or power value of the first significant peak and the maximum magnitude, energy or power values of one or more subsequent peaks. The one or more subsequent peaks may, for example, include the subsequent peak having the 2nd-highest-energy and / or the subsequent peak having the 3rd-highest-energy.

[0233] The ratio between the magnitude, energy or power of the first significant peak and subsequent peaks can be used for estimating the rendering quality factor(s) 804, which is the output of block 803. The higher the ratio, the higher the quality factor.

[0234] In some alternative examples, estimating the rendering quality factor(s) 804 may involve calculating the ratio of the energy of the direct response to the sum of the energy in the entire indirect response. In other examples, estimating the rendering quality factor(s) 804 may involve calculating the ratio of the energy of the direct response to the sum of the energy in the entire response, including the direct and indirect portions of the entire response.

[0235] An alternative implementation is to use the direct & reflection selection algorithm described earlier in the document to determine the individual peaks and selecting the direct value and the first two reflections. The ratio of energy factors can still be used in this case to determine the rendering quality factor.

[0236] Another alternative implementation involves forming a direct-to-noise energy ratio. The noise energy can be found by choosing the maximum peak attributed to the noise portion of the delay waveform. Calculating the direct (first significant peak) to noise energy ratio helps determine if the direct path is present, and if so how audible the corresponding audio signal is. This ratio decreases if a loudspeaker is facing away from the listening position, is occluded, or if the direct response is otherwise obstructed relative to the listening position.

[0237] Another alternative implementation involves selecting the argmax to the left of (the closest peak, in time, prior to) the highest energy peak and the argmax to the right of (the closest peak, in time, after) the highest energy peak. The ratios of those three peaks may be used to calculate the quality factor.

[0238] The rendering quality factor can be further augmented by combining the relative loudspeaker position information with the loudspeaker rendering quality factors. ForD25030W001 example, suppose that speaker A and speaker B are both in a similar zone, such that speaker A and speaker B might both be used to render spatial audio coming from that zone or direction. A zone might be, for example, some angular section relative to the listening position, such as 45° + / -15° from center front. However, suppose that speaker A has a higher rendering quality factor than speaker B. In such a situation, the speaker feed signals for speaker B may be penalized (reduced in amplitude or power) accordingly. Alternatively, the speaker feed signals for speaker B may be mixed into the speaker feed signals for one or more other speakers, e.g., mixed into the speaker feed signals for the closest speaker or the two closest speakers.

[0239] Figure 8B shows details of block 803 of Figure 8A according to some examples. In this example, block 803 includes blocks 807, 808, 809, 812, 813, 815 and 816. According to this example, blocks 807, 808 and 809 are grouped as process 804, blocks 812 and 813 are grouped as process 805 and blocks 815 and 816 are grouped as process 806. In this example, block 807 involves calculating peak magnitudes and, optionally, noise magnitude, e.g., as described above. In this example, block 808 involves calculating ratios between the magnitudes calculated in block 807, or between weighted combinations of the magnitudes calculated in block 807.

[0240] Since distance plays a direct role in the meaning of the rendering quality factor, in some examples a normalization factor may be applied to make the rendering quality factors of different speakers comparable. The normalization factor can be derived using assumed distances for certain speakers within a group of speakers (e.g., a television may be assumed to be about 2 meters away from the listener). For each speaker distance a lookup table based upon an assumed room reflectivity may then be used by adding the distance normalization factor to the quality factor. Accordingly, in this example, block 809 involves normalizing for noise floor and / or relative distance, based in part on the loudspeaker position data 810. In this example, block 809 involves generating one or more single-speaker rendering quality factors 811. In some implementations, if a speaker is relatively close to an actual or assumed listening position and that speaker has a low rendering quality factor, that speaker may be penalized more than a speaker that is further away and that has the same rendering quality factor, or a similar rendering quality factor.

[0241] After one or more single-speaker rendering quality factors 811 are computed, they may be augmented by computing the same rendering quality factors for multiple different code lengths. Accordingly, in this example, the one or more single-speaker rendering quality factors 811 generated in process 804 are input to block 812 of process 805. In this example,D25030W001 block 812 involves rendering the one or more single-speaker rendering quality factors 811 across different DSSS code lengths. Quality factors computed from the delay waveforms resulting from different DSSS code orders will have a dependence on the code’s theoretical SNR. Therefore, accounting for code order when computing the rendering quality factors (see block 809), especially those involving noise energy estimates, may be beneficial. One or more revised, single-speaker rendering quality factor 814 may be formed in block 813 by taking some weighted combination or other relative comparison of the rendering quality factors for each code order. In some examples, block 813 may involve calculating a mean or median rendering quality factor. Additionally, outliers may optionally be removed from consideration in block 813.

[0242] After single speaker rendering quality factors (811) and / or revised single speaker rendering quality factors (814) are calculated, they may then be compiled across speakers for comparison. Accordingly, in this example the one or more single-speaker rendering quality factors 811 generated in process 804 or the one or more revised, single-speaker rendering quality factor 814 of process 805 are input to block 815 of process 806. In this example, block 815 involves aggregating single-speaker rendering quality factors across the loudspeakers of a playback environment. Optionally, the device descriptors 819 may be used in block 815 to compare rendering quality factors only among speakers of similar rendering capabilities. A weighted combination or otherwise relative comparison of rendering quality factors can then be assessed in block 816. For example, if one speaker has relatively poor rendering quality factors compared to the group, (e.g. significantly below the median), then the speaker with poor rendering quality factors may be penalized further or used to inform a rendering quality factor threshold for the setup. In this example, rendering quality factors for all loudspeakers 817 are output by process 806.

[0243] For speakers that are intended to be reflection based speakers (e.g., as indicated by the device descriptor data), the ordering of the selection of the peaks should be changed slightly such that the first or earliest peak is not assumed to correspond with a direct arrival. For example, in a delay waveform for an “up firing” speaker, the second peak may be the desirable peak and the other peaks may be used for calculating the quality factor, according to the ratio between the energy of the second peak and the energy of one or more other peaks. The one or more other peaks may be one or more subsequent peaks. In some examples, the one or more other peaks may include the first peak. Some or all other aspects of the above-described rendering quality determination process may remain the same.D25030W001

[0244] Some examples may involve computing a rendering quality factor across all of the loudspeakers in a group of loudspeakers of a playback environment, e.g., as described in the "Speakers in different rooms" section below. This may be desirable because it may not be sufficient to only rely on the “noise floor” (the average or peak) energy in some instances. In one such example, the first significant peak of one loudspeaker may be compared to the first significant peak of another loudspeaker to derive a rendering quality factor. This method may reveal instances of, e.g., a loudspeaker in a doorway pointing away from the listener, which could otherwise be assumed to have a high quality factor. The magnitude of the loudspeaker in the doorway will generally be much lower than the loudspeaker in the room.

[0245] In the next section a series of normalized delay waveforms shall be shown. The vertical axis is the normalized level, and the horizontal axis represent sample numbers.

[0246] Speaker in Ideal Conditions

[0247] Figure 9 shows an example of a normalized delay waveform corresponding to the measurement of a speaker in ideal conditions. It can be seen that the direct path dominates the signal significantly and there are minimal reflections present in the DW. This results in a high rendering quality factor.

[0248] Figure 9:High Rendering Quality Factor

[0249] The salient features of Figure 9 are as follows:

[0250] 900: A DW graph indicating a high rendering quality factor

[0251] 901: The y axis normalized to the largest value;

[0252] 902: The x axis showing samples;

[0253] 903: The direct response peak;

[0254] 904: Subsequent reflections are very small compared to the direct peak.

[0255] Speakers Pointing Away

[0256] Figure 10 shows an example of a normalized delay waveform corresponding to the measurement of a speaker that is pointing away from a listening position.

[0257] The salient features of Figure 10 are as follows:

[0258] 1000: A DW graph indicating an example of a speaker pointing away from a listening position;

[0259] 1001: The y axis normalized to the largest value;

[0260] 1002: The x axis in samples

[0261] 1003: The peak corresponding to the direct response;

[0262] 1004: Subsequent reflections that are larger than the direct response.D25030W001

[0263] Figure 10 illustrates that when the speakers point away from a listening position, the dominant peak is typically the second reflection or in some cases a later reflection. This results in a reduced rendering quality factor when compared to a speaker that is pointing directly at the user. It is important to note that the second peak does not necessarily become fully dominant and is directly related to the geometry of the playback environment, the reflectivity of surfaces within the playback environment and the degree to which the speaker is pointing away from the microphone (the position of which is a proxy for a listening position).

[0264] Furniture

[0265] Placing a speaker in furniture, e.g., in a bookshelf, results in larger subsequent peaks in the reflection peaks when compared to speakers placed outside of the furniture. This results in a reduced rendering quality factor when compared to speaker that are pointing directly at the user.

[0266] Figure 11 illustrates an example of an effect of furniture on delay waveforms. The salient features of Figure 11 are as follows:

[0267] 1100: A graph showing the effect of furniture on delay waveforms;

[0268] 1101: The y axis normalized to the largest value;

[0269] 1102: The x axis in samples

[0270] 1103: The peak corresponding to the direct response

[0271] 1104: Subsequent reflections that are larger than those in a DW for a speaker in ideal conditions.

[0272] Occluded Speakers

[0273] Figure 12 illustrates an example of an effect of loudspeaker occlusion on delay waveforms. The salient features of Figure 12 are as follows:

[0274] 1200: A graph showing the effect of occlusion on the delay waveform;

[0275] 1201: The y axis normalized to the largest value;

[0276] 1202: The x axis in samples;

[0277] 1203: The peak corresponding to the direct response; and

[0278] 1204: The subsequent reflections;

[0279] Speakers that do not have a direct acoustic path between the speaker and the listener are another common case that can be detected with the rendering quality factor. In this case the subsequent reflections 1204 (the 2nd and 3rd peaks) are relatively louder than the peak 1203 corresponding to the direct path (in the clean case). In some implementations, if occlusion is detected another measurement at a different frequency may be performed toD25030W001 allow the waves to diffract around the object. The direct path may be present in the plot due to diffraction.

[0280] Speakers in different rooms

[0281] Figure 13 shows an example of the effects on delay waveforms when a speaker is placed in a room that is different from the room in which the microphone (or listener) is.

[0282] The salient features of Figure 13 are as follows:

[0283] 1300: A graph showing the effect on a delay waveform when the microphone is in one room and a speaker is in another room;

[0284] 1301: The y axis normalized to the largest value;

[0285] 1302: The x axis in samples;

[0286] 1303: The peak corresponding to the direct response; and

[0287] 1304: Subsequent peaks; and

[0288] 1305: The noise floor, which is very high because the signals are very small (because the signals are normalized in this example).

[0289] One of the primary issues caused by having speakers in different rooms is that comparing the first peak of the DW to subsequent peaks can be insufficient for evaluating the rendering quality factor because the peak that corresponds to the reflected peak is not large enough. An example of where this occurs is when a speaker is placed in or near a doorway and pointed in such a way that the acoustic path corresponding to the highest-energy peak is due to a reflection. In this case the indirect path is the main path to the capturing microphone and the direct path corresponds to a very small peak.

[0290] Figure 13 shows an example where the speaker is placed in another room with an open door in between. It can be seen that the level of the normalized signal is significantly higher because the level of noise within all the signals can be assumed to be the same.

[0291] The general problem of detecting speakers in different rooms can be solved by comparing different speakers’ delay waveforms with each other because the relative level of the direct peak gives an indication of the rendering quality factor. If the direct peak of a speaker has a very low level when compared to the direct peak of other speakers (more than what is expected after correcting using the tuning time sensitivity value) then this indicates a low quality factor.

[0292] Other alternative implementations may look at the relative SNR of the signal to determine whether the speaker is in another room. There is a signal level below which noD25030W001 reasonable device that is in the same room will produce with any microphone. This can be used as an absolute threshold to augment the quality factor.

[0293] Different Ways of Using the Rendering Quality Factor

[0294] There are a few alternative ways that an implementation may use the rendering quality factor for generating coefficients.

[0295] The rendering quality factor can be used directly as a penalty within the cost function(s) of a flexible rendering coefficient generator.

[0296] An alternative embodiment would be to map the rendering quality factor to an equivalent loudspeaker rotation factor. The mapping of quality factor values to a level of rotation of the speaker away from the listener can then be used directly in the rendering coefficient generation.

[0297] Some implementations may tweak a distance model based upon the rendering quality factor. For example, some distance models have an implicit critical distance where the diffuse to direct ratio is assumed. The quality factor may be used in those implementations for augmenting such distance models.

[0298] Another alternative embodiment is to reduce the amount of virtualization or change the range at which virtualization is attempted based on the rendering quality factor. If all the speakers within a constellation have a low rendering quality factor, the rendering module (e.g., the Tenderer 603 of Figure 6) may disable the virtualization and revert back to a different panning approach, for example center of mass amplitude panning.

[0299] Another alternative embodiment is to show the listener the rendering quality factor directly (e.g., on a graphical user interface (GUI) presented on a display) such that the user has a known metric to optimize their listening experience. Some such embodiments may use a lookup table that includes a set of example quality factors that correspond to various scenarios described above. A GUI may prompt the user with specific text that corresponds to the likely scenario. For example, if the control system has determined that the speaker is likely to be pointing away from the user, the text may say that the speaker is likely pointing away.

[0300] Furthermore, other aspects of a signal chain may be adjusted based upon the rendering quality factor. For example, some implementations may simplify audio object rendering, e.g., by not rendering particular audio objects based upon an audio object priority that is based, at least in part, on the rendering quality factor. In other examples, audio objects having a relatively low (e.g., below a threshold) audio object priority that is based, at least in part, on the rendering quality factor may be rendered at a different location, such as a locationD25030W001 that would involve playback by one or more loudspeakers having a sufficiently high rendering quality factor. Other examples may involve changing the way that dialog enhancement is performed. For example, the dialog enhancer may be turned to be more aggressive in its suppression of background noise or boosting of the dialog if the rendering quality factor of one or more speakers is low.

[0301] Figure 14 is a flow diagram that outlines another example of a disclosed method. The blocks of method 1400, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and / or described. The method 1400 may be performed by an apparatus or system, such as by the control system 150 of the apparatus 100 that is shown in Figure 1B and described above.

[0302] In this example, block 1405 involves determining, by a control system and based at least in part on microphone signals obtained by a microphone of a hand-held sensing device, level data corresponding to a combination of direct and indirect arrivals of audio data corresponding to one or more audio calibration signals emitted by each loudspeaker of a plurality of loudspeakers. As noted elsewhere herein, the term “level data” may include, for example, the amplitude (also referred to herein as magnitude), power, and / or energy of the peaks of delay waveforms that are based on audio calibration signals, such as DSSS audio calibration signals. In some examples, “level data” may include the amplitude, power and / or energy of samples in the noise region, direct response peaks, or peaks in the indirect response region of a delay waveform.

[0303] According to this example, block 1410 involves calibrating, by the control system, an audio data rendering process for audio played back by the plurality of loudspeakers, the audio data rendering process being based, at least in part, on the level data. In this example, the calibrating comprises determining one or more loudness correction gains and performing a correction for audio reproduced by at least one loudspeaker of the plurality of loudspeakers, the correction being based, at least in part, on the loudness correction gains.

[0304] In some examples, the one or more audio calibration signals may not be audible to humans. In some such examples, the one or more audio calibration signals may be played back at the same time that an audio content stream is being played back over the same loudspeaker system. According to some such examples, the audio calibration signal component of audio device playback sound (e.g., as represented by the curve 230 of Figure 2B) may be played back at a level sufficiently below the levels of the content stream component of the audio device playback sound (e.g., as represented by the curve 225 of Figure 2B) such that the audio calibration signal component is not audible to a human listener. In some examples,D25030W001 the microphone signals may be obtained while the hand-held sensing device is being pointed at each loudspeaker of the plurality of loudspeakers. In some examples, a frequency range of the audio calibration signals may vary over time.

[0305] According to some examples, the one or more audio calibration signals may use, or may be based on, a Code Division Multiple Access scheme. In some examples, the one or more audio calibration signals may include direct sequence spread spectrum (DSSS) signals. In some such examples, the DSSS signals may utilize orthogonal spreading codes. In some instances, the one or more calibration signals may be simultaneously emitted by two or more loudspeakers of the plurality of loudspeakers.

[0306] In some examples, the level data may be determined from a delay waveform from a matched filter. In some such examples, determining one or more loudness correction gains is based, in part, on one or more stored loudspeaker device descriptors indicating measured loudspeaker response.

[0307] According to some examples, the correction may be, or may include, a single broadband gain per loudspeaker. In other words, the correction may be the same across a frequency band. However, in some examples, the correction may be, or may include, a frequency-dependent gain per speaker. In other words, the correction may vary across frequency bands. In some such examples, the correction may vary according to measurements made with calibration signals having different frequency ranges. The correction determined for one frequency band may be different from the correction determined for another frequency band.

[0308] In some examples, the correction may be constrained to a particular threshold, such as a loudness correction threshold. Some examples are described above with reference to Figure 5, e.g., examples in which tuning-based loudness matching can be used as a ground truth to constrain the amount of loudness correction that occurs based on the relative sensitivities 502.

[0309] According to some examples, method 1400 may involve determining, by the control system, a rendering quality factor. In some examples, the rendering quality factor may be determined based, at least in part, on a comparison of the magnitude, energy or power of a peak corresponding to one of the direct acoustic paths with the magnitude, energy or power of a peak corresponding to one or more of the indirect acoustic paths. Alternatively, or additionally, the rendering quality factor may be determined based, at least in part, on a comparison of the magnitude, energy or power of the peak corresponding to one of the direct acoustic paths with noise magnitude, energy or power.D25030W001

[0310] In some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) level data of one or more peaks corresponding to one or more of the indirect acoustic paths of sound from the loudspeaker. According to some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker. In some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of the level data of the peak corresponding to the direct acoustic path and level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker. According to some examples, for each loudspeaker, determining the rendering quality factor may involve calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a noise level data. In some examples, the comparison may be based on level data in a delay waveform, e.g., on level data corresponding to a peak corresponding to the direct acoustic path, level data corresponding to one or more peaks corresponding to indirect acoustic paths, and / or noise level data in the delay waveform.

[0311] According to some examples, method 1400 may involve using, by the control system, the rendering quality factor to alter the audio data rendering process. For example, the speaker feed signals for a loudspeaker may be boosted or suppressed based, at least in part, on the loudspeaker’s rendering quality factor.

[0312] In some examples, method 1400 may involve using, by the control system, the rendering quality factor for configuring a dialog enhancer. For example, the dialog enhancer may be turned to be more aggressive in its suppression of background noise or boosting of the dialog if the rendering quality factor of one or more speakers is low.

[0313] According to some examples, method 1400 may involve using, by the control system, the rendering quality factor for modifying an audio object render, e.g., by rendering the audio object in a different location so that it will be reproduced by one or more loudspeakers having a higher rendering quality factor.

[0314] In some examples, the rendering quality factor may be used for selecting a different configuration within a codec bitstream, e.g., for selecting a dialog enhancementD25030W001 presentation or selecting a channel-based render (e.g., a Dolby 7.1 render, a Dolby 5.1 render or a stereo-based render) instead of an audio object-based render.

[0315] According to some examples, method 1400 may involve controlling, by the control system, a user interface to provide a user prompt based, at least in part, on the rendering quality factor. For example, the control system may show the listener the rendering quality factor directly (e.g., on a graphical user interface (GUI) presented on a display) such that the user has a known metric to optimize their listening experience. A GUI may prompt the user with specific text that corresponds to the likely scenario. For example, if the control system has determined that the speaker is likely to be pointing away from the user, the text may say that the speaker is likely pointing away.

[0316] In some examples, each loudness correction gain of the one or more loudness correction gains may be based on a relative loudness estimate. In some such examples, the relative loudness estimate may be a loudness estimate relative to the loudness of a reference loudspeaker. According to some examples, the relative loudness estimate may be based, at least in part, on a signal level estimate of a direct arrival peak. In some examples, the relative loudness estimate may be based, at least in part, on the signal level estimate minus a noise estimate. For example, the relative loudness estimate may be based, at least in part, on the signal level estimate of a direct arrival peak of a delay waveform minus a level estimate of a noise peak of the delay waveform or minus an estimate of the average noise of the delay waveform.

[0317] According to some examples, method 1400 may involve generating the audio calibration signals or audio calibration signal parameters. In some such examples, method 1400 may involve providing the audio calibration signals or the audio calibration signal parameters to each loudspeaker of the plurality of loudspeakers.

[0318] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer readable medium (e.g., a disc) which stores code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems can be or include a programmable general purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including an embodiment of disclosed methods or steps thereof. Such a general purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem that is programmed (and / orD25030WO01 otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0319] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform required processing on audio signal(s), including performance of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory) which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system may be implemented as a general purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0320] Another aspect of present disclosure is a computer readable medium (for example, a disc or other tangible storage medium) which stores code for performing (e.g., coder executable to perform) one or more examples of the disclosed methods or steps thereof.

[0321] While specific embodiments of the present disclosure and applications of the disclosure have been described herein, it will be apparent to those of ordinary skill in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the disclosure described and claimed herein. It should be understood that while certain forms of the disclosure have been shown and described, the disclosure is not to be limited to the specific embodiments described and shown or the specific methods described.

Claims

D25030W001CLAIMSWhat is claimed is:

1. An audio processing method, comprising:determining, by a control system and based at least in part on microphone signals obtained by a microphone of a hand-held sensing device, level data corresponding to both direct acoustic paths and indirect acoustic paths of sound corresponding to one or more audio calibration signals emitted by each loudspeaker of a plurality of loudspeakers; and calibrating, by the control system, an audio data rendering process for audio played back by the plurality of loudspeakers, the audio data rendering process being based, at least in part, on the level data, wherein the calibrating comprises determining one or more loudness correction gains and performing a correction for audio reproduced by at least one loudspeaker of the plurality of loudspeakers, the correction being based, at least in part, on the loudness correction gains.

2. The method of claim 1, wherein the one or more audio calibration signals are not audible to humans.

3. The method of claim 1 or claim 2, wherein the one or more audio calibration signals utilize a Code Division Multiple Access scheme.

4. The method of any one of claims 1-3, wherein the one or more audio calibration signals include direct sequence spread spectrum (DSSS) signals utilizing orthogonal spreading codes.

5. The method of any one of claims 1-4, wherein the level data is determined from a delay waveform from a matched filter.

6. The method of claim 5, wherein determining one or more loudness correction gains is based, in part, on one or more stored loudspeaker device descriptors indicating measured loudspeaker response.

7. The method of any one of claims 1-6, wherein the correction comprises a single broadband gain per speaker.

8. The method of any one of claims 1-7, wherein the correction comprises a frequencydependent gain per speaker.D25030WO01 9. The method of any one of claims 1-8, wherein the correction is constrained to a loudness correction threshold.

10. The method of any one of claims 1-9, further comprising determining, by the control system, a rendering quality factor based at least in part on a comparison of the level data of a peak corresponding to one of the direct acoustic paths with level data of a peak corresponding to one or more of the indirect acoustic paths, a comparison of the level data of the peak corresponding to one of the direct acoustic paths with noise level data, or both.

11. The method of claim 10, wherein, for each loudspeaker, determining the rendering quality factor involves calculating a ratio between (a) level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) level data of one or more peaks corresponding to one or more of the indirect acoustic paths of sound from the loudspeaker.

12. The method of claim 10, wherein, for each loudspeaker, determining the rendering quality factor involves calculating a ratio between (a) the level data of the peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker.

13. The method of claim 10, wherein, for each loudspeaker, determining the rendering quality factor involves calculating a ratio between (a) the level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) a sum of the level data of the peak corresponding to the direct acoustic path and level data of a plurality of peaks corresponding to the indirect acoustic paths of sound from the loudspeaker.

14. The method of claim 10, wherein, for each loudspeaker, determining the rendering quality factor involves calculating a ratio between (a) the level data of a peak corresponding to the direct acoustic path of sound from the loudspeaker and (b) the noise level data.

15. The method of any one of claims 10-14, further comprising using, by the control system, the rendering quality factor to alter the audio data rendering process.

16. The method of any one of claims 10-15, further comprising using, by the control system, the rendering quality factor for configuring a dialog enhancer.

17. The method of any one of claims 10-16, further comprising using, by the control system, the rendering quality factor for modifying an audio object render.D25030WO01 18. The method of any one of claims 10-17, further comprising using, by the control system, the rendering quality factor for selecting a different configuration within a codec bitstream.

19. The method of claim 18, wherein selecting a different configuration within a codec bitstream involves one or more of selecting a dialog enhancement presentation or selecting a channel -based render instead of an audio object-based render.

20. The method of any one of claims 10-19, further comprising controlling, by the control system, a user interface to provide a user prompt based, at least in part, on the rendering quality factor.

21. The method of any one of claims 1-20, wherein the one or more audio calibration signals are simultaneously emitted by two or more loudspeakers of the plurality of loudspeakers.

22. The method of any one of claims 1-21, wherein the microphone signals are obtained while the hand-held sensing device is being pointed at each loudspeaker of the plurality of loudspeakers.

23. The method of any one of claims 1-22, wherein a frequency range of the audio calibration signals varies over time.

24. The method of any one of claims 1-23, wherein each loudness correction gain of the one or more loudness correction gains is based on a relative loudness estimate and wherein the relative loudness estimate comprises a loudness estimate relative to a loudness of a reference loudspeaker.

25. The method of claim 24, wherein the relative loudness estimate is based, at least in part, on a signal level estimate of a direct arrival peak.

26. The method of claim 25, wherein the relative loudness estimate is based, at least in part, on the signal level estimate minus a noise estimate.

27. The method of any one of claims 1-26, further comprising generating the audio calibration signals or audio calibration signal parameters and providing the audio calibration signals or the audio calibration signal parameters to each loudspeaker of the plurality of loudspeakers.D25030WO01 28. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any one of claims 1-27.

29. A computer program including instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any one of claims 1-27.

30. A computing apparatus, comprising:at least one processor;a display; andmemory storing instructions, which when executed by the at least one processor, cause the computing apparatus to perform the method of any one of claims 1-27.