Methods, apparatus and systems for clock drift estimation and mitigation in asynchronous listener centric acoustic mapping
Patent Information
- Application Number
- PCT/US2026/017044
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2026-02-09
- Filing Date
- 2026-02-27
- Publication Date
- 2026-09-10
Smart Images

Figure US2026017044_10092026_PF_FP_ABST
Abstract
Description
D25032WO01 METHODS, APPARATUS AND SYSTEMS FOR CLOCK DRIFT ESTIMATION AND MITIGATION IN ASYNCHRONOUS LISTENER CENTRIC ACOUSTIC MAPPINGCROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the priority benefit of U.S. Provisional Patent Application Nos. 63 / 978,330 filed February 9, 2026, and 63 / 767,283 filed March 5, 2025, each of which is hereby incorporated by reference in their entireties.TECHNICAL FIELD
[0002] This disclosure pertains to audio processing systems and methods, particularly audio processing systems that involve listener-centric acoustic mapping (LCAM).BACKGROUND
[0003] Audio devices and systems are widely deployed. Although existing systems and methods for loudspeaker mapping and flexible rendering are known, improved systems and methods would be desirable.NOTATION AND NOMENCLATURE
[0004] Throughout this disclosure, including in the claims, the terms “speaker,” “loudspeaker” and “audio reproduction transducer” are used synonymously to denote any sound-emitting transducer (or set of transducers). A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single, common speaker feed or by multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuitry branches coupled to the different transducers.
[0005] Throughout this disclosure, including in the claims, the expression performing an operation “on” a signal or data (e.g., filtering, scaling, transforming, or applying gain to, the signal or data) is used in a broad sense to denote performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or pre-processing prior to performance of the operation thereon).
[0006] Throughout this disclosure including in the claims, the expression “system” is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, inD25032W001 which the subsystem generates M of the inputs and the other X - M inputs are received from an external source) may also be referred to as a decoder system.
[0007] Throughout this disclosure including in the claims, the term “processor” is used in a broad sense to denote a system or device programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chip set), a digital signal processor programmed and / or otherwise configured to perform pipelined processing on audio or other sound data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chip set.
[0008] Throughout this disclosure including in the claims, the term “couples” or “coupled” is used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections.
[0009] As used herein, a “smart device” is an electronic device, generally configured for communication with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near-field communication, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., that can operate to some extent interactively and / or autonomously. Several notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smartwatches, smart bands, smart key chains and smart audio devices. The term “smart device” may also refer to a device that exhibits some properties of ubiquitous computing, such as artificial intelligence.
[0010] Herein, we use the expression “smart audio device” to denote a smart device which is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of virtual assistant functionality). A singlepurpose audio device is a device (e.g., a television (TV)) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker and / or at least one camera), and which is designed largely or primarily to achieve a single purpose. For example, although a TV typically can play (and is thought of as being capable of playing) audio from program material, in most instances a modern TV runs some operating system on which applications run locally, including the application of watching television. In this sense, a single-purpose audio device having speaker(s) and microphone(s) is often configured to run a local application and / or service to use the speaker(s) and microphone(s) directly. Some singlepurpose audio devices may be configured to group together to achieve playing of audio over a zone or user configured area.D25032W001
[0011] One common type of multi-purpose audio device is an audio device that implements at least some aspects of virtual assistant functionality, although other aspects of virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multi-purpose audio device is configured for communication. Such a multi-purpose audio device may be referred to herein as a “virtual assistant.” A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide an ability to utilize multiple devices (distinct from the virtual assistant) for applications that are in a sense cloud-enabled or otherwise not completely implemented in or on the virtual assistant itself. In other words, at least some aspects of virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which a virtual assistant may communication via a network, such as the Internet. Virtual assistants may sometimes work together, e.g., in a discrete and conditionally defined way. For example, two or more virtual assistants may work together in the sense that one of them, e.g., the one which is most confident that it has heard a wakeword, responds to the wakeword. The connected virtual assistants may, in some implementations, form a sort of constellation, which may be managed by one main application which may be (or implement) a virtual assistant.
[0012] Herein, “wakeword” is used in a broad sense to denote any sound (e.g., a word uttered by a human, or some other sound), where a smart audio device is configured to awake in response to detection of (“hearing”) the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, to “awake” denotes that the device enters a state in which it awaits (in other words, is listening for) a sound command. In some instances, what may be referred to herein as a “wakeword” may include more than one word, e.g., a phrase.
[0013] Herein, the expression “wakeword detector” denotes a device configured (or software that includes instructions for configuring a device) to search continuously for alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wakeword event is triggered whenever it is determined by a wakeword detector that the probability that a wakeword has been detected exceeds a predefined threshold. For example, the threshold may be a predetermined threshold which is tuned to give a reasonable compromise between rates of false acceptance and false rejection. Following a wakeword event, a device might enter a state (which may be referred to as an “awakened” state or a stateD25032W001 of “attentiveness”) in which it listens for a command and passes on a received command to a larger, more computationally-intensive recognizer.
[0014] As used herein, the terms “program stream” and “content stream” refer to a collection of one or more audio signals, and in some instances video signals, at least portions of which are meant to be heard together. Examples include a selection of music, a movie soundtrack, a movie, a television program, the audio portion of a television program, a podcast, a live voice call, a synthesized voice response from a smart assistant, etc. In some instances, the content stream may include multiple versions of at least a portion of the audio signals, e.g., the same dialogue in more than one language. In such instances, only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is intended to be reproduced at one time.
[0015] As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “or” is to be read as “and / or” unless the context clearly indicates otherwise. The term “based on” is to be read as “based at least in part on.” The term “one example implementation” and “an example implementation” are to be read as “at least one example implementation.” The term “another implementation” is to be read as “at least one other implementation.” The terms “determined,” “determines,” or “determining” are to be read as obtaining, receiving, computing, calculating, estimating, predicting, or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.
[0016] Moreover, as used herein the following acronyms have the following meanings:• BPSK: Binary Phase Shift Keying• CDMA: Code Domain Multiple Access• dB: Decibel• dBSPL: Sound Pressure Level in decibel units• DSSS: Direct Sequence Spread Spectrum• DW: Delay Waveform• IMU: Inertial Measurement Unit• LCAM: Listener-Centric Acoustic Mapping• SPL: Sound Pressure Level• SNR: Signal-To-Noise RatioSUMMARYD25032W001
[0017] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some instances, the method(s) may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some methods involve estimating, by a control system, a clock drift between a playback system clock domain of a playback system and a microphone clock domain of a microphone of a hand-held device, the clock drift being a rate of change of instantaneous offset between the playback system clock domain and the microphone clock domain. In some examples, the estimating may be based on multiple audio calibration signals played back by each loudspeaker of a plurality of loudspeakers of the playback system and captured by the microphone over a capture time interval. According to some examples, the estimating may involve processing microphone signals from the microphone using reference calibration signals.
[0018] Some methods may involve asynchronously mitigating, by the control system and after the capture time interval, the clock drift. As used herein, the term “asynchronous” refers to, for example, mitigating the clock drift of previously-recorded measurements. This type of asynchronous mitigation of clock drift differs from, for example, synchronizing the playback system clock domain and the microphone clock domain for future measurements.
[0019] According to some examples, the estimating may involve estimating times of arrival (ToAs) of audio calibration signals from each loudspeaker of the plurality of loudspeakers over multiple time periods and estimating the clock drift based on estimated ToAs. In some examples, the mitigating may involve applying a temporally variant offset to the estimated ToAs, the temporally variant offset being proportional to an estimated clock drift.
[0020] In some examples, the estimating may be based on metrics of received audio calibration signals that are used to direct a discrete search. According to some examples, the estimating may be based on kurtosis of received audio calibration signals. The kurtosis may be used as a measure of a mismatch between the microphone clock domain and the playback system clock domain.
[0021] Some methods may involve resampling received microphone signals at a new rate to produce a resampled microphone signal, determining if the kurtosis of an inferred transfer function signal between reference signal and the resampled microphone signal increased or decreased as compared to the kurtosis of the received microphone signals, selecting a new resampling rate and continuing a process of resampling and kurtosis change evaluation until a convergence criterion is met. The convergence criterion may, for example, be a kurtosis threshold or a kurtosis change threshold. According to some examples, method 1000 may involve resampling received microphone signals on a search grid of resamplingD25032WO01 factors, conducting a brute-force search and estimating a local maximum of kurtosis corresponding to the search grid.
[0022] According to some examples, the mitigation may be performed coherently. In some examples, performing the mitigating coherently may involve resampling the microphone signals at a corrected rate. In some examples, performing the mitigating coherently may involve resampling the reference calibration signals at a corrected rate to produce corrected reference calibration signals. The estimating may involve processing the microphone signals using first matched filters based on the reference calibration signals. The mitigating may involve producing second matched filters based on the corrected reference calibration signals and processing the microphone signals using the second matched filters.
[0023] However, in some examples the mitigation may be performed incoherently. For example, the estimating may involve estimating times of arrival (ToA) of audio calibration signals from each loudspeaker of the plurality of loudspeakers to produce ToA estimates. Performing the mitigating incoherently may involve augmenting of a time base of the ToA estimates.
[0024] According to some examples, the hand-held device may be a cellular telephone or a television remote control device. In some examples, the multiple audio calibration signals may be selected based upon measured movement of the hand-held device. The measured movement may be estimated from data quality of the microphone signals, data quality of IMU signals, or a combination thereof.
[0025] Some methods may involve causing, by the control system, a user interface to provide a user prompt to perform additional measurements. For example, method 1000 may involve causing, by the control system, a display to provide a visual prompt to perform additional measurements and / or causing, by the control system, one or more loudspeakers to provide an audio prompt to perform additional measurements.
[0026] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus is, or includes, an audio processing system having an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof.
[0027] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on oneD25032W001 or more computer-readable, non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more non-transitory media having software stored thereon.
[0028] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1A shows an example of implementing an LCAM process in an audio environment.
[0030] Figure 1B is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.
[0031] Figure 2A shows examples of an audio DSSS signal and an audio playback signal.
[0032] Figure 2B is a graph that shows examples of the powers of two DSSS signals with different bandwidths but located at the same central frequency.
[0033] Figure 3 shows an example of a delay waveform (DW).
[0034] Figure 4 shows examples of audio samples in the playback domain and corresponding audio samples in the record domain.
[0035] Figure 5 is a graph that shows locations of loudspeakers and a calibration device in a playback environment.
[0036] Figure 6 shows a graph that shows a series of delay waveforms that are “walking” due to clock drift between the playback domain and the record domain.
[0037] Figure 7 shows an example of how an effective audio sample rate difference between the playback domain and the record domain can be compensated for according to a resampling process.
[0038] Figure 8: Schematic showing the resampling search for maximum kurtosis to determine the best estimate of Δs.
[0039] Figure 9 shows examples of loudspeaker range estimates of an LCAM process before and after clock drift mitigation.
[0040] Figure 10 is a flow diagram that outlines another example of a disclosed method.D25032W001 DETAILED DESCRIPTION OF EMBODIMENTS
[0041] Flexible rendering is a playback solution which delivers immersive audio experiences from a constellation of loudspeakers that can be flexibly placed around the room, not necessarily in locations that conform to a canonical surround sound layout such as Dolby 5.1 or 7.1. A larger number of loudspeakers allows for greater immersion, because the spatiality of the media presentation may be leveraged. In some instances, flexible rendering may involve rendering to more than one type of loudspeaker. To configure a flexible Tenderer, a map needs to be created that describes the layout of the loudspeakers and optionally the position of a listener in a playback environment.
[0042] Some previously-deployed acoustic mapping solutions for flexible rendering have focused on the use of microphones within the speakers, such as in the case of smart speakers, to build a map of a loudspeaker constellation instead of being listener-centric. Such solutions are constrained in their scope of support to loudspeakers with in-built microphones, such as smart speakers.
[0043] This disclosure provides listener-centric acoustic mapping solutions for flexible rendering. The methods disclosed herein do not require any sound capturing apparatus within the loudspeakers, thereby expanding the scope of mapping beyond smart speakers to all loudspeakers in general.
[0044] Some listener-centric acoustic mapping (LCAM) calibration processes involve a user pointing a sensing device at each of the loudspeakers in a playback environment to be calibrated sequentially. The system (e.g., the sensing device) may prompt the user to point the sensing device at a specific loudspeaker, or may allow the user to point the sensing device at the loudspeakers in any order the user chooses, depending on the particular implementation. The control system may prompt the user (e.g., via a user interface system of the sensing device) when to point the sensing device at the next loudspeaker in either case. As used herein, the terms “point ” “pointing,” etc., used in the context of pointing the sensing device at a loudspeaker, may refer to various gestures by the user and / or orientations of the sensing device, but generally refer to the user holding the sensing device in the direction of the loudspeaker, relative to the person’s location, such that the sensing device will be between the person and the loudspeaker. In some cases, pointing the sensing device at a loudspeaker may involve the user extending one or both of the user’s hands that is / are holding the sensing device in the direction of the loudspeaker, extending at least one of the user’s arms, a corresponding hand of which is holding the sensing device, in the direction of the loudspeaker, etc.D25032W001
[0045] Figure 1A shows an example of implementing an LCAM process in an audio environment. As with other figures provided herein, the types and numbers of elements shown in Figure 1A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, other audio environments 115 may include more than three loudspeakers.
[0046] In this example, a user 105 — also referred to herein as a person or a listener — is holding a mobile device 100 that includes a directionally-sensitive sensor system 130 and a microphone system 140. The mobile device 100 is an example of what may be referred to herein as a “calibration device,” a “handheld calibration device” or a “handheld sensing device.” The mobile device 100 may be cellular telephone, a remote control device, a tablet device, or another type of mobile device. The sensor system 130 may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more accelerometers — which may collectively be referred to an inertial measurement unit (IMU) — one or more cameras, one or more radios or combinations thereof. In some examples, the sensor system 130 may be configured for light detection and ranging (LiDAR).
[0047] According to this example, the user 105 provides input 101 to the mobile device 100 to initiate a calibration process. Calibration signals 120A-C then play out from all loudspeakers 110A-C in the constellation. In some examples, the calibration signals 120A-C may be audio signals that are included with rendered audio content that is played back by the loudspeakers 110A-C. Simultaneously, sound data — including the calibration signals 120A-C — is collected by the microphone system 140. In this example, the user 105 is required to reposition (rotate and / or translate, as indicated by the arrow 102) the mobile device 100 for the collection of azimuth data. According to some examples, the mobile device 100 may provide a prompt — such as an audio prompt and / or a visual prompt — to the user 105 indicating when to point the mobile device 100 to each of the loudspeakers during the calibration process. In some examples, the mobile device 100 may indicate the progress of the calibration process, e.g., by displaying a graphical user interface (GUI) that indicates the estimated time remaining for obtaining calibration signals from a particular loudspeaker, the percentage of the calibration process that has been completed for a particular loudspeaker, etc.
[0048] Collecting the azimuth data may involve obtaining azimuth angles for each of the loudspeakers 110A-C in a loudspeaker plane, in a listener plane, etc. In the example shown in Figure 1 A, the azimuth angle θA corresponding to the loudspeaker 110A is measured relative to the y axis of a coordinate system having x and y axes parallel to the floor of the audio environment 115 and having its origin inside of the user 105. In other examples, the azimuth angles may be measured relative to the x axis or to another axis. In some such examples, theD25032W001 mobile device 100 may include a user interface system. A control system of the mobile device 100 may be configured to prompt the user 105, via the user interface system, to position and re-position the mobile device 100 for the collection of azimuth data.
[0049] Various examples of calibration processes are disclosed in detail herein. In some examples, a control system of the mobile device 100 may be configured to perform some or all of the calibration process(es). Alternatively, or additionally, one or more other devices may be configured to perform some or all of the calibration process(es). In some such examples, one or more servers or one or more other devices, such as a television, a laptop computer or a smart home hub, may be configured to perform some or all of the calibration process(es), based at least in part on data obtained by the mobile device 100.
[0050] Figure 1B is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure IB are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 100 may be configured for performing at least some of the methods disclosed herein. In some implementations, the apparatus 100 may be, or may include, one or more components of an audio system. For example, the apparatus 100 may be an audio device, such as a smart audio device, in some implementations. In other examples, the apparatus 100 may be a mobile device (such as a cellular telephone or a remote control), a laptop computer, a tablet device, a television or another type of device. The mobile device 100 of Figure 1A is an instance of the apparatus 100 of Figure IB.
[0051] According to some examples, the audio environment 100 of Figure 1A may include an orchestrating device, such as what may be referred to herein as a smart home hub. The smart home hub (or other orchestrating device) may be an instance of the apparatus 100. In some implementations, one or more of the loudspeakers 110A-110C may be capable of functioning as an orchestrating device.
[0052] According to some alternative implementations the apparatus 100 may be, or may include, a server. In some such examples, the apparatus 100 may be, or may include, an encoder. Accordingly, in some instances the apparatus 100 may be a device that is configured for use within an audio environment, such as a home audio environment, whereas in other instances the apparatus 100 may be a device that is configured for use in “the cloud,” e.g., a server.
[0053] In this example, the apparatus 100 includes a sensor system 130, a microphone system 140 and a control system 150. The microphone system 140 includes one or moreD25032W001 microphones. According to some examples, the microphone system 140 includes an array of microphones. The array of microphones may, in some instances, be configured for receiveside beamforming, e.g., according to instructions from the control system 150. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to instructions from the control system 150. Alternatively, or additionally, the control system 150 may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to microphone signals received from the microphone system 140.
[0054] In some implementations, the control system 150 may be configured for performing, at least in part, the methods disclosed herein. The control system 150 may, for example, include a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.
[0055] In some implementations, the control system 150 may reside in more than one device. For example, in some implementations a portion of the control system 150 may reside in a device within one of the environments depicted herein and another portion of the control system 150 may reside in a device that is outside the environment, such as a server, a mobile device (e.g., a smartphone or a tablet computer), etc. In other examples, a portion of the control system 150 may reside in a device within one of the environments depicted herein and another portion of the control system 150 may reside in one or more other devices of the environment. For example, control system functionality may be distributed across multiple smart audio devices of an environment, or may be shared by an orchestrating device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 150 may reside in a device that is implementing a cloud-based service, such as a server, and another portion of the control system 150 may reside in another device that is implementing the cloud-based service, such as another server, a memory device, etc. The interface system 145 also may, in some examples, reside in more than one device.
[0056] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 155 shown in Figure IB and / or in the control system 150. Accordingly,D25032W001 various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may, for example, be executable by one or more components of a control system such as the control system 150 of Figure IB.
[0057] The sensor system 130 may include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more cameras, one or more radios or combinations thereof. In some implementations, the sensor system 130 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some implementations, the sensor system 130 may include one or more cameras. In some implementations, the cameras may be free-standing cameras. In some examples, one or more cameras of the sensor system 130 may reside in a smart audio device, which may be a single purpose audio device or a virtual assistant. In some such examples, one or more cameras of the sensor system 130 may reside in a television, a mobile phone or a smart speaker. In some examples, the apparatus 100 may be configured to receive sensor data for one or more sensors residing in one or more other devices in an audio environment via the interface system 145.
[0058] The interface system 145 — when present — may, in some implementations, include a wired or wireless interface that is configured for communication with one or more other devices of an audio environment. The audio environment may, in some examples, be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. The interface system 145 may, in some implementations, be configured for exchanging control information and associated data with audio devices of the audio environment. The control information and associated data may, in some examples, pertain to one or more software applications that the apparatus 100 is executing.
[0059] The interface system 145 may, in some implementations, be configured for receiving, or for providing, a content stream. The content stream may include audio data. The audio data may include, but may not be limited to, audio signals. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. Metadata may, for example, have been provided by what may be referred to herein as an “encoder.” In some examples, the content stream may include video data and audio data corresponding to the video data.
[0060] The interface system 145 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB)D25032W001 interfaces). According to some implementations, the interface system 145 may include one or more wireless interfaces, e.g., configured for Wi-Fi or Bluetooth™ communication.
[0061] The interface system 145 may, in some examples, include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 145 may include one or more interfaces between the control system 150 and a memory system, such as the optional memory system 155 shown in Figure IB. However, the control system 150 may include a memory system in some instances. The interface system 145 may, in some implementations, be configured for receiving input from one or more microphones in an environment.
[0062] In some implementations, the apparatus 100 may include the display system 160 shown in Figure IB. The display system 160 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the display system 160 may include one or more organic light-emitting diode (OLED) displays. In some examples, the display system 160 may include one or more displays of a smart audio device. In other examples, the display system 160 may include a television display, a laptop display, a mobile device display, or another type of display. In some examples wherein the apparatus 100 includes the display system 160, the sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 160. According to some such implementations, the control system 150 may be configured for controlling the display system 160 to present one or more graphical user interfaces (GUIs). In some examples, a user interface system of the apparatus 100 may include the display system 160, a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 160, the microphone system 140, the loudspeaker system 110, or combinations thereof.
[0063] According to some implementations, the apparatus 100 may include the optional loudspeaker system 110 shown in Figure IB. The optional loudspeaker system 110 may include one or more loudspeakers, which also may be referred to herein as “speakers” or, more generally, as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), the apparatus 100 may not include a loudspeaker system 110.
[0064] According to some such examples the apparatus 100 may be, or may include, a smart audio device. In some such implementations the apparatus 100 may be, or may include, a wakeword detector. For example, the apparatus 100 may be, or may include, a virtual assistant.Calibration SignalsD25032W001
[0065] Various types of calibration signals may be used in the methods disclosed in this document. Under sequential playback conditions, any calibration signal with a spectrally-rich autocorrelation in the bandwidth of concern could be used to measure a DW-like response. Examples include time-stretched-pulse (TSP) calibration signals, maximum length sequence (MLS) calibration signals, sine sweep calibration signals, etc. In some implementations, the calibration signals may be Direct Sequence Spread Spectrum (DSSS) signals. DSSS calibration signals are particularly beneficial for implementations involving the simultaneous playback of calibration signals from multiple loudspeakers of a playback environment.
[0066] DSSS signals have previously been deployed in the context of telecommunications. When DSSS signals are used in the context of telecommunications, DSSS signals are used to spread out the transmitted data over a wider frequency range before it is sent over a channel to a receiver. Most or all of the disclosed implementations, by contrast, do not involve using DSSS signals to modify or transmit data. Instead, such disclosed implementations involve sending DSSS signals between audio devices of an audio environment. What happens to the transmitted DSSS signals between transmission and reception is, in itself, the transmitted information. That is one significant difference between how DSSS signals are used in the context of telecommunications and how DSSS signals are used in the disclosed implementations.
[0067] Moreover, the disclosed implementations involve sending and receiving acoustic DSSS signals, not sending and receiving electromagnetic DSSS signals. In many disclosed implementations, the acoustic DSSS signals are inserted into a content stream that has been rendered for playback, such that the acoustic DSSS signals are included in played-back audio. According to some such implementations, the acoustic DSSS signals are not audible to humans, so that a person in the audio environment would not perceive the acoustic DSSS signals, but would only detect the played-back audio content.
[0068] Another difference between the use of acoustic DSSS signals as disclosed herein and how DSSS signals are used in the context of telecommunications involves what may be referred to herein as the “near / far problem.” In some instances, the acoustic DSSS signals disclosed herein may be transmitted by, and received by, many audio devices in an audio environment. The acoustic DSSS signals may potentially overlap in time and frequency. Some disclosed implementations rely on how the DSSS spreading codes are generated to separate the acoustic DSSS signals. In some instances, the audio devices may be so close to one another that the signal levels may encroach on the acoustic DSSS signal separation, so it may be difficult to separate the signals. That is one manifestation of the near / far problem, some solutions for which are disclosed herein.D25032W001
[0069] A DSSS signal typically includes data, a carrier signal and a spreading code. To implement DSSS, in some examples binary phase shift keying (BPSK) modulation may be utilized. If the need to transmit data over the channel is omitted, then one can express the BPSK signal, s(t), as follows:s[n] = Ac[n] cos(φ0n) (1)
[0070] In the foregoing equation, A represents the amplitude of the DSSS signal, c[n] represents the spreading code, and cos() represents a sinusoidal carrier with a digital frequency of φ0derived from a physical carrier frequency of f0.
[0071] Figure 2A shows examples of an audio DSSS signal and an audio playback signal. The curve 230 in Figure 2A corresponds to an example of s[n] in the equation above. The DSSS signal 230 includes a main lobe 235, in which the highest-level portions of the DSSS signal 230 can be found. One of the potential advantages of some disclosed implementations involving acoustic DSSS signals is that by spreading the signal one can reduce the perceivability of the DSSS signal component of audio device playback sound, because the amplitude of the DSSS signal component is reduced for a given amount of energy in the acoustic DSSS signal.
[0072] This allows us to place the DSSS signal component of audio device playback sound (e.g., as represented by the curve 230 of Figure 2A) at a level sufficiently below the levels of the content stream component of the audio device playback sound (e.g., as represented by the curve 225 of Figure 2A) such that the DSSS signal component is not perceivable to a listener. Some disclosed implementations exploit the masking properties of the human auditory system to optimize the parameters of the DSSS signal in a way that maximizes the signal-to-noise ratio (SNR) of the derived DSSS signal observations and / or reduces the probability of perception of the DSSS signal component. Some disclosed examples involve applying a weight to the levels of the content stream component and / or applying a weight to the levels of the DSSS signal component. Some such examples apply noise compensation methods, wherein the acoustic DSSS signal component is treated as the signal and the content stream component is treated as noise. Some such examples involve applying one or more weights according to (e.g., proportionally to) a play / listen objective metric.DSSS Spreading Codes
[0073] As noted elsewhere herein, in some examples calibration signals may be, or may include, one or more DSSS signals based on DSSS spreading codes. The spreading codes used to spread the carrier wave in order to create the DSSS signal(s) are extremely important. The set of DSSS spreading codes is preferably selected so that the corresponding DSSS signals have the following properties:D25032W001 1. A sharp main lobe in the autocorrelation waveform;2. Low sidelobes at non-zero delays in the autocorrelation waveform;3. Low cross-correlation between any two spreading codes within the set of spreading codes to be used if multiple devices are to access the medium simultaneously (e.g., to simultaneously play back modified audio playback signals that include a DSSS signal component); and4. The DSSS signals are unbiased, (have zero DC component).
[0074] The family of spreading codes (e.g., Gold codes, which are commonly used in the GPS context) typically characterizes the above four points. If multiple audio devices are all playing back modified audio playback signals that include a DSSS signal component simultaneously and each audio device uses a different spreading code (with good crosscorrelation properties, e.g., low cross-correlation), then a receiving audio device should be able to receive and process all of the acoustic DSSS signals from all of the loudspeakers producing the acoustic DSSS signals simultaneously by using a code domain multiple access (CDMA) method. By using a CDMA method, multiple audio devices can send acoustic DSSS signals simultaneously, in some instances using a single frequency band. Spreading codes may be generated during run time and / or generated in advance and stored in a memory, e.g., in a data structure such as a lookup table.
[0075] Choosing a particular set of codes, c(t), that have good auto- and crosscorrelation properties allows for employing a code division multiple access (CDMA) scheme. As noted above, one such set of codes that is widely used is the Gold Code family. CDMA schemes, including those utilizing Gold codes, are widely used for telecommunications, positioning, navigation and timing applications. This results in the simultaneous use of set of signals that can be expressed as follows:ssi[n] = Aici[n] cos(φ0,in) (2)In Equation 2, the i subscript indicates the ithsignal.
[0076] In the embodiments disclosed in this document, DSSS refers to a set of signals which are, in general, synthesized using Gold Codes as the spreading code c[n]. While only BPSK is described in the examples provided in this document, quadrature phase shift keying (QPSK) can also be used.
[0077] A Gold Code of order K, is a binary sequence of length L = 2K− 1 This is then sampled at a rate fc, the chipping rate of the sequence. This chipping rate specifies how fast each chip of the binary sequence is transmitted. Note, the term chip, not bit, is used for the Gold Code as it must be known by both the transmitter and receiver ahead of time and no data is transmitted in the Gold Code itself. If the audio sampling rate is fsthen φc= fc / fsis theD25032W001 digital frequency of the code. A typical DSSS calibration signal for the purposes of acoustic calibration may have fc= 6000, f0= 12000, K = 13 which results in a code periodicity of approximately 1.365 seconds or M = 65528 samples.
[0078] The sampled spreading code c[n] may be obtained by sampling the Gold Code sequence, GC, using the digital frequency of the code, e.g., as follows:c[n] = GC[floor(φcn)] (3)
[0079] Figure 2B is a graph that shows examples of the powers of two DSSS signals with different bandwidths but located at the same central frequency. In these examples, Figure 2B shows the spectra of two DSSS signals 235A and 235B that are both centered on the same center frequency 237. In some examples, the DSSS signal 235A may be produced by one audio device of an audio environment (e.g., by the audio device 110A of Figure 1A) and the DSSS signal 235B may be produced by another audio device of the audio environment (e.g., by the audio device HOB).
[0080] According to this example, the DSSS signal 235B is chipped at a higher rate (in other words, a greater number of bits per second are used in the spreading signal) than the DSSS signal 235A, resulting in the bandwidth 240B of the DSSS signal 235B being larger than the bandwidth 240A of the DSSS signal 235A. For a given amount of energy for each DSSS signal, the larger bandwidth of the DSSS signal 235B results in the amplitude and perceivability of the DSSS signal 235B being relatively lower than those of the DSSS signal 235A. A higher-bandwidth DSSS signal also results in higher delay-resolution of the baseband data products, leading to higher-resolution estimates of acoustic scene metrics that are based on the DSSS signal (such as time of flight estimates, time of arrival (To A) estimates, range estimates, direction of arrival (DoA) estimates, etc.). However, a higher-bandwidth DSSS signal also increases the noise-bandwidth of the receiver, thereby reducing the SNR of the extracted acoustic scene metrics. Moreover, if the bandwidth of a DSSS signal is too large, coherence and fading issues associated with the DSSS signal may become present.
[0081] The length of the spreading code used to generate a DSSS signal limits the amount of cross-correlation rejection. For example, a 10 bit Gold code has just -26dB rejection of an adjacent code. This may give rise to an instance of the above-described near / far problem, in which a relatively low-amplitude signal may be obscured by the cross correlation noise of another louder signal. Some of the novelty of the systems and methods described in this disclosure involves orchestration schemes that are designed to mitigate or avoid such problems.Receiving Calibration Signals
[0082] In the CDMA scheme, receiving individual signals can be done using a matched filter, which is optimal in the sense of maximizing the output signal-to-noise ratio (SNR) of theD25032W001 filter. During a calibration process, a microphone of a hand-held sensing device will produce a microphone signal d[n] containing a mixture of noise and a plurality of calibration signals. If multiple loudspeakers are simultaneously transmitting calibration signals, this microphone signal will contain the calibration signals it hears from multiple speakers after travelling through the playback environment to the microphone. After travelling through the playback environment, the calibration signals are altered (lower amplitude, delayed, etc.). The microphone signal also contains noise signals, etc. Moreover, in addition to noise, the calibration signals may be masked by audio content that is also being played back by the loudspeakers. This audio content is not used for the calibration measurement itself, but to improve the user experience.
[0083] According to some examples, as a modulated signal is received, a complex replica Tj[n] of the calibration signal for reception such as the following may be used:ri[n] = Aici[n] cos(φ0,in) + jAici[n] sin(φ0,in) (4)
[0084] In Equation 3, A represents the amplitude of the DSSS signal, c represents the spreading code, φ0represents the digital frequency of the carrier wave and i represents the index of the code and the index of a loudspeaker that is emitting calibration signals based on that code.
[0085] A control system may use the complex replica rt[n] of the calibration signal to produce j / j, the complex matched filter output for the ithcalibration signalyi[T] = Σm=0ri[T - m] d[m] (5)
[0086] Accordingly, yirepresents the circular convolution of riand d. In Equation 4, T represents the discrete time sample of the circular convolution signal, M represents the length of the code in samples and m represents the convolution summation variable. In some implementations, the control system may be configured to implement fast versions of this algorithm that utilize the Fourier transform and exploit the convolution theorem.Delay waveforms
[0087] The matched filter output yiis a complex signal. For the purposes of peak picking, level estimation, and range finding, only the envelope of the matched filter output is required. This is called the delay waveform (DW), which may be computed by either taking the magnitude or power of the complex matched filter output, yi. The power of the delay waveform may be computed as follows:DWi[T] = Real(yi[T])2+ Imag(yi[T])2(6A)
[0088] Figure 3 shows an example of a delay waveform (DW). In the graph shown in Figure 3, the horizontal axis indicates samples in a time interval and the vertical axis indicatesD25032W001 power in Watts. The direct response 305 of the DW 300 corresponds to the earliest peak that is above a noise threshold (e.g., above 1000 units, above 2000 units, etc.). The noise threshold may vary according to characteristics of the particular playback environment. Often, the direct response 305 peak is also the strongest (highest-power) peak. The direct response 305 corresponds to sound waves traversing a direct acoustic path from a loudspeaker to a listening position (or to a proxy for a listening position, such as the microphone of the hand-held sensing device 100).
[0089] The indirect response 310 is the remainder of the response. It is comprised of the reflections in the acoustic space. The indirect response 310 can, in some instances, carry on for a significant amount of time (such as hundreds of milliseconds).
[0090] The noise region 315 corresponds with samples obtained prior to the time of the direct response 305. Any samples occurring before the direct response 305 do not correspond to a physically possible acoustic path. Accordingly, samples in the noise region 315 are suitable for noise level estimation.
[0091] Unlike methods which utilize the Wiener filter to estimate an impulse response, some methods disclosed in this document use a matched filter yi[T], which produces an impulse response convolved with the autocorrelation of the calibration code, e.g., as follows:
[0092] yi[T] = h[T] ⊗ R[T] (6B)
[0093] In Equation 6, ® represents the circular convolution operator and R [T] represents the autocorrelation of the calibration signal. It may be assumed that there is insignificant carrier frequency mismatch and there is only a need to consider the delay component.
[0094] Estimating Parameters from the Delay Waveform
[0095] The lag, also referred to as a delay, of the direct response corresponds to the Time of Arrival (ToA) of the calibration signal. Because the sensing device and the playback system are not synchronized, the delay of the direct response does not correspond to the Time of Flight (ToF) that is linearly related to the distance between the phone and speaker. The ToA of the ithcalibration signal, τi, in the DW may be estimated using a peak picking algorithm, the most naive of these being the max peak picker:
[0096] τi= argmax(DWi) (6C)
[0097] More elaborate peak picking schemes can deal with strong reflections and obstructions in the playback environment.
[0098] The level of the direct response is a function of the relative loudness of the speaker, Gi, and the distance between the phone and the speaker, d;-. The relative loudnessD25032W001 corresponds to how loud the speaker is at a fixed distance playing the DSSS signal. In some examples, it is assumed that the speakers in the constellation have roughly the same target curve (the approximate frequency response of a loudspeaker).
[0099] The direct response power level can be modelled as follows: / -•.
[0100] Si= kτ̂'i,q(6D)
[0101] In Equation 6D, k represents a scaling factor associated with the gain of the calibration device. Here, it may be assumed that the calibration device is itself not calibrated and that an unknown scaling factor k means that the absolute level of any signal in the microphone cannot be measured.
[0102] The direct response level can be estimated by taking the value of the earliest peak in the delay waveform. There are many ways of estimating what the earliest peak is. One such approach is to find the absolute peak and then, within a short window (e.g., in the range of 80 to 120 milliseconds) earlier than this, to look for all peaks which are either 1) significantly above the noise floor (e.g., more than twice the power of the noise floor), or 2) not too far down from the absolute peak level (e.g., above 15% of the argmax, above 20% of the argmax, above 25% of the argmax, etc.). Once the earliest peak is found, the maximum of the earliest peak can be taken as the level estimate Si.
[0103] Alternatively, knowledge of the autocorrelation waveform (7?[T], the autocorrelation of the calibration signal) may be exploited. The shape of the autocorrelation waveform can be used to interpolate the samples surrounding the peak to refine the estimate of the signal level. This interpolation addresses the fact that the peak value corresponding to the direct response is likely to be located at some delay between samples. In some embodiments, the noise in the autocorrelation signal is ignored. The peak response of the DW may then be modelled as a triangle. Recalling that DW[τi] is the maximum value of the DW, then the estimate of the direct peak level, Sj, using the triangle signal model can be made as follows:
[0104] Si= (DW[τi] + dL + Mei) / 2 (6E)
[0105] In Equation 6E, Meirepresents the next largest sample corresponding to the peak and may be expressed as follows:
[0106] Mei= max(DW[τi− 1], DW[τi+ 1]). (6F)
[0107] In Equation 6F, dL represents the level change of the delay waveform between the samples and may be expressed as follows:
[0108] dL = −min(DW[τi− 1], DW[τi+ 1]) (6G)
[0109] This process comprises the autocorrelation shape based interpolation.D25032W001
[0110] The noise level, N, can be estimated by processing the samples in the noise region 315 of the DW. As yt[T] can be modelled as a complex Gaussian process, the noise samples in the power DW will follow a chi-squared distribution. If, instead of taking the power DW, the magnitude DW (MDW) is computed, the MDW may be used synonymously with DW (except where differentiation between them is required):
[0111] MDWi[τ] = √Real(yi[τ]')2+ Imag(yi[τ]')2(6H)
[0112] Then the noise-region follows a Rician distribution, and a Gaussian approximation can be sufficient in typical operating conditions. One way to estimate the noise level, aN, is to take the standard deviation of the noise only region Tw
[0113] σN= std (MDWi[τ]) ∀τ ∈ TN(61)
[0114] During a calibration process, the calibration signal may be played periodically until the calibration process is over. If there are Nddevices in the playback system and Q is the number of complete calibration periods played out, then there are QNddel ay waveforms to extract data from. Here, q indicates the current calibration period. If t represents the current time, then q may be expressed as follows:q = floor (7)In Equation 7, T represents the length of the calibration period, e.g., in seconds, and “floor” represents a function that rounds down to the nearest integer.
[0115] The impact of clock drift
[0116] The time of arrival (ToA) of the ithcalibration signal, Tj, in the DW may be estimated using a peak picking algorithm, the most naive of these being the max peak picker:τi= argmax(DWi) (8)
[0117] In Equation 8, τ̂irepresents the estimated ToA of the ithcalibration signal. More elaborate peak picking schemes can deal with strong reflections and obstructions.
[0118] As used herein, the term “local replica” refers to the calibration signal that is provided to the loudspeakers of a playback system for playback during a calibration process. In some examples, the loudspeakers will play back the calibration signal sequentially. When one loudspeaker is playing back the calibration signal, the other loudspeakers may be sent silence to avoid opening and closing the soundcard that is providing the local replica to the loudspeakers, such that the recording is only started once. This process ensures that all of the calibration signals and corresponding DWs are comparable with each other. As noted elsewhere herein, a variety of calibration signals may be used. These calibration signals may vary over time. The played-back calibration signals are captured by the microphone system ofD25032W001 a calibration device that is configured to obtain microphone data for an LCAM process. The loudspeakers will play back the calibration signals according to a clock domain of the playback system, which may be referred to herein as the “playback domain.” The played-back calibration signal will be captured and sampled according to a clock domain of the calibration device. The clock domain of the calibration device may be referred to herein as a “record domain.” If there is a clock frequency mismatch between the playback domain and the record domain, then the period of the local replica and the period of the calibration signal played out by the playback system will be different. As a result, the peak corresponding to the direct arrival, e.g., τ̂i, will appear to “walk” in subsequent DWs. As noted below, this phenomenon is referred to as “peak walking” and refers to an increase or decrease in the time of the direct arrival peak over time, depending on whether the calibration device’s audio sample rate or the playback system’s audio sample rate is larger. If the playback system’s clock rate is rpand the calibration device’s clock rate is rm, the clock drift parameter Ar may be expressed as follows:Δr = rm− rp(9 A)
[0119] The difference in clock rates between the playback domain and the record domain results in an effective (and often fractional) difference Δfsin the audio sample rate between the local replica and the recorded signal, which may be expressed as follows:(9B)
[0120] In Equation 9B, fmrepresents the calibration device’s audio sample rate and fprepresents the playback system’s audio sample rate. This difference Δfsin the audio sample rates causes a continuous change in the latency between audio samples in the two domains.
[0121] Figure 4 shows examples of audio samples in the playback domain and corresponding audio samples in the record domain. Figure 4 shows an example of audio samples 401a in the playback domain 405 and of audio samples 401b in the record domain 410. In the example shown in Figure 4, the effective audio sample rate is higher in the record domain 410 than in the playback domain 405. According to the example shown in Figure 4, the calibration time for loudspeaker 1 starts at time 415a and ends at time 415c. However, the audio sample at time 415a in the playback domain 405 corresponds to the audio sample at time 415b in the record domain 410. Similarly, the audio sample at time 415c in the playback domain 405 corresponds to the audio sample at time 415d in the record domain 410. In the example shown in Figure 4, the calibration time for loudspeaker 4 starts at time 415e. However, the audio sample at time 415e in the playback domain 405 corresponds to the audio sample at time 415f in the record domain 410. It may be seen that the time interval between time 415c and 415d is larger than the time interval between time 415a and time 415b, and thatD25032W001 the time interval between time 415e and 415f is yet larger than the time interval between time 415c and time 415d. This is an example of the continuous change in the latency between audio samples in the two domains.
[0122] The difference Δfsin the audio sample rate between the audio samples in the playback domain and the corresponding audio samples in the record domain causes the measured ToAs, relative to the calibration device, to drift over time. This ToA drift may be expressed as follows:TDoAij= dij / c + (Δfs / fm)ticJm
[0123] In Equation 10, TDoAij represents the relative Time Difference of Arrival at the calibration device between playback devices i (measured) and j (reference), dijrepresents the relative distance of devices i and j to the calibration device, tirepresents the time elapsed since the start of calibration and c represents the speed of sound.
[0124] Since the relative TDoAs are converted directly into relative ranges, this causes the range error of each subsequently measured DW to increase over time. As used herein, the term “range” corresponds to distance, particularly to the estimated distance from a particular loudspeaker to the calibration device.
[0125] Figure 5 is a graph that shows locations of loudspeakers and a calibration device in a playback environment. In this example, the calibration device 100 is located at the origin of the graph 500, such that the locations of loudspeakers 1, 2, 3, 4 and 5 are shown relative to the location of the calibration device 100. According to this example, the “ground truth” (GT) locations of the loudspeakers 1-5 are shown as squares and the measured locations, as determined by an LCAM process, are shown as circles. As exemplified by Figure 5, the range error of the measured locations (with respect to calibration device 100) increases with device (loudspeaker) index number, because the devices were measured in order of ascending index. In other words, the range error for loudspeaker 4 is greater than the range error for loudspeaker 3, the range error for loudspeaker 5 is greater than the range error for loudspeaker 4, etc. Accordingly, Figure 5 shows an example of the effect of constant clock drift on measured relative ranges.
[0126] Consider a speaker playing back calibration signals in the form of DSSS tones looping over (repeating) in isolation for an extended period of time. As shown in Figure 4, we would observe that the index of the device Time of Arrival either increases or decreases over time in each successively observed DW, depending on whether the calibration device’s audio sample rate or the playback system’s audio sample rate is larger. Figure 6 shows additional examples of this phenomenon, which is referred to as “peak walking.” If a loudspeaker in theD25032W001 collection of loudspeakers in the playback environment plays back DSSS calibration signals for a longer period of time, the peak will continue to walk further.
[0127] The range error observed therefore depends on which DW is selected for conversion to a relative distance, also referred to herein as range. The earlier in the calibration process the chosen DW, the less corrupted the result will be by the peak walking caused by the clock drift. This issue may be further complicated if the “dwell” or calibration device measurement times vary from one calibrated loudspeaker to another. In some examples, this variation in dwell time may be caused, at least in part, by adaptive progress functionality, e.g., based on responses to one or more user prompts indicating that a loudspeaker should be measured for a longer period of time. The relative distance calculation for a loudspeaker that has a shorter dwell time will be corrupted to a lesser degree than the relative distance range calculation for a loudspeaker with a longer dwell time.
[0128] Considering now that, in some examples, this loudspeaker may be one of multiple loudspeakers of a playback environment that are playing back calibration signals sequentially during a calibration process, the range error is further compounded by the change of “focus device,” in other words in the transitions from the end of a time interval during which one loudspeaker is being measured to the beginning of a time interval during which another loudspeaker is being measured. There is an additional latency during the transition between when the calibration device starts to record calibration signals played back by the next focus device (loudspeaker) and the loudspeaker plays back. This latency may be referred to herein as a device transition delay. The range estimation for the last loudspeaker to play back during the calibration process has a range error compounded by all of the sources of range error since the start of the recording, including all sources of range error introduced as each loudspeaker is measured.
[0129] When these incorrect relative maps are used for deriving a flexible rendering configuration, the flexible Tenderer will use incorrect loudspeaker locations for the rendering process. This will result in a perceived panning error for the listener. The perceived position of audio will be shifted based upon the range errors of the uncorrected LCAM process.Clock drift estimation using Peak Walk Tracking
[0130] Figure 6 shows a graph that shows a series of delay waveforms that are “walking” due to clock drift between the playback domain and the record domain. According to this example, the vertical axis of the graph 600 indicates amplitude and the horizontal axis indicates time, in units of sample numbers. In this example, Figure 6 shows a set of seven contiguous DWs 605a, 605b, 605c, 605d, 605e, 605f, 605g, which correspond to a calibrationD25032W001 device’s recordings of the same calibration signals played back by a single loudspeaker, in sequence, during a calibration process. According to this example, clock drift is causing the main peak of each of the DWs 605a-605g to walk over time. One may observe that the time interval from the main peak 610a, corresponding to DW 605a, to the main peak 610b, corresponding to DW 605b, is less than the time interval from the main peak 610a to the main peak 610c, corresponding to DW 605c. The time interval between the main peak 610a and each subsequent main peak continues to increase, with the largest time interval being that between the main peak 610a and the main peak 610g, corresponding to DW 605g. By tracking the ToA of each subsequent DW, the clock drift between the playback domain and the record domain can be estimated.
[0131] The estimated times of arrival measured at a fixed distance or “range” during multiple DW periods can be used to estimate the clock drift. If τ̂i,qis the estimated ToA of the ith loudspeaker in the qth code period then these estimated ToAs can be collected for all DWs with a sufficient amount of SNR, e.g., above 10dB. A model (e.g., a function or curve) can be fitted to those estimated ToA data points in order to estimate the clock drift. One such example involves fitting a range of [q, τ̂i,q] points collected with sufficient SNR to a linear model. The gradient of this linear model provides the clock drift estimate Δ̂fMultiple Measurements and User-Directed Sampling of the DW
[0132] By making multiple measurements, the clock drift can be estimated using peak walk tracking, e.g., as described above. The sampling of these multiple measurements may be while the user is pointing the calibration device at a loudspeaker over multiple adjacent windows or, in some implementations, when the user is not pointing the calibration device at a loudspeaker, during time intervals in which the calibration signals received by the calibration device have the appropriate SNR (e.g., SNR that is greater than or equal to an SNR threshold, such as 8dB, 10dB, 12dB, 15dB, 20dB, etc. If the calibration device is not pointing at the loudspeaker being measured, the received calibration signals may have a poor SNR due to body shadowing and other effects.
[0133] Assuming that the clock drift between the playback domain and the record domain is monotonic, then the calibration signal can be sampled with an appropriate SNR with the maximum drift. In some such examples, a user may be asked by a system (e.g., according to a visual and / or audio prompt provided by the calibration device) to remeasure the first speaker’s DW by pointing the phone at the first speaker again. This will provide the system with two delay waveforms that have a good signal to noise ratio and with the maximum drift over time.D25032W001
[0134] Furthermore, by using an accelerometer, movement sensor or camera in a calibration device or calibration system to detect calibration device movement, and only sampling the window when the calibration device is not moving significantly, a calibration process may ensure that the delay waveform is not smeared by movement of the calibration device. In some examples, the calibration device or calibration system may also implement a position model to ensure subsequent measurements are made while the calibration device is in the same location, or within a threshold offset from the same location.
[0135] Other implementations may involve inspecting the shape of the delay waveforms to ensure that they have approximately the same shape, e.g., that the direct and indirect arrival peaks are offset by approximately the same amounts, before including them in the series of delay waveforms. This ensures that in a reflective room the position of the calibration device is approximately the same when measuring calibration signals from multiple loudspeakers. If the calibration device has moved (or is moving), the relative position of the direct and reflected peaks will be different from one delay waveform to the next.
[0136] Resampling-based clock drift mitigation and clock drift estimation using received signal metrics
[0137] Figure 7 shows an example of how an effective audio sample rate difference between the playback domain and the record domain can be compensated for according to a resampling process. Figure 7 shows an example of a local replica 705 of a calibration signal in the playback domain and a corresponding recorded signal 710 in the playback domain 410. In this example the sample 401a of the local replica 705 at time 715a corresponds to the sample 401b of the recorded signal 710 at time 715b. Clock drift between the playback and record domains may result in an effective audio sample rate difference Δs between the local replica and the recorded signal. If Δs is known, then the recorded signal can be resampled to remove the effect of clock drift, as indicated by the resampling process 720 in Figure 7. In this example, the effective audio sample rate difference between the playback domain and the record domain can be compensated by a resampling process 720 prior to the DW inference process 725, which may involve applying a matched filter to a recorded DSSS calibration signal. The resampling process 720 and the DW inference process 725 may, for example, be implemented by an instance of the control system 150 of Figure IB. Figure 7 shows an example of a delay waveform 730 that is produced by the DW inference process 725.
[0138] Clock drift estimation using received signal metrics
[0139] However, Δfsis not known. One solution to this is to resample on a sparse search grid of resampling factors and choose the best option based on some signal metrics. One suchD25032W001 metric could be the kurtosis, K, of the resulting delay waveform, which may be expressed as follows:_ £[(X- / i)4]E [(X- / 1)2]2(11)
[0140] In Equation 11, E represents the expectation operator, X represents the signal sample values and p represents the mean sample value. When the playback and record domain are operating at different sampling frequencies, the effect is to smear the peak of the DW. Kurtosis acts as a measure of the “peakiness” of the DW. Therefore, maximizing the kurtosis metrics helps us estimate the correct A / s.
[0141] To find the fractional Δfs, the measured signal can be resampled according to different values of Δfson a sparse search grid. A more precise Δfscan be estimated by performing an interpolation about the maximum kurtosis, for example, a parabolic interpolation, as shown in Figure 8.
[0142] Figure 8 shows an example of a resampling search for maximum kurtosis to make an estimate of the effective audio sample rate difference Δ̂fsbetween those of the local replica and the recorded signal. In this example, an instance of the control system 150 of Figure IB is implementing instances of a resampler block 815, a matched filter block 825 and a kurtosis calculation block 830. According to this example, the resampler block 815 is configured to resample microphone signals 810 corresponding to recordings of played-back audio calibration signals according to different values of A / s: in this example, the resampler block 815 is configured to resample the microphone signals 810 to produce resampled microphone signals 820a, 820b, 820c, 820d and 820e, corresponding to Δfsvalues of +3 Hz, +2 Hz, +1 Hz, 0 Hz, and -1 Hz, respectively. In this example, the matched filter block 825 receives the resampled microphone signals 820a, 820b, 820c, 820d and 820e, and the local replica 805, and outputs modified delay waveforms 827 to the kurtosis calculation block 830.
[0143] In the graph 800, points 835a, 835b, 835c, 835d and 835e correspond to kurtosis values produced by the kurtosis calculation block 830 corresponding to the resampled microphone signals 820a, 820b, 820c, 820d and 820e, respectively. Here, the local maximum 845 of the parabolic curve that is fitted to the points 835a, 835b and 835c indicates that the Δ̂fsvalue 850 would be a good estimate. Without parabolic interpolation, the maximum kurtosis 835b corresponding to resampled microphone signal 820b indicates that the corresponding Δ̂fsvalue of +2Hz would be a good estimate.
[0144] This estimation can be performed once over the entire calibration sequence, or multiple times on segmented portions of the calibration sequence, for example, once for each playback device. If multiple estimations have been made, they can be treated as estimations ofD25032W001 variable clock drift and applied separately, or a statistical function can be applied, such as mean or median, to extract an estimate of constant clock drift Δ̂f.
[0145] The fractional sample shift of ToAimay be stated simply as δi= Δ̂f · ti, where tirepresents the time at which DWi(the delay waveform for loudspeaker i) was measured relative to the start of the calibration.Clock Drift Mitigation and Correcting Loudspeaker Range Estimates
[0146] Figure 9 shows examples of loudspeaker range estimates of an LCAM process before and after clock drift mitigation. In this example, the range estimates shown in Figure 5 are provided to a clock drift mitigation process 905. The clock drift mitigation process 905 may, for example, be implemented by an instance of the control system 150 of Figure IB. The loudspeaker range estimates resulting from the clock drift mitigation process 905 are shown in the graph 900, which shows the locations of loudspeakers 0 through 5 after an LCAM process that includes the clock drift mitigation process 905. The graph 900 is an example of what may be referred to herein as a “corrected loudspeaker map” or a “corrected device map.” When the ToA corrections have been applied to all devices i ∈ 1... Nd(or zero to 5 in this example), the range errors due to clock drift are compensated for, resulting in lower localization error and better panning performance by the flexible Tenderer.Clock drift mitigation by regenerating the local replica
[0147] The clock drift mitigation methods of this section may be referred to as “coherent” clock drift mitigation methods. One way to mitigate the clock drift on the received DWs is to use a local replica in the matched filter that has a corrected clock rate. In some such examples, the calibration signal and the replica may be generated using the digital frequency (pcof the DSSS code corresponding to the calibration signal, which may be expressed as (pc= fc / fs and the the digital frequency of the carrier signal (p0, which may be expressed as (p0= fo / fs ■> wherecrepresents the chipping rate,srepresents the sample rate of the first local replica and f0represents the carrier frequency. To generate a local replica that is synchronous with the calibration signal that is played by the playback system (that has a clock frequency of fp) then the local replica may be generated using'e = J As+ f <12)To generate the sampled the Gold Code sequence c' [n], and,< / >'o = 7A (13)Js+ fTo generate the complex carrier, yielding the synchronous local replica:rr'i[n] = Aic'i[n] cos(φ'0,in) + jAic'i[n] sin(φ'0,in) (14)D25032W001
[0148] In Equations 12 and 13, Δf represents the effective shift in the audio sampling rate (e.g., 48 kHz + Δf) that can be used to correct the digital frequency φ'0used in the cosine function. By using Ay, the control system can obtain the drift-embedded new replica that can be used directly with the recorded version without resampling it, because the replica is regenerated via Equation 14. The synchronous local replica may be used in a second application of the matched filtering process to generate a set of DWs that have clock drift mitigation.Clock drift mitigation by augmenting the time base of ToA estimates
[0149] The clock drift mitigation methods of this section may be referred to as “incoherent” clock drift mitigation methods. After the clock drift estimate, Δ̂f, has been made, this can be used to augment the estimated ToAs extracted from the set of DWs that are affected by clock drift. One way to augment the estimated ToAs, assuming that the clock drift is stationary for the duration of the calibration time QT, is to apply a simple offset that is a function of the current code period q to obtained the corrected ToAs (note in this disclosure all ToAs are expressed in samples), e.g., as follows:τ̂'i,q= τ̂i,q- (Δ̂f / fs)qT (15)
[0150] In Equation 15, τ̂i,qrepresents the unmitigated ToA estimation for the ith loudspeaker in the qth code period, τ̂'i,qrepresents the mitigated ToA estimation for the ith loudspeaker in the qth code period, and(Δ̂f / fs)qT is an offset to the unmitigated ToA estimation, where q represents the number of code repetitions since the beginning of the calibration process and T represents the length of the code in seconds.
[0151] An alternative method is to directly compensate for the range error at the end of the mapping process (the LCAM process). Some such examples involve creating a model of [q, di,q] where d is the distance to compensate, which may be calculated as follows:
[0152] In Equation 16, c represents the speed of sound. For each device i, this method involves identifying which calibration q is used for the relative distance measurement and adjusting the relative device range measurement by di,q. The second term, which is an optional extension, represents the sum of all the previous distance correction terms in the recording corresponding to the last repetition q = Q − 1 for devices j ∈ 1... i − 1.
[0153] Figure 10 is a flow diagram that outlines another example of a disclosed method. The blocks of method 1000, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shownD25032W001 and / or described. The method 1000 may be performed by an apparatus or system, such as by the control system 150 of the apparatus 100 that is shown in Figure IB and described above.
[0154] In this example, block 1005 involves estimating, by a control system, a clock drift between a playback system clock domain of a playback system and a microphone clock domain of a microphone of a hand-held device, the clock drift being a rate of change of instantaneous offset between the playback system clock domain and the microphone clock domain. In this example, the estimating is based on multiple audio calibration signals played back by each loudspeaker of a plurality of loudspeakers of the playback system and captured by the microphone over a capture time interval. According to this example, the estimating involves processing microphone signals from the microphone using reference calibration signals.
[0155] According to this example, block 1010 involves asynchronously mitigating, by the control system and after the capture time interval, the clock drift. As used herein, the term “asynchronous” refers to, for example, mitigating the clock drift of previously-recorded measurements. This type of asynchronous mitigation of clock drift differs from, for example, synchronizing the playback system clock domain and the microphone clock domain for future measurements.
[0156] According to some examples, the estimating may involve estimating times of arrival (ToAs) of audio calibration signals from each loudspeaker of the plurality of loudspeakers over multiple time periods and estimating the clock drift based on estimated ToAs. In some examples, the mitigating may involve applying a temporally variant offset to the estimated ToAs, the temporally variant offset being proportional to an estimated clock drift.
[0157] In some examples, the estimating may be based on metrics of received audio calibration signals that are used to direct a discrete search. According to some examples, the estimating may be based on kurtosis of received audio calibration signals. The kurtosis may be used as a measure of a mismatch between the microphone clock domain and the playback system clock domain.
[0158] In some examples, method 1000 may involve resampling received microphone signals at a new rate to produce a resampled microphone signal, determining if the kurtosis of an inferred transfer function signal between reference signal and the resampled microphone signal increased or decreased as compared to the kurtosis of the received microphone signals, selecting a new resampling rate and continuing a process of resampling and kurtosis change evaluation until a convergence criterion is met. The convergence criterion may, for example, be a kurtosis threshold or a kurtosis change threshold. According to some examples, method 1000 may involve resampling received microphone signals on a search grid of resamplingD25032WO01 factors, conducting a brute-force search and estimating a local maximum of kurtosis corresponding to the search grid.
[0159] According to some examples, the mitigation may be performed coherently. In some examples, performing the mitigating coherently may involve resampling the microphone signals at a corrected rate. In some examples, performing the mitigating coherently may involve resampling the reference calibration signals at a corrected rate to produce corrected reference calibration signals. The estimating may involve processing the microphone signals using first matched filters based on the reference calibration signals. The mitigating may involve producing second matched filters based on the corrected reference calibration signals and processing the microphone signals using the second matched filters.
[0160] However, in some examples the mitigation may be performed incoherently. For example, the estimating may involve estimating times of arrival (ToA) of audio calibration signals from each loudspeaker of the plurality of loudspeakers to produce ToA estimates. Performing the mitigating incoherently may involve augmenting of a time base of the ToA estimates.
[0161] According to some examples, the hand-held device may be a cellular telephone or a television remote control device. In some examples, the multiple audio calibration signals may be selected based upon measured movement of the hand-held device. The measured movement may be estimated from data quality of the microphone signals, data quality of IMU signals, or a combination thereof.
[0162] In some examples, method 1000 may involve causing, by the control system, a user interface to provide a user prompt to perform additional measurements. For example, method 1000 may involve causing, by the control system, a display to provide a visual prompt to perform additional measurements and / or causing, by the control system, one or more loudspeakers to provide an audio prompt to perform additional measurements.
[0163] Some aspects of present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer readable medium (e.g., a disc) which stores code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems can be or include a programmable general purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including an embodiment of disclosed methods or steps thereof. Such a general purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem that is programmed (and / orD25032WO01 otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.
[0164] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform required processing on audio signal(s), including performance of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory) which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system may be implemented as a general purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.
[0165] Another aspect of present disclosure is a computer readable medium (for example, a disc or other tangible storage medium) which stores code for performing (e.g., coder executable to perform) one or more examples of the disclosed methods or steps thereof.
[0166] While specific embodiments of the present disclosure and applications of the disclosure have been described herein, it will be apparent to those of ordinary skill in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the disclosure described and claimed herein. It should be understood that while certain forms of the disclosure have been shown and described, the disclosure is not to be limited to the specific embodiments described and shown or the specific methods described.
Claims
D25032WO01CLAIMSWhat is claimed is:
1. An audio processing method, comprising:a. estimating, by a control system, a clock drift between a playback system clock domain of a playback system and a microphone clock domain of a microphone of a hand-held device, the clock drift being a rate of change of instantaneous offset between the playback system clock domain and the microphone clock domain, the estimating being based on multiple audio calibration signals played back by each loudspeaker of a plurality of loudspeakers of the playback system and captured by the microphone over a capture time interval, wherein the estimating comprises processing microphone signals from the microphone using reference calibration signals; andb. asynchronously mitigating, by the control system and after the capture time interval, the clock drift.
2. The method of claim 1, wherein the estimating involves estimating times of arrival (ToAs) of audio calibration signals from each loudspeaker of the plurality of loudspeakers over multiple time periods and estimating the clock drift based on estimated ToAs.
3. The method of claim 2, where the mitigating comprises applying a temporally variant offset to the estimated ToAs, the temporally variant offset being proportional to an estimated clock drift.
4. The method of any one of claims 1-3, where the estimating is based on metrics of received audio calibration signals that are used to direct a discrete search.
5. The method of any one of claims 1-4, wherein the estimating is based on kurtosis of received audio calibration signals.
6. The method of claim 5, wherein the kurtosis is used as a measure of a mismatch between the microphone clock domain and the playback system clock domain.
7. The method of claim 5 or claim 6, further comprising resampling received microphone signals at a new rate to produce a resampled microphone signal, determining if the kurtosis of an inferred transfer function signal between reference signal and the resampled microphone signal increased or decreased as compared to the kurtosis of the received microphone signals, selecting a new resampling rate andD25032WO01 continuing a process of resampling and kurtosis change evaluation until a convergence criterion is met.
8. The method of claim 7, wherein the convergence criterion is a kurtosis threshold or a kurtosis change threshold.
9. The method of claim 5 or 6, further comprising resampling received microphone signals on a search grid of resampling factors, conducting a brute-force search and estimating a local maximum of kurtosis corresponding to the search grid.
10. The method of any one of claims 1-9, wherein the mitigation is performed coherently.
11. The method of claim 10, wherein performing the mitigating coherently comprises resampling the microphone signals at a corrected rate.
12. The method of claim 10, wherein performing the mitigating coherently comprises resampling the reference calibration signals at a corrected rate to produce corrected reference calibration signals.
13. The method of claim 12, wherein the estimating comprises processing the microphone signals using first matched filters, the first matched filters being based on the reference calibration signals, and wherein the mitigating comprises producing second matched filters based on the corrected reference calibration signals and processing the microphone signals using the second matched filters.
14. The method of claim 1, wherein the mitigation is performed incoherently.
15. The method of claim 14, wherein the estimating involves estimating times of arrival (ToA) of audio calibration signals from each loudspeaker of the plurality of loudspeakers to produce ToA estimates and wherein performing the mitigating incoherently comprises augmenting of a time base of the ToA estimates.
16. The method of any one of claims 1-15, wherein the hand-held device is a cellular telephone or a television remote control device.
17. The method of any one of claims 1-16, wherein the multiple audio calibration signals are selected based upon measured movement of the hand-held device.
18. The method of claim 17, wherein the measured movement is estimated from data quality of the microphone signals, data quality of IMU signals, or a combination thereof.
19. The method of any one of claims 1-18, further comprising causing, by the control system, a user interface to provide a user prompt to perform additional measurements.D25032WO01 20. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of claims 1-19.
21. A computer program including instructions which, when executed by a computing apparatus, cause the computing apparatus to perform the method of any of claims 1-19.
22. A computing apparatus, comprising:at least one processor;a display; andmemory storing instructions, which when executed by the at least one processor, cause the computing apparatus to perform the method of any of claims 1-19.
