Automated dialogue enhancement system

Abstract

A method for enhancing the intelligibility of dialogue in digital media content includes receiving an input audio signal that includes dialogue mixed with background audio, separating the input audio signal into dialogue and background stems, and segmenting the dialogue stem into utterance segments. The method additionally includes computing, for each utterance segment, intelligibility scores that identify segments requiring intelligibility enhancement. The method further includes applying gain adjustments to the background and / or dialogue stems during the identified segments and recombining the adjusted stems to generate an enhanced audio signal with improved intelligibility. Various other systems, methods, and computer-readable media are also disclosed.

Description

Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO AUTOMATED DIALOGUE ENHANCEMENT SYSTEMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 790,556, filed 17 April 2025, and U.S. Patent Application No. 19 / 332,521, filed 18 September 2025, the disclosures of which are incorporated, in their entirety, by these references.BACKGROUND

[0002] Cinematic productions rely on intricate audio mixes that blend dialogue, music, and sound effects to create an immersive experience. Unfortunately, the dialogue in these productions may become difficult to understand due to a combination of factors, including excessive background music and sound effects, overly dynamic mixes, low-quality audio recordings, and / or creative intent that degrades speech clarity. Playback environments (such as inadequate speakers, poor room acoustics, etc.) and hearing impairments can also exacerbate the issue. Traditional remedies, such as manual mixing adjustments, simple volume increases, or applying an across-the-board boost to the frequency ranges occupied by speech, often fail to adequately address the issue. For example, conventional approaches may amplify background noise along with speech, disrupt the original artistic balance, and / or involve significant manual effort. Such limitations hinder scalability across diverse content libraries and can compromise the creative intent of the original production. This disclosure, therefore, identifies and addresses a need for a more precise, automated, and scalable approach for enhancing dialogue intelligibility while preserving the intended mix and supporting a variety of delivery formats and user needs, including accessibility considerations.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO SUMMARY

[0003] As will be described in greater detail below, this disclosure describes an automated, context-aware system that selectively enhances dialogue intelligibility in mixed audio by analyzing and segmenting the audio, identifying problematic utterances using intelligibility measurement mechanisms and intelligibility scores, and applying targeted gain adjustments to dialogue and / or background stems. In one example, a computer-implemented method for enhancing the intelligibility of dialogue includes receiving an input audio signal that includes dialogue mixed with background audio, separating the input audio signal into a dialogue stem and a background stem, and segmenting the dialogue stem into a plurality of utterance segments that include speech. The method also includes computing an intelligibility score for each utterance segment using at least one intelligibility measurement mechanism and identifying, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement. In addition, the method includes calculating, for each identified utterance segment, a gain adjustment to apply to the background stem and / or the dialogue stem and then applying, during each identified utterance segment, the gain adjustment to the background stem and / or the dialogue stem. Finally, the method includes recombining the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

[0004] In some examples, separating the input audio signal includes applying a machine learning model that is trained to separate speech from non-speech audio in mixed content. In addition, identifying the one or more utterance segments that require intelligibilityAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO enhancement can include determining that the one or more utterance segments fail to satisfy a predefined threshold for intelligibility. In one example, the intelligibility score is computed using a user-customizable intelligibility measurement mechanism.

[0005] The gain adjustment can represent a gain attenuation applied to the background stem and / or a gain boost applied to the dialogue stem. In addition, the gain adjustment can be calculated based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment. In some cases, the gain adjustment is subject to a maximum adjustment limit to prevent over-processing.

[0006] In one example, applying the gain adjustment includes (1) generating gain adjustment metadata based on the calculated gain adjustment and (2) transmitting the gain adjustment metadata to a playback device for customized application (client-side) during audio playback.

[0007] In some examples, the method also includes generating, for each identified utterance segment, a gain adjustment envelope. This gain adjustment envelope can be based at least in part on an utterance start point that identifies when the utterance segment begins, an utterance end point that identifies when the utterance segment ends, audio events proximate the utterance start and end points, a loudness level associated with the dialogue segment at the utterance segment, and / or a loudness level associated with the background segment at the utterance segment. In one example, the method also includes configuring attack and release parameters associated with the gain adjustment envelope to minimize audio artifacts.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0008] In one example, a corresponding system includes at least one physical processor and physical memory with computer-executable instructions that, when executed by the physical processor, cause the physical processor to receive an input audio signal that includes dialogue mixed with background audio, separate the input audio signal into a dialogue stem and a background stem, and segment the dialogue stem into a plurality of utterance segments that include speech. The physical processor also computes an intelligibility score for each utterance segment based on at least one intelligibility measurement mechanism and identifies, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement. In addition, the physical processor calculates, for each identified utterance segment, a gain adjustment to apply to the background stem and / or the dialogue stem and then applies, during each identified utterance segment, the gain adjustment to the background stem and / or the dialogue stem. Finally, the physical processor recombines the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

[0009] In one example, the above-described method is encoded as computer-readable instructions on a computer-readable medium. In this example, the computer-readable medium includes one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to receive an input audio signal that includes dialogue mixed with background audio, separate the input audio signal into a dialogue stem and a background stem, and segment the dialogue stem into a plurality of utterance segments that include speech. In this example, the instructions also cause the computing device to compute an intelligibility score for each utterance segment based on at leastAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO one intelligibility measurement mechanism and identify, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement. In addition, the instructions cause the computing device to calculate, for each identified utterance segment, a gain adjustment to apply to the background stem and / or the dialogue stem and then apply, during each identified utterance segment, the gain adjustment to the background stem and / or the dialogue stem. Finally, the instructions cause the computing device to recombine the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

[0010] Features from any of the embodiments described herein may be used in combination with one another in accordance with the general principles described herein. These and other embodiments, features, and advantages will be more fully understood upon reading the following detailed description in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings illustrate a number of exemplary embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the present disclosure.

[0012] FIG. 1 is a block diagram of an example automated dialogue enhancement system configured to improve intelligibility in mixed audio content.

[0013] FIG. 2 is a flow diagram of an example method for improving dialogue intelligibility in mixed audio content.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0014] FIG. 3 illustrates an example separation of an input audio signal into a dialogue stem and a background stem.

[0015] FIG. 4 illustrates an example application of intelligibility scores and gain adjustments to a variety of utterance segments.

[0016] FIG. 5 illustrates the application of a gain adjustment to grouped utterance segments.

[0017] FIG. 6 illustrates an example process for generating a gain adjustment envelope based on audio events proximate to utterance start and end points.

[0018] FIG. 7 is a block diagram of an example content distribution ecosystem.

[0019] FIG. 8 is a block diagram of an example distribution infrastructure within the content distribution ecosystem shown in FIG. 7.

[0020] FIG. 9 is a block diagram of an example content player within the content distribution ecosystem shown in FIG. 7.

[0021] Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTSAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0022] The disclosed system introduces an automated dialogue enhancement frameworkthat advances prior approaches by leveraging advanced algorithms, machine learning models, and an improved system architecture. The system is designed to flexibly accommodate a variety of audio input configurations, including mixed audio signals containing both dialogue and background elements, as well as pre-separated stems. As detailed below, the system employs a multi-step process that includes source separation to generate dialogue and background stems, intelligibility analysis using one or more intelligibility measurement mechanisms and intelligibility scores, and dynamic audio recombining. Source separation can utilize various techniques (including machine learning models or metadata-driven extraction) to isolate speech from background music and effects, even in scenarios where pre-separated stems or metadata are unavailable or where noise is embedded within the dialogue.

[0023] Intelligibility analysis measures the clarity of dialogue in relation to competing background sounds, enabling precise identification of moments requiring enhancement. The system supports a range of intelligibility measurement mechanisms and can apply fixed thresholds or statistical methods (such as z-score-based outlier detection) to determine which utterances require enhancement. For utterances identified with poor intelligibility, the system calculates the appropriate gain adjustment (such as attenuation of the background, boosting of the dialogue, or both) based on factors including the relative loudness of stems and the severity of the intelligibility deficit. Configurable limits can also be imposed to prevent over-processing and maintain a natural sound. In addition, adjustments can contain signal-dependent attack and release envelopes to avoid abrupt changes and audio artifacts. The enhanced stems are thenAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO recombined, either server-side or in real time by the playback device, to produce an audio stream with improved intelligibility.

[0024] The system can generate both subtly enhanced mixes for general audiences and more aggressive enhancements for users with specific needs, such as those with hearing impairments or in challenging listening environments. By automating dialogue enhancement through flexible source separation, modular intelligibility analysis, and adaptive recombining, the disclosed system addresses issues of dialogue clarity in cinematic audio. The solution is scalable, content- and language-agnostic, and compatible with a wide range of distribution and playback environments, providing a significant improvement over manual or volume-based approaches.

[0025] The following will provide, with reference to FIGS. 1-6, detailed descriptions of exemplary system architectures, processing flows, and methods for automated dialogue enhancement, including block diagrams of the enhancement system, flowcharts illustrating the audio processing pipeline, visualizations of source separation and intelligibility scoring, and example waveforms demonstrating the application of targeted gain adjustments to improve dialogue clarity in mixed audio content. FIGS. 7-9 illustrate exemplary embodiments of a content distribution ecosystem, including a block diagram of the overall distribution infrastructure (FIG.7), a detailed view of the components and services within the distribution infrastructure (FIG. 8), and a block diagram of a content player configured to receive, decode, and play back enhanced audio content (FIG. 9).

[0026] FIG. 1 is a block diagram of an example dialogue enhancement system 100 configured to improve intelligibility in mixed audio content. As shown in FIG. 1, system 100Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO includes a source separation module 110 configured to separate an input audio signal 102 into discrete components (e.g., background stem 104 and dialogue stem 106) and an automated dialogue enhancement system 150 configured to analyze these discrete components and generate an enhanced audio signal 114 with improved dialogue intelligibility. In this example, dialogue enhancement system 150 includes an intelligibility module 120 configured to (1) divide dialogue stem 106 into multiple utterance segments 108 and (2) compute intelligibility scores 112 for each utterance segment. Dialogue enhancement system 150 also includes a gain analysis module 130 and an enhancement module 140 configured to calculate and apply targeted gain adjustments to the dialogue and / or background stems to improve dialogue intelligibility.

[0027] Modules 110, 120, 130, and 140 generally represent any type or form of component, device, and / or set of computer-executable instructions. Although illustrated as separate elements, one or more of the modules illustrated in FIG. 1 can represent portions of a single module or application. In certain embodiments, these modules represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, and as will be described in greater detail below, these modules can be stored and configured to run on one or more computing devices, such as the devices illustrated in FIGS. 7-9. One or more of these modules can also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.

[0028] FIG. 2 is a flow diagram of an exemplary computer-implemented method 200 for improving dialogue intelligibility in mixed audio content. The steps shown in FIG. 2 can beAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO performed by any suitable computer-executable code and / or computing system, including the systems illustrated in FIGS. 1 and 7-9. In one example, each of the steps shown in FIG. 2 represent an algorithm whose structure includes and / or is represented by multiple sub-steps, examples of which will be provided in greater detail below.

[0029] As illustrated in FIG. 2, at step 210 the system described herein receives a mixed audio input signal that includes dialogue mixed with background audio. For example, in the embodiment illustrated in FIG. 1, source separation module 110 in FIG. 1 receives input audio signal 102, which represents a mixed audio signal that includes both dialogue and background audio. The phrase "mixed audio signal," as used herein, generally refers to an audio signal, file, or package that contains a combination of dialogue (speech) and background audio elements, such as music, sound effects, and / or ambient noise, combined into one or more tracks.

[0030] In one example, input audio signal 102 includes embedded metadata (such as metadata provided with ADM BWAV or IAB files) that explicitly identifies tracks that contain dialogue, music, and / or effects. In another example, input audio signal 102 may lack such metadata and may instead represent a composite audio track or file where dialogue, music, and effects are intermixed without explicit identification, requiring the use of advanced source separation algorithms to isolate the individual audio components, as described in greater detail below.

[0031] The systems described herein can perform step 210 in several ways depending on the stage of the content lifecycle and the available resources. In one example, step 210 can be executed as part of a content creation and / or post-production engine, where metadataAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO and / or source separation tools are used to generate dialogue, music, and effects stems before distribution. Alternatively, step 210 can be executed by a content distribution or streaming engine (e.g., distribution infrastructure 710 in FIG. 7) afterthe content has been created, applying source separation algorithms to mixed audio files to extract the necessary stems for downstream processing, enhancement, or adaptive playback.

[0032] At step 220, the system next separates the input audio signal into a dialogue stem and a background stem. For example, in the embodiment illustrated in FIG. 1, source separation module 110 separates input audio signal 102 into at least background stem 104 and dialogue stem 106.

[0033] The term "stem," as used herein, generally refers to an isolated audio track representing a specific component of a mixed audio signal, such as dialogue, music, sound effects, etc. In addition, the term "dialogue stem" generally refers to an isolated audio track that only contains the primary spoken language or main speech elements. Finally, the term "background stem" generally refers to an isolated audio track that contains non-speech elements, such as music, sound effects, and / or ambient noise.

[0034] The systems described herein can perform step 220 in a variety of ways. In one example, source separation module 110 is configured to process metadata embedded in an audio format containing multiple channels and sound objects, such as ADM BWAV or IAB, to identify and extract individual audio stems. For example, when processing a movie soundtrack delivered in a multi-channel format, source separation module 110 can utilize embedded metadata to automatically extract separate dialogue and music / effects stems from one or more audio tracks,Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO enabling precise downstream enhancement of the dialogue without affecting the other audio elements.

[0035] In another example, source separation module 110 represents a machine learning model trained to separate speech (e.g., dialogue) from non-speech audio (e.g., music, effects, background noise, etc.) directly from mixed audio content. For example, and as shown in FIG. 3, when processing a television episode or film soundtrack that is only available as a single mixed audio track 302 without separate stems or embedded metadata, source separation module 110 can apply a machine learning model to analyze the composite audio and generate discrete, isolated audio tracks that contain primarily dialogue (e.g., dialogue stem 304) and background (e.g., background stem 306) audio. Examples of such machine learning models include, without limitation, CASS, Demucs and mdx_net.

[0036] In some cases, source separation module 110 creates a single background stem that contains all non-speech elements (e.g., music, sound effects, etc.) associated with input audio signal 102. In other cases, source separation module 110 can create a background stem for each category of non-speech (e.g., a first background stem for music, a second background stem for effects, a third background stem for ambient noise, etc.).

[0037] Returning to FIG. 2, at step 230 the system next segments the dialogue stem into a plurality of utterance segments. For example, in the embodiment illustrated in FIG. 1, intelligibility module 120 in FIG. 1 separates dialogue stem 106 into multiple utterance segmentsAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0038] The phrase "utterance segment," as used herein, generally refers to individual units of spoken language within an audio signal. An utterance segment can represent a single word, phrase, exclamation, or other linguistically or perceptually meaningful unit. As will be described in greater detail below in connection with FIG. 5, an utterance segment can also represent a continuous stretch of speech, such as a sequence of spoken words that are only separated by pauses to breathe. The duration of an utterance segment can vary, but in some embodiments can range from approximately 200 milliseconds to several seconds, depending on the natural breaks in speech and the granularity of the segmentation algorithm.

[0039] The systems described herein can perform step 230 in a variety of ways. In one example, intelligibility module 120 can include a speech activity detection algorithm (such as Netflix's Speech Music Activity Detection algorithm) configured to analyze a dialogue stem to detect the presence and boundaries of speech. For example, and as shown in FIG.4, intelligibility module 120 can segment a dialogue stem 440 into a plurality of utterance segments 402, 404, 406, 408, 410, 412, 414, 416, and 418. In this example, intelligibility module 120 operates by analyzing short windows of audio (e.g., around 200 milliseconds each) and extracting a variety of acoustic features from each segment, such as spectral content, energy, and temporal dynamics. Using statistical models and / or machine learning classifiers, intelligibility module 120 then classifies each window as containing speech, music / effects, or another type of audio content. By aggregating these classifications across the entire dialogue stem, intelligibility module 120 can precisely determine the start and end points of spoken language, effectively segmenting the dialogue stem into discrete utterance segments that include only spoken language.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0040] Returning to FIG. 2, at step 240 the system then computes, for each utterance segment created in step 230, an intelligibility score that is based on at least one intelligibility measurement mechanism. For example, in the embodiment illustrated in FIG. 1, intelligibility module 120 creates an intelligibility score 112 for each of utterance segments 108.

[0041] As used herein, the term "intelligibility score" refers to a quantitative metric computed for a segment of audio (e.g., an utterance segment) that reflects the degree to which the speech in that segment is understandable. In addition, the term "intelligibility measurement mechanism" refers to any quantitative or qualitative measure, algorithm, or assessment technique used to evaluate the degree to which speech or dialogue within an audio signal is understandable to a listener, including but not limited to mechanisms based on signal analysis, machine learning models, user feedback, or accessibility-focused criteria.

[0042] Intelligibility module 120 can use a variety of different intelligibility measurement mechanisms and scoring systems to compute intelligibility scores. For example, intelligibility module 120 can utilize STOI (Short-Time Objective Intelligibility), which measures the correlation between short-time segments of clean and processed speech to generate an intelligibility score. In more complex or real-world scenarios, intelligibility module 120 may use e-STOI (extended STOI), an improved version of STOI that incorporates longer time segmentsand additional normalization steps for greater robustness to noise and distortion. Alternatively, intelligibility module 120 may use mosa_STOI, which leverages modern speech representations, such as embeddings from self-supervised models, to provide flexible and potentially nonintrusive intelligibility assessment. For binaural or spatial audio applications, intelligibility moduleAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO 120 may employ BiDWGP (Binaural Distortion-based Weighted Glimpse Proportion), a parametric model designed to predict speech intelligibility in binaural listening environments. In some examples, intelligibility module 120 computes intelligibility scores on a scale from 0 (pure noise) to 1 (dialogue only), with higher scores representing greater clarity and understandability of the dialogue in the presence of background audio.

[0043] As detailed above, intelligibility scores can be computed by a variety of different intelligibility measurement mechanisms. In certain embodiments, intelligibility module 120 computes the intelligibility score using a user-customizable intelligibility measurement mechanism, allowing the system to adapt to different user needs and preferences. For example, system 100 may provide, and an end user with mild hearing loss may select, an intelligibility measurement mechanism specifically designed to account for reduced speech perception, prompting the system to apply a lower intelligibility threshold and more aggressive enhancement parameters to ensure clearer dialogue. Conversely, system 100 may provide, and an end user with normal hearing may select, a standard intelligibility measurement mechanism with a higher threshold, resulting in more subtle adjustmentsthat preserve the original audio mix. The modular design of system 100 enables the integration and swapping of alternative intelligibility measurement mechanisms, such as those specifically tailored for end users with hearing loss or other accessibility requirements. This customization can be performed by a system operator (of, e.g., distribution infrastructure 710 in FIG. 7), the end user of a playback device (e.g., content player 720 in FIG. 7), or both, enabling the creation of personalized audio mixes with thresholds and enhancement parameters adjusted to optimize dialogue clarity for individual listeners. ByAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO providing and supporting a range of intelligibility measurement mechanisms, the disclosed system can be tailored to different content types, user needs, and playback scenarios, ensuring optimal dialogue clarity across diverse audio environments.

[0044] Returning to FIG. 2, at step 250 the system next identifies, based on an analysis of the intelligibility scores computed in step 240, one or more utterance segments that require intelligibility enhancement. For example, in the embodiment illustrated in FIG. 1, intelligibility module 120 in FIG. 1 can determine that one or more of utterance segments 108 require intelligibility enhancement.

[0045] Intelligibility module 120 can identify utterance segments requiring intelligibility enhancement in a variety of ways, including by applying fixed intelligibility thresholds or statistical outlier detection techniques. For example, intelligibility module 120 may designate a segment for enhancement if its computed intelligibility score fails to satisfy a predefined threshold for intelligibility, such as 0.7, 0.75, 0.8, 0.85, etc. on a scale of 0 (pure noise) to 1 (dialogue only). For example, in the embodiment shown in FIG. 4, intelligibility module 120 may determine that utterance segments 404, 406, 408, 410, 412, 414, 416, and 418 each fail to satisfy a predefined threshold for intelligibility, such as .80, and thus require intelligibility enhancement. In contrast, intelligibility module 120 may determine that utterance segment 402 satisfies the predefined threshold for intelligibility and does not require intelligibility enhancement.

[0046] Alternatively, intelligibility module 120 can employ statistical methods, such as z-score-based outlier detection, to dynamically identify segments with unusually lowAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO intelligibility relative to the overall distribution within a given audio program. In this approach, intelligibility module 120 first computes intelligibility scores for all utterance segments and then calculates the mean and standard deviation of these scores. Intelligibility module 120 then flags segments whose intelligibility scores fall below a certain number of standard deviations from the mean (e.g., two standard deviations) as outliers and selects the same for enhancement. This approach allows the system to adaptively identify problematic segments based on the unique characteristics of each audio program, rather than relying solely on a fixed threshold.

[0047] In either of the above-described approaches, the disclosed system only performs audio processing on utterance segments that require intelligibility enhancement, preserving the natural character and creative intent of the original mix in all other regions. This reduces the amount of processing required and avoids unnecessary alterations, reducing the risk of introducing artifacts or disrupting the listener's experience.

[0048] Returning to FIG. 2, at step 260 the system next calculates, for each utterance segment identified in step 250, a gain adjustment to apply to the background stem and / or the dialogue stem. For example, in the embodiment illustrated in FIG. 4, gain analysis module 130 may calculate gain adjustments 422, 424, 426, 428, 430, 432, and 434 to apply to the background stem 450 at utterance segments 404, 406, 408, 412, 414, 416, and 418, respectively, since these utterance segments have intelligibility scores that fail to satisfy a predefined threshold (e.g., .80).

[0049] The term "gain adjustment," as used herein, generally refers to a modification of the amplitude or volume level of an audio signal. Examples of gain adjustments include, without limitation, a gain attenuation (such as a gain attenuation applied to one or more portionsAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO of one or more background stems) and / or a gain boost (such as a gain boost applied to one or more portions of a dialogue stem). Gain adjustments can also include applying both a gain boost (e.g., to one or more portions of a dialogue stem) and a gain attenuation (e.g., to one or more portions of a background stem). In some embodiments, a gain adjustment can include selectively attenuating or boosting one or more frequency bands within the background stem and / or the dialogue stem, allowing the system to target those spectral regions most important for speech intelligibility. As explained in greater detail below, these gain adjustments enable the system to selectively enhance the clarity and intelligibility of dialogue without broadly altering the overall audio mix, thereby preserving the creative intent and listening experience.

[0050] The systems described herein can calculate gain adjustments in a variety of ways. In one example, gain analysis module 130 calculates a gain adjustment to apply based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment. Based on this difference, gain analysis module 130 determines the amount of gain adjustment necessary to achieve a preferred intelligibility level, such as by attenuating the background stem by a determined amount, boosting the dialogue stem by a determined amount, or both. For example, if the loudness of the dialogue stem at a particular utterance segment is measured to be 4 Loudness Units (LU) lower than the background stem, gain analysis module 130 may determine that a 5 LU attenuation should be applied to the background stem during that segment to achieve the preferred intelligibility level. Alternatively, if the background is already at an appropriate level but the dialogue is unusually quiet, the module may instead apply a 3 LU boost to the dialogue stem. In some cases, gain analysis moduleAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO 130 may apply both a moderate attenuation to the background stem and a slight boost to the dialogue stem at the identified utterance segment, ensuring that the overall adjustment brings the dialogue to the desired prominence without introducing audio artifacts or disrupting the original mix. The term "Loudness Unit," as used herein, refers to a standardized measurement of perceived audio loudness, typically used in broadcast and audio engineering to quantify changes in loudness level. One loudness unit corresponds to a one-decibel (1 dB) change in integrated loudness. Loudness units can be expressed as LUFS (Loudness Units relative to Full Scale) or LKFS (Loudness K-weighted relative to Full Scale), both of which are calculated using algorithms such as ITU-R BS.1770 and are widely used to ensure consistent and perceptually balanced audio levels across different content and playback environments.

[0051] Gain analysis module 130 can also scale the gain adjustment according to the severity of the intelligibility deficit. For example, if gain analysis module 130 determines that an utterance segment (such as utterance segment 412 in FIG. 4) has a significantly lower intelligibility score with dialogue heavily masked by background noise, gain analysis module 130 may apply a larger attenuation (e.g., gain adjustment 426) to the background stem at this segment, such as by reducing its volume by 8 LU. Conversely, if the intelligibility deficit is mild (as is the case, e.g., with utterance segment 410 in FIG. 4), gain analysis module 130 may apply only a slight attenuation to the background stem (such as a 2 LU reduction) at this segment, ensuring that the adjustment is proportional to the actual need to avoid unnecessary alteration of the audio mix.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0052] In some examples, the above-described gain adjustment is subject to configurable limits, such as a maximum adjustment limit, to prevent over-processing and maintain a natural sound. For example, if a background stem at a particular utterance segment is already at a low loudness level, further attenuation could result in an unnatural or unbalanced mix, potentially collapsing the stereo image or making the background audio nearly inaudible. To address this, the system may set and impose a configurable maximum attenuation limit (such as 8 LU) so that even if the calculated gain adjustment suggests a 12 LU reduction, only an 8 LU attenuation is actually applied. This ensures that dialogue enhancement does not compromise the integrity or naturalness of the overall audio experience.

[0053] In one example, to avoid unnatural breaks and abrupt changes in audio levels, the disclosed systems intelligently groups utterance segments based on the natural flow of human speech. Specifically, gain analysis module 130 may analyze the timing and continuity between adjacent utterance segments and automatically join utterance segments that are only separated by brief pauses (such as breaks of less than 500 milliseconds, which often correspond to natural stops for breath or short pauses in conversation). For example, in the embodiment illustrated in FIG. 5, gain analysis module 130 may determine that, because utterance segments 502 and 504 are only separated by a brief pause (likely caused by the speaker stopping to take a breath), these utterance segments should be treated as a singular utterance from a gain adjustment perspective, and may in turn create a single gain adjustment 510 to apply to dialogue stem 530 and / or background stem 540 during both utterance segments. By treating closely spaced segments as a single, continuous unit, the system can apply gain adjustments smoothlyAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO across an entire grouping, rather than introducing separate attenuations for each individual segment. This approach prevents choppy or distracting audio transitions and ensures that dialogue enhancement is both effective and unobtrusive to the listener.

[0054] In some embodiments, the disclosed system may generate a gain adjustment "envelope" to apply to the background and / or dialogue stems. Again adjustment envelope refers to a time-varying profile that defines how the gain adjustment applied to an audio signal changes over the duration of a specific segment or utterance. In other words, instead of applying an abrupt, instantaneous change in gain, gain analysis module 130 can generate a gain adjustment envelope that ramps the desired gain adjustment up or down gradually to avoid introducing audio artifacts or unnatural transitions. For example, in the embodiment illustrated in FIG. 4, gain analysis module 130 can generate a gain adjustment 430 for utterance segment 414 that applies an increasing amount of gain attenuation to background stem 450 over a few seconds, as opposed to instantaneously.

[0055] The shape of a gain adjustment envelope can be tailored based on a variety of factors, including the start and end points of an utterance, the presence of nearby audio events, and / or the relative loudness of the dialogue and background stems, among potentially other factors. For example, as illustrated in FIG. 6, if an utterance segment 602 begins at point 604 immediately after a loud background event (e.g., preceding audio event 608), gain analysis module 130 may create a gain adjustment envelope 620 that includes a gradual attack (starting, e.g., at adjusted start point 614), allowing the attenuation to ramp down smoothly so that the change is masked by the preceding audio event 608. Similarly, if utterance segment 602 ends atAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO point 606 just before another significant background sound (e.g., subsequent audio event 612), gain analysis module 130 can gradually extend the release portion of gain adjustment envelope 620 (to, e.g., adjusted end point 616) so that the gain returns to its original level in a way that is masked by subsequent audio event 612. This approach generates context-aware, psychoacoustically masked gain envelopes that are carefully shaped to align with surrounding audio events, ensuring that attenuation is applied in a manner that is unobtrusive and not audible to the listener, thereby preventing abrupt or unnatural changes in background sound.

[0056] In some embodiments, the system imposes a maximum adjustment threshold that limits how far the start or end points of a gain adjustment envelope can be shifted based on proximate audio events. For example, if a significant background event occurs more than a predetermined duration (such as five seconds) away from the utterance segment, the system will not extend the gain adjustment envelope beyond this threshold. This ensures that the timing of gain adjustments remains practical and does not introduce excessive delays or unnatural transitions, while still allowing for context-aware smoothing when relevant audio events occur within a reasonable proximity to the dialogue segment.

[0057] Returning to FIG. 2, at step 270 the system next applies, during each identified utterance segment, the calculated gain adjustment to the background stem and / or the dialogue stem. For example, in the embodiment illustrated in FIG. 1, enhancement module 140 can apply the gain adjustment calculated in step 260 to background stem 104 and / or dialogue stem 106.

[0058] The systems disclosed herein can perform step 270 in a variety of ways. In some embodiments, enhancement module 140 applies the gain adjustment during the encodingAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO or post-production process, resulting in a pre-mixed enhanced audio stream that is distributed to the playback devices of end users. For example, as illustrated in FIGS. 7-9, enhancement module 140 may apply the gain adjustment during the encoding or post-production stage within distribution infrastructure 710, so that the resulting pre-mixed enhanced audio stream is transmitted to content player 720 for playback by end users.

[0059] Alternatively, the system may generate and transmit (e.g., as part of distribution infrastructure 710 in FIGS. 7-9) gain adjustment metadata to a playback device (e.g., content player 720), enabling the device to apply the calculated adjustments locally and customize the enhancement according to specific listening conditions and preferences. For example, a mobile device or streaming scenario with limited speaker quality or higher ambient noise may require more aggressive attenuation to problematic utterance segments within the background stem to maximize dialogue clarity, while a high-fidelity home theater setup may require only subtle adjustments to preserve the richness and balance of the original mix. Similarly, an end user with hearing loss may configure their playback device to apply more pronounced gain adjustments or select an intelligibility measurement mechanism tailored for reduced speech perception, ensuring that dialogue remains clear and accessible even in challenging listening environments. This adaptive approach allows the system to optimize the listening experience for each user's specific context by dynamically adjusting the degree and manner of dialogue enhancement based on factors such as device capabilities, ambient noise levels, user preferences, and accessibility needs.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0060] Returning to FIG. 2, at step 280 the system then recombines the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility. For example, in the embodiment illustrated in FIG. 1, enhancement module 140 recombines background stem 104 with dialogue stem 106 to produce an enhanced audio signal 114.

[0061] The systems described herein can perform step 280 in a variety of ways. In some embodiments, the enhanced dialogue and background stems are recombined server-side to produce a new audio stream that is distributed directly to end users. In other embodiments, the system can transmit gain adjustment metadata along with the original stems, allowing the playback device to perform the recombining locally and adapt the enhancement in real time based on user preferences or playback conditions. This flexibility ensures that the dialogue enhancement process can be seamlessly integrated into different distribution and playback workflows, providing improved intelligibility and audio quality across a wide range of devices and listening environments.

[0062] As detailed above, the systems and methods described herein provide a sophisticated and targeted approach to improving dialogue intelligibility in audio content such as movies and television programs. Rather than relying on simple volume increases or broad, indiscriminate adjustments that can compromise the original audio mix, the disclosed system utilizes advanced algorithms to analyze audio, separate speech from background elements, and identify specific moments where dialogue clarity is insufficient. Targeted gain adjustments are then applied (by attenuating the background and / or boosting the dialogue) only during thoseAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO segments where enhancement is required, thereby preserving the overall balance and creative intent of the production.

[0063] This approach offers several important advantages. For example, by focusing processing efforts solely on segments with low intelligibility, the system minimizes unnecessary alterations (which in turn reduces CPU and memory usage) and reduces the risk of introducing audio artifacts or disrupting the listener's experience. The solution is also highly adaptable, functioning effectively across diverse content types, distribution workflows, and playback environments, and can be further customized to accommodate users with hearing loss or other accessibility needs. Whether implemented during production, by a streaming service, or at the playback device, the result is a more intelligible and accessible dialogue experience that remains faithful to the original artistic vision.

[0064] The following will provide, with reference to FIGS. 7-9, an exemplary ecosystem in which the above-described approach can be deployed. Specifically, FIG. 7 illustrates an ecosystem in which content is provisioned to end nodes and in which requests for content are steered to specific end nodes. The discussion corresponding to FIGS. 8 and 9 presents an overview of an exemplary distribution infrastructure and an exemplary content player used during playback sessions, respectively. These exemplary ecosystems and distribution infrastructures can be implemented in any of the embodiments described above, including FIGS.1-6.

[0065] FIG. 7 is a block diagram of a content distribution ecosystem 700 that includes a distribution infrastructure 710 in communication with a content player 720. In someAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO embodiments, distribution infrastructure 710 is configured to encode data at a specific data rate and to transfer the encoded data to content player 720. Content player 720 is configured to receive the encoded data via distribution infrastructure 710 and to decode the data for playback to a user. The data provided by distribution infrastructure 710 includes, for example, audio, video, text, images, animations, interactive content, haptic data, virtual or augmented reality data, location data, gaming data, or any other type of data that is provided via streaming.

[0066] Distribution infrastructure 710 generally represents any services, hardware, software, or other infrastructure components configured to deliver content to end users. For example, distribution infrastructure 710 includes content aggregation systems, media transcoding and packaging services, network components, and / or a variety of other types of hardware and software. In some cases, distribution infrastructure 710 is implemented as a highly complex distribution system, a single media server or device, or anything in between. In some examples, regardless of size or complexity, distribution infrastructure 710 includes at least one physical processor 712 and at least one memory 714. One or more modules 716 are stored or loaded into memory 714 to enable adaptive streaming, as discussed herein.

[0067] Content player 720 generally represents any type or form of device or system capable of playing audio and / or video content that has been provided over distribution infrastructure 710. Examples of content player 720 include, without limitation, mobile phones, tablets, laptop computers, desktop computers, televisions, set-top boxes, digital media players, virtual reality headsets, augmented reality glasses, and / or any other type or form of device capable of rendering digital content. As with distribution infrastructure 710, content player 720Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO includes a physical processor 722, memory 724, and one or more modules 726. Some or all of the adaptive streaming processes described herein is performed or enabled by modules 726, and in some examples, modules 716 of distribution infrastructure 710 coordinate with modules 726 of content player 720 to provide adaptive streaming of digital content.

[0068] In certain embodiments, one or more of modules 716 and / or 726 in FIG. 7 represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, and as will be described in greater detail below, one or more of modules 716 and 726 represent modules stored and configured to run on one or more general-purpose computing devices. One or more of modules 716 and 726 in FIG. 7 also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.

[0069] In addition, one or more of the modules, processes, algorithms, or steps described herein transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules recited herein receive audio data to be encoded, transform the audio data by encoding it, output a result of the encoding for use in an adaptive audio bit-rate system, transmit the result of the transformation to a content player, and render the transformed data to an end user for consumption. Additionally or alternatively, one or more of the modules recited herein transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0070] Physical processors 712 and 722 generally represent any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, physical processors 712 and 722 access and / or modify one or more of modules 716 and 726, respectively. Additionally or alternatively, physical processors 712 and 722 execute one or more of modules 716 and 726 to facilitate adaptive streaming of digital content. Examples of physical processors 712 and 722 include, without limitation, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) that implement softcore processors, application-specific integrated circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, and / or any other suitable physical processor.

[0071] Memory 714 and 724 generally represent any type or form of volatile or nonvolatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, memory 714 and / or 724 stores, loads, and / or maintains one or more of modules 716 and 726. Examples of memory 714 and / or 724 include, without limitation, random access memory (RAM), read only memory (ROM), flash memory, hard disk drives (HDDs), solid-state drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, and / or any other suitable memory device or system.

[0072] FIG. 8 is a block diagram of exemplary components of content distribution infrastructure 710 according to certain embodiments. Distribution infrastructure 710 includes storage 810, services 820, and a network 830. Storage 810 generally represents any device, set of devices, and / or systems capable of storing content for delivery to end users. Storage 810Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO includes a central repository with devices capable of storing terabytes or petabytes of data and / or includes distributed storage systems (e.g., appliances that mirror or cache content at Internet interconnect locations to provide faster access to the mirrored content within certain regions). Storage 810 is also configured in any other suitable manner.

[0073] As shown, storage 810 may store a variety of different items including content 812, user data 814, and / or log data 816. Content 812 includes television shows, movies, video games, user-generated content, and / or any other suitable type or form of content. User data 814 includes personally identifiable information (Pll), payment information, preference settings, language and accessibility settings, and / or any other information associated with a particular user or content player. Log data 816 includes viewing history information, network throughput information, and / or any other metrics associated with a user's connection to or interactions with distribution infrastructure 710.

[0074] Services 820 includes personalization services 822, transcoding services 824, and / or packaging services 826. Personalization services 822 personalize recommendations, content streams, and / or other aspectsofa user's experience with distribution infrastructure 710. Transcoding services 824 compress media at different bitrates which, as described in greater detail below, enable real-time switching between different encodings. Packaging services 826 package encoded video before deploying it to a delivery network, such as network 830, for streaming.

[0075] Network 830 generally represents any medium or architecture capable of facilitating communication or data transfer. Network 830 facilitates communication or dataAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO transfer using wireless and / or wired connections. Examples of network 830 include, without limitation, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), the Internet, power line communications (PLC), a cellular network (e.g., a global system for mobile communications (GSM) network), portions of one or more of the same, variations or combinations of one or more of the same, and / or any other suitable network. For example, as shown in FIG. 8, network 830 includes an Internet backbone 832, an internet service provider 834, and / or a local network 836. As discussed in greater detail below, bandwidth limitations and bottlenecks within one or more of these network segments triggers video and / or audio bit rate adjustments.

[0076] FIG. 9 is a block diagram of an exemplary implementation of content player 720 of FIG. 7. Content player 720 generally represents any type or form of computing device capable of reading computer-executable instructions. Content player 720 includes, without limitation, laptops, tablets, desktops, servers, cellular phones, multimedia players, embedded systems, wearable devices (e.g., smart watches, smart glasses, etc.), smart vehicles, gaming consoles, internet-of-things (loT) devices such as smart appliances, variations or combinations of one or more of the same, and / or any other suitable computing device.

[0077] As shown in FIG. 9, in addition to processor 722 and memory 724, content player 720 includes a communication infrastructure 902 and a communication interface 922 coupled to a network connection 924. Content player 720 also includes a graphics interface 926 coupled to a graphics device 928, an input interface 934 coupled to an input device 936, and a storage interface 938 coupled to a storage device 940.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0078] Communication infrastructure 902 generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of communication infrastructure 902 include, without limitation, any type orform of communication bus (e.g., a peripheral component interconnect (PCI) bus, PCI Express (PCIe) bus, a memory bus, a frontside bus, an integrated drive electronics (IDE) bus, a control or register bus, a host bus, etc.).

[0079] As noted, memory 724 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and / or other computer-readable instructions. In some examples, memory 724 stores and / or loads an operating system 908 for execution by processor 722. In one example, operating system 908 includes and / or represents software that manages computer hardware and software resources and / or provides common services to computer programs and / or applications on content player 720.

[0080] Operating system 908 performs various system management functions, such as managing hardware components (e.g., graphics interface 926, audio interface 930, input interface 934, and / or storage interface 938). Operating system 908 also provides process and memory management models for playback application 910. The modules of playback application 910 includes, for example, a content buffer 912, an audio decoder 918, and a video decoder 920.

[0081] Playback application 910 is configured to retrieve digital content via communication interface 922 and play the digital content through graphics interface 926. Graphics interface 926 is configured to transmit a rendered video signal to graphics device 928. In normal operation, playback application 910 receives a request from a user to play a specificAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO title or specific content. Playback application 910 then identifies one or more encoded video and audio streams associated with the requested title. After playback application 910 has located the encoded streams associated with the requested title, playback application 910 downloads sequence header indices associated with each encoded stream associated with the requested title from distribution infrastructure 710. A sequence header index associated with encoded content includes information related to the encoded sequence of data included in the encoded content.

[0082] In one embodiment, playbackapplication 910 begins downloadingthe content associated with the requested title by downloading sequence data encoded to the lowest audio and / or video playback bitrates to minimize startup time for playback. The requested digital content file is then downloaded into content buffer 912, which is configured to serve as a first-in, first-out queue. In one embodiment, each unit of downloaded data includes a unit of video data or a unit of audio data. As units of video data associated with the requested digital content file are downloaded to the content player720, the units of video data are pushed into the content buffer 912. Similarly, as units of audio data associated with the requested digital content file are downloaded to the content player 720, the units of audio data are pushed into the content buffer 912. In one embodiment, the units of video data are stored in video buffer 916 within content buffer 912 and the units of audio data are stored in audio buffer 914 of content buffer 912.

[0083] A video decoder 920 reads units of video data from video buffer 916 and outputs the units of video data in a sequence of video frames corresponding in duration to the fixed span of playback time. Reading a unit of video data from video buffer 916 effectively deAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO queues the unit of video data from video buffer 916. The sequence of video frames is then rendered by graphics interface 926 and transmitted to graphics device 928 to be displayed to a user.

[0084] An audio decoder 918 reads units of audio data from audio buffer 914 and outputs the units of audio data as a sequence of audio samples, generally synchronized in time with a sequence of decoded video frames. In one embodiment, the sequence of audio samples is transmitted to audio interface 930, which converts the sequence of audio samples into an electrical audio signal. The electrical audio signal is then transmitted to a speaker of audio device 932, which, in response, generates an acoustic output.

[0085] In situations where the bandwidth of distribution infrastructure 710 is limited and / or variable, playback application 910 downloads and buffers consecutive portions of video data and / or audio data from video encodings with different bit rates based on a variety of factors (e.g., scene complexity, audio complexity, network bandwidth, device capabilities, etc.). In some embodiments, video playback quality is prioritized over audio playback quality. Audio playback and video playback quality are also balanced with each other, and in some embodiments audio playback quality is prioritized over video playback quality.

[0086] Graphics interface 926 is configured to generate frames of video data and transmit the frames of video data to graphics device 928. In one embodiment, graphics interface 926 is included as part of an integrated circuit, along with processor 722. Alternatively, graphics interface 926 is configured as a hardware accelerator that is distinct from (i.e., is not integrated within) a chipset that includes processor 722.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0087] Graphics interface 926 generally represents any type or form of device configured to forward images for display on graphics device 928. For example, graphics device 928 is fabricated using liquid crystal display (LCD) technology, cathode-ray technology, and lightemitting diode (LED) display technology (either organic or inorganic). In some embodiments, graphics device 928 also includes a virtual reality display and / or an augmented reality display. Graphics device 928 includes any technically feasible means for generating an image for display. In other words, graphics device 928 generally represents any type or form of device capable of visually displaying information forwarded by graphics interface 926.

[0088] As illustrated in FIG. 9, content player 720 also includes at least one input device 936 coupled to communication infrastructure 902 via input interface 934. Input device 936 generally represents any type or form of computing device capable of providing input, either computer or human generated, to content player 720. Examples of input device 936 include, without limitation, a keyboard, a pointing device, a speech recognition device, a touch screen, a wearable device (e.g., a glove, a watch, etc.), a controller, variations or combinations of one or more of the same, and / or any other type or form of electronic input mechanism.

[0089] Content player 720 also includes a storage device 940 coupled to communication infrastructure 902 via a storage interface 938. Storage device 940 generally represents any type or form of storage device or medium capable of storing data and / or other computer-readable instructions. For example, storage device 940 is a magnetic disk drive, a solid-state drive, an optical disk drive, a flash drive, or the like. Storage interface 938 generallyAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO represents any type or form of interface or device for transferring data between storage device 940 and other components of content player 720.

[0090] Example Embodiments:

[0091] Example 1. A computer-implemented method comprising: receiving an input audio signal comprising dialogue mixed with background audio; separating the input audio signal into a dialogue stem and a background stem; segmenting the dialogue stem into a plurality of utterance segments that include speech; for each utterance segment, computing an intelligibility score based on at least one intelligibility measurement mechanism; identifying, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement; for each identified utterance segment, calculating a gain adjustment to apply to at least one of the background stem or the dialogue stem; applying, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; and recombining the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

[0092] Example 2. The computer-implemented method of Example 1, wherein separating the input audio signal comprises applying a machine learning model that is trained to separate speech from non-speech audio in mixed content.

[0093] Example 3. The computer-implemented method of any of Examples 1-2, wherein the intelligibility score is computed using a user-customizable intelligibility measurement mechanism.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0094] Example 4. The computer-implemented method of any of Examples 1-3, wherein identifying the one or more utterance segments that require intelligibility enhancement comprises determining that the one or more utterance segments fail to satisfy a predefined threshold for intelligibility.

[0095] Example 5. The computer-implemented method of any of Examples 1-4, wherein the gain adjustment comprises at least one of a gain attenuation applied to the background stem and a gain boost applied to the dialogue stem.

[0096] Example 6. The computer-implemented method of any of Examples 1-5, wherein the gain adjustment is calculated based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment.

[0097] Example 7. The computer-implemented method of any of Examples 1-6, wherein the gain adjustment is subject to a maximum adjustment limit to prevent overprocessing.

[0098] Example 8. The computer-implemented method of any of Examples 1-7, further comprising generating, for each identified utterance segment, a gain adjustment envelope based at least in part on at least one of: an utterance start point that identifies when the utterance segment begins; an utterance end point that identifies when the utterance segment ends; audio events proximate the utterance start and end points; a loudness level associated with the dialogue segment at the utterance segment; and a loudness level associated with the background segment at the utterance segment.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0099] Example 9. The computer-implemented method of any of Examples 1-8, wherein generating the gain adjustment envelope comprises configuring attack and release parameters associated with the gain adjustment envelope to smooth the gain adjustment envelope and avoid audio artifacts.

[0100] Example 10. The computer-implemented method of any of Examples 1-9, wherein applying the gain adjustment comprises at least one of generating, based on the calculated gain adjustment, gain adjustment metadata and transmitting the gain adjustment metadata to a playback device for customized application during audio playback.

[0101] Example 11. A system comprising: at least one physical processor; and physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to: receive an input audio signal comprising dialogue mixed with background audio; separate the input audio signal into a dialogue stem and a background stem; segment the dialogue stem into a plurality of utterance segments that include speech; for each utterance segment, compute an intelligibility score based on at least one intelligibility measurement mechanism; identify, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement; for each identified utterance segment, calculate a gain adjustment to apply to at least one of the background stem or the dialogue stem; apply, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; and recombine the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO

[0102] Example 12. The system of Example 11, wherein separating the input audio signal comprises applying a machine learning model that is trained to separate speech from nonspeech audio in mixed content.

[0103] Example 13. The system of any of Examples 11-12, wherein the intelligibility score is computed using a user-customizable intelligibility measurement mechanism.

[0104] Example 14. The system of any of Examples 11-13, wherein identifying the one or more utterance segments that require intelligibility enhancement comprises determining that the one or more utterance segments fail to satisfy a predefined threshold for intelligibility.

[0105] Example 15. The system of any of Examples 11-14, wherein the gain adjustment comprises at least one of: a gain attenuation applied to the background stem; and a gain boost applied to the dialogue stem.

[0106] Example 16. The system of any of Examples 11-15, wherein the gain adjustment is calculated based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment.

[0107] Example 17. The system of any of Examples 11-16, wherein the gain adjustment is subject to a maximum adjustment limit to prevent over-processing.

[0108] Example 18. The system of any of Examples 11-17, wherein the computerexecutable instructions further cause the physical processor to generate, for each identified utterance segment, a gain adjustment envelope based at least in part on at least one of: an utterance start point that identifies when the utterance segment begins; an utterance end point that identifies when the utterance segment ends; audio events proximate to the utterance startAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO and end points; a loudness level associated with the dialogue segment at the utterance segment; and a loudness level associated with the background segment at the utterance segment.

[0109] Example 19. The system of any of Examples 11-18, wherein applying the gain adjustment comprises at least one of generating, based on the calculated gain adjustment, gain adjustment metadata; and transmitting the gain adjustment metadata to a playback device for customized application during audio playback.

[0110] Example 20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to receive an input audio signal comprising dialogue mixed with background audio; separate the input audio signal into a dialogue stem and a background stem; segment the dialogue stem into a plurality of utterance segments that include speech; for each utterance segment, compute an intelligibility score based on at least one intelligibility measurement mechanism; identify, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement; for each identified utterance segment, calculate a gain adjustment to apply to at least one of the background stem orthe dialogue stem; apply, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; and recombine the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

[0111] As detailed above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable ofAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing device(s) may each include at least one memory device and at least one physical processor.

[0112] In some examples, the term "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations, or combinations of one or more of the same, or any other suitable storage memory.

[0113] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the above-described memory device. Examples of physical processors include, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.

[0114] Although illustrated as separate elements, the modules described and / or illustrated herein may represent portions of a single module or application. In addition, in certain embodiments one or more of these modules may represent one or more software applicationsAttorney Docket No.: 039400.002902Client Reference No.: P61700-WO or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or illustrated herein may represent modules stored and configured to run on one or more of the computing devices or systems described and / or illustrated herein. One or more of these modules may also represent all or portions of one or more special-purpose computers configured to perform one or more tasks.

[0115] In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0116] In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, without limitation, transmissiontype media, such as carrier waves, and non-transitory-type media, such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical-storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0117] The process parameters and sequence of the steps described and / or illustrated herein are given by way of example only and can be varied as desired. For example,Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO while the steps illustrated and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0118] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the present disclosure.

[0119] Unless otherwise noted, the terms "connected to" and "coupled to" (and their derivatives), as used in the specification and claims, are to be construed as permitting both direct and indirect (i.e., via other elements or components) connection. In addition, the terms "a" or "an," as used in the specification and claims, are to be construed as meaning "at least one of." Finally, for ease of use, the terms "including" and "having" (and their derivatives), as used in the specification and claims, are interchangeable with and have the same meaning as the word "comprising."

Claims

Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO WHAT IS CLAIMED IS:

1. A computer-implemented method comprising:receiving an input audio signal comprising dialogue mixed with background audio; separating the input audio signal into a dialogue stem and a background stem; segmenting the dialogue stem into a plurality of utterance segments that include speech; computing, using at least one intelligibility measurement mechanism, an intelligibility score for each utterance segment;identifying, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement;for each identified utterance segment, calculating a gain adjustment to apply to at least one of the background stem or the dialogue stem;applying, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; andrecombining the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

2. The computer-implemented method of claim 1, wherein separating the input audio signal comprises applying a machine learning model that is trained to separate speech from non-speech audio in mixed content.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO 3. The computer-implemented method of claim 1, wherein the intelligibility score is computed using a user-customizable intelligibility measurement mechanism.

4. The computer-implemented method of claim 1, wherein identifying the one or more utterance segments that require intelligibility enhancement comprises determining that the one or more utterance segments fail to satisfy a predefined threshold for intelligibility.

5. The computer-implemented method of claim 1, wherein the gain adjustment comprises at least one of:a gain attenuation applied to the background stem; anda gain boost applied to the dialogue stem.

6. The computer-implemented method of claim 1, wherein the gain adjustment is calculated based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment.

7. The computer-implemented method of claim 1, wherein the gain adjustment comprises at least one of attenuating or boosting one or more frequency bands in at least one of the background stem and the dialogue stem.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO 8. The computer-implemented method of claim 1, wherein the gain adjustment is subject to a maximum adjustment limit to prevent over-processing.

9. The computer-implemented method of claim 1, further comprising generating, for each identified utterance segment, a gain adjustment envelope based at least in part on at least one of:an utterance start point that identifies when the utterance segment begins;an utterance end point that identifies when the utterance segment ends;audio events proximate the utterance start and end points;a loudness level associated with the dialogue segment at the utterance segment; and a loudness level associated with the background segment at the utterance segment.

10. The computer-implemented method of claim 9, wherein generating the gain adjustment envelope comprises configuring attack and release parameters associated with the gain adjustment envelope to smooth the gain adjustment envelope and avoid audio artifacts.

11. The computer-implemented method of claim 1, wherein applying the gain adjustment comprises at least one of:generating, based on the calculated gain adjustment, gain adjustment metadata; and transmitting the gain adjustment metadata to a playback device for customized application during audio playback.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO 12. A system comprising:at least one physical processor; andphysical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:receive an input audio signal comprising dialogue mixed with background audio; separate the input audio signal into a dialogue stem and a background stem; segment the dialogue stem into a plurality of utterance segments that include speech;compute, using at least one intelligibility measurement mechanism, an intelligibility score for each utterance segment;identify, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement;for each identified utterance segment, calculate a gain adjustment to apply to at least one of the background stem or the dialogue stem;apply, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; andrecombine the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO 13. The system of claim 12, wherein separating the input audio signal comprises applying a machine learning model that is trained to separate speech from non-speech audio in mixed content.

14. The system of claim 12, wherein the intelligibility score is computed using a user-customizable intelligibility measurement mechanism.

15. The system of claim 12, wherein identifying the one or more utterance segments that require intelligibility enhancement comprises determining that the one or more utterance segments fail to satisfy a predefined threshold for intelligibility.

16. The system of claim 12, wherein the gain adjustment is calculated based at least in part on a comparison of the loudness of the dialogue stem and the background stem at each identified utterance segment.

17. The system of claim 12, wherein the gain adjustment is subject to a maximum adjustment limit to prevent over-processing.Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO 18. The system of claim 12, wherein the computer-executable instructions further cause the physical processor to generate, for each identified utterance segment, a gain adjustment envelope based at least in part on at least one of:an utterance start point that identifies when the utterance segment begins;an utterance end point that identifies when the utterance segment ends;audio events proximate to the utterance start and end points;a loudness level associated with the dialogue segment at the utterance segment; and a loudness level associated with the background segment at the utterance segment.

19. The system of claim 12, wherein applying the gain adjustment comprises at least one of:generating, based on the calculated gain adjustment, gain adjustment metadata; and transmitting the gain adjustment metadata to a playback device for customized application during audio playback.

20. A non-transitory computer-readable medium comprising one or more computerexecutable instructions that, when executed by at least one processor of a computing device, cause the computing device to:receive an input audio signal comprising dialogue mixed with background audio; separate the input audio signal into a dialogue stem and a background stem; segment the dialogue stem into a plurality of utterance segments that include speech;Attorney Docket No.: 039400.002902Client Reference No.: P61700-WO compute, using at least one intelligibility measurement mechanism, an intelligibility score for each utterance segment;identify, based on an analysis of the intelligibility scores, one or more utterance segments that require intelligibility enhancement;for each identified utterance segment, calculate a gain adjustment to apply to at least one of the background stem or the dialogue stem;apply, during each identified utterance segment, the gain adjustment to at least one of the background stem or the dialogue stem; andrecombine the dialogue stem with the background stem to produce an enhanced audio signal with improved intelligibility.

Images