Method and electronic device for determining bitrates for encoding audio elements in mix presentation

Abstract

In an embodiment of the disclosure, the method may include obtaining a mix presentation including a plurality of audio elements. The method may include determining, for each of the plurality of the audio elements in the mix presentation, a perceptual importance. The method may include allocating, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements. The method may include generating an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.

Description

METHOD AND ELECTRONIC DEVICE FOR DETERMINING BITRATES FOR ENCODING AUDIO ELEMENTS IN MIX PRESENTATION

[0001] This application generally relates to bitrate determinations when encoding multiple audio elements in a mix presentation.

[0002] A loudspeaker converts an electrical audio signal into a corresponding sound. Loudspeakers can be used for playing music, listening to audio content corresponding to video content (e.g., audio of a TV show or a movie), etc. An entertainment system often involves multiple loudspeakers that play audio. For example, an entertainment system may include a pair of left-right stereo loudspeakers, a subwoofer, a center loudspeaker, a pair of left-right surround loudspeakers, and / or a pair of left-right rear surround loudspeakers. The number of loudspeakers in a system are often referred to by an x.y convention, where x is the number of loudspeakers used in the system and y refers to the number of subwoofers used in the system.

[0003] One important aspect of delivering audio for a set of speakers (e.g., home entertainment loudspeakers, headphones, etc.) is audio coding, which involves coding and transmitting audio data with an efficient perceptual quality vs. bitrate tradeoff.

[0004] In an embodiment of the disclosure, the method may include obtaining a mix presentation including a plurality of audio elements. The method may include determining, for each of the plurality of the audio elements in the mix presentation, a perceptual importance. The method may include allocating, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements. Bitrates corresponding to at least two of the plurality of the audio elements may be different. The method may include generating an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.

[0005] In an embodiment of the disclosure, an electronic device may comprise at least one processor comprising processing circuitry, and at least one memory including one or more instructions. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to obtain a mix presentation including a plurality of audio elements. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to determine, for each of the plurality of the audio elements in the mix presentation, a perceptual importance. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to allocate, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements. Bitrates corresponding to at least two of the plurality of the audio elements are different. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to generate an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.

[0006] In an embodiment of the disclosure, one or more non-transitory computer-readable storage media storing instructions that are operable when executed by one or more processors, cause the electronic device to perform the method.

[0007] Figure 1 illustrates an example method for determining the bitrate of each of multiple audio elements in a mix presentation.

[0008] Figure 2 illustrates an example method implementing certain techniques of the approach of Figure 1.

[0009] Figure 3 illustrates a detailed example implementation of a perceptual importance determination technique.

[0010] Figure 4 illustrates an example embodiment in which bitrate allocation and encoding are performed on a frame-by-frame basis.

[0011] Figure 5 illustrates an example embodiment in which metadata is used to adjust bitrate determinations for audio elements.

[0012] Figure 6 illustrates an example computing system.

[0013] Immersive audio can include audio objects, which is an audio track or stem with associated (typically spatial) metadata. Audio objects require a rendering method using the metadata to make the audio listenable for a particular set of speakers. In practice, each audio object is associated with a mono signal.

[0014] Audio elements can include one of: 1) a channel-based element such as 7.1, 5.1., stereo, etc.; 2) a scene-based element such as Ambisonics; or 3) an audio object element. Elements can have more than one channel per element; in other words, an element can be represented by multiple channels ck per audio element. An audio element may have one or more time-domain signals, and type (channel, scene, object) metadata.

[0015] A mix presentation is a set of audio elements intended for joint presentation with simultaneous or interactive rendering. For instance, a mix presentation that includes an Ambisonics element along with two stereo elements, where the two stereo elements carry different languages that can be changed, is one example of a mix presentation. In a mix presentation, a bitstream can contain several audio elements, such as channel-based simple elements (mono, stereo), or immersive channel-based bed, scene-based, audio objects with dynamic spatial metadata.

[0016] Whether audio content is intended as a mix presentation is usually defined by the content creator and signaled in the transmission format. A mix presentation includes metadata that describes how the audio elements are rendered and mixed together for playback through loudspeakers or headphones in different situations. Unlike in standard audio reproduction, all transmitted audio is typically not rendered together simultaneously. For example, in the case of a sports broadcast, a mix presentation may include 3 audio elements: two stereo languages (e.g. Spanish, English) and a 5.1 multichannel element with a common background, as well as metadata defining how to decode, render and mix these elements and to allow the end user to switch between languages.

[0017] For mix presentations, a single bitstream representing the mix encodes multiple audio elements. Encoding should maximize quality while achieving a bitrate that can be efficiently transmitted, and sending multiple audio elements requires allocating the bitrate among those elements. In existing approaches, the elements of a mix presentation are required to be coded with separate audio codec (e.g. Opus) instances. Thus, the bitrates of each mix presentation audio element must be decided before actual transmission, and these bitrates are given to the codec instances as parameters. In addition, simplistic solutions, such as assigning equal bitrate per each audio element in a mix presentation, are suboptimal.

[0018] Figure 1 illustrates an example method for determining the bitrate of each of multiple audio elements in a mix presentation. In the example of Figure 1, mix presentation audio elements 104 are provided for simultaneous encoding and playback. Mix presentation metadata 102 may also be included, for example to specify a particular gain for a particular element, provide language options, etc. mix presentation metadata 102 and mix presentation audio elements 104 may be used as inputs to the perceptual importance calculation engine 110. For convenience of description, mix presentation audio elements may be referred to as a mix presentation.

[0019] Perceptual importance calculation engine 110 of the electronic device may determine a relative perceptual importance, how each distinct audio element will be encoded, as described more fully below. Bitrate allocation engine 115 takes the output (e.g., the relative perceptual importance) of the perceptual importance calculation engine 110 and allocates bitrate for each of a plurality of audio elements in the mix presentation. An embodiment of the disclosure may use three hyperparameters to make this allocation: (1) the total available bitrate for all the audio content (e.g., whole content); (2) a lower bitrate limit for each element, specifying the lowest bitrate that any particular element can achieve and (3) a maximum bitrate limit for each element, specifying the highest bitrate that element can achieve.

[0020] In an embodiment of the disclosure, low and maximum bitrate limits may depend on the codec or, other factors. For example, the bitrate limits may be set as a function of an audio element type. A bitrate may be allocated to each of a plurality of audio elements based on respective relative perceptual importance metrics. Given the allocated bitrate (and, optionally, one or more additional encoding parameters), each audio codec instance may perform bit allocation within its corresponding audio element. Meanwhile, relative perceptual importance may also be referred to simply as perceptual importance. The perceptual importance may be represented as a quantified scalar value, such as a metric.

[0021] Each codec instance 120 typically processes one audio element, and an input to that instance is a target bitrate for the element. For instance, a stereo audio element and a 5.1 multichannel audio element would each be encoded using a separate codec instance 120. Once each codec instance 120 encodes its audio element, the end result is an encoded mix presentation 125.

[0022] Figure 2 illustrates an example method implementing certain techniques of the approach of Figure 1. Operation S210 of the example method of Figure 2 includes obtaining a mix presentation including a plurality of audio element. For example, the electronic device may access a mix presentation input audio that includes multiple audio elements, for instance as shown in element 104 of the example of Figure 1. Operation S220 of the example method of Figure 2 includes determining, for each of the plurality of the audio elements in the mix presentation, a perceptual importance of that audio element to the mix presentation. An embodiment of the disclosure perform operation S220 by determining and accounting for channel correlations within each audio element. For example, to account for the channel correlations within each stereo-, multichannel- or scene-based audio element, an embodiment of the disclosure transform the audio element channel signals with an energy-packing, decorrelating transform, as described more fully below. In this approach, the bitrate requirement mainly depends on the amount of uncorrelated energy of each element.

[0023] In an embodiment of the disclosure, the electronic device may perform correlation analysis and metadata accounting. Metadata accounting refers to tracking or estimating metadata overhead separately from coded audio payload.

[0024] In an embodiment of the disclosure, the electronic device may perform decorrelating transform and sum. Each decorrelating-transformed signal of the audio element is then summed together, and the remainder of the system operates on perceptually weighted band energies of the audio elements in the transformed domain after the correlation analysis and the metadata accounting. An embodiment of the disclosure may then use two factors in a perceptual importance measure as described in U.S. Patent Application Publication No. 2025 / 0046321, which description is incorporated herein by reference. These factors are independent of element channel locations / positional metadata and of decoder rendering. First, a signal that has more total energy needs more bits, compared to a signal that is mostly silent. The electronic device may calculate or determine the total energy as the sum of perceptually weighted band energies. Second, the electronic device may also analyze how much each audio element is masked by the other elements. The electronic device may approximate the masking signal by the sum of audio elements: the masking signal (aka "sum signal") includes all elements of the mix presentation that are deemed as masking the analyzed element at given time. For most situations, a good approximation is that this sum includes all audio elements, but it can include a subset of e.g. the currently active elements in a given presentation. The final unmasking factor may be averaged and normalized over multiple playback presentations with different masking signals, in an embodiment of the disclosure.

[0025] Masking analysis (the second factor) complements total energy determinations (the first factor) with a local unmasking average. The final measure may be calculated as the weighted sum of the two factors with, e.g., relations 0.2 and 0.8, for example. In an embodiment of the disclosure, a perceptual importance determination can take into account the mix presentation relevant metadata detailing e.g., the possible dynamic changes to the playback levels for each audio element. Meanwhile, operations performed by the perceptual importance calculation engine to determine perceptual importance may be referred to as perceptual importance calculation or perceptual importance determination. Figure 3 illustrates a detailed example implementation of a perceptual importance determination technique. In the example of Figure 3, there are n audio elements, and the audio element time domain signal is adjusted according to the playback metadata 300 of the mix presentation. In an embodiment of the disclosure, the electronic device may adjust, for an audio element, a time-domain signal according to playback metadata of a Mix Presentation. An energy-packing, decorrelating transform 301 is used to account for element channel correlations. The electronic device may apply an energy-packing, decorrelating transform to the adjusted signal to account for channel correlations within the audio element. For instance, an embodiment of the disclosure may use principal component analysis (PCA) as the decorrelating transform, while other embodiments may use a singular value decomposition, for example.

[0026] In order to remove correlations between tracks ck in a particular audio element, where ck is greater than 1, and then consider only the remaining uncorrelated signals within each element, an embodiment of the disclosure first represent each audio element of a mix presentation with elements and multiple tracks with a single-track signal indicating the sum of the uncorrelated signals:

[0027]

[0028] where Ek is an audio signal ( ) matrix with tracks and n samples. Operation indicates sum across the element tracks. represents a ( ) matrix obtained with a linear, energy-packing and decorrelating transform such as PCA.

[0029] In an embodiment of the disclosure, PCA may be a linear transform 1) to pack energy as efficiently as possible to each transformed component starting from the first, 2) while constraining the transformed components to be orthogonal, i.e. decorrelated. Decorrelating transform (e.g., PCA-like transform) may be also be obtained with Singular Value Decomposition. For stereo audio (2D signal), a common coding technique is Mid-Side coding: instead of coding the left (L) and right channels, they are first transformed to Mid: (L+R) / 2 and Side: (L-R) / 2. This is often close to the optimal components that would result from PCA.

[0030] PCA has conventionally been applied as a tool for multichannel audio compression. In contrast, an embodiment of the disclosure uses PCA for analyzing an audio codec's capability to compress multichannel material (e.g., for bitrate allocation). PCA may be performed on a short-term, frame-based basis, or as a longer-term approximation.

[0031] After obtaining the single-track element principal component sum signals , each element is analyzed in perceptual frequency bands via Short-Time Fourier Transform (STFT), so that the frequency bins are grouped together in bands. The purpose of the banding is to utilize a frequency-dependent weighting mimicking audio codec analysis. For the element signal , an embodiment of the disclosure calculate the perceptually-weighted energy per time-frequency tile (t, i) as

[0032]

[0033] where indicates the frequency band index, the number of bins in the band, and the predetermined perceptual weights. Operation sums over the frequency bins within band . The outcomes of this determination, , are then used in the perceptual importance calculation, accounting for both 1) the total frequency-weighted energy of the element, and 2) the average measure of how much the element is locally unmasked, results in relative perceptual weights for all elements of the mix presentation.

[0034] In the example of Figure 3, each of the n audio-element signals are adjusted according to their respective playback metadata, and the decorrelation operation 301 is applied to each separate audio-element signal in the mix. For each of the n audio-element signals, the decorrelated components of that signal are summed to form the individual transformed signal for that element (illustrated as element 302). The electronic device may sum components output from the decorrelating transform to generate an individual transformed signal for the audio element and obtain a sum signal. In addition, the sum of all transformed signals 303 is used to mask each individual transformed signal 302. The electronic device may generate a masking signal by summing transformed signals of a plurality of audio elements that are deemed to mask the audio element.

[0035] In the example of Figure 3, each individual transformed signal 302 and the sum of all transformed signals 303 are processed via respective Short-Time Fourier Transforms (STFT) 304. Each STFT signal is grouped into perceptual frequency bands, which bands can originate from the codec. Banding is defined by STFT bin indices 305. The electronic device may group each STFT signal into a plurality of perceptual frequency bands. The energy of each frequency band normalized by the number of bins in the band is calculated for each time frame 306. The electronic device may calculate an energy of each perceptual frequency band, normalized by a number of STFT bins included in the band.

[0036] In an embodiment of the disclosure, a priori relative perceptual importance per frequency band 307 is utilized to weight each band 308 similarly in each the sum signal and the individual element signal. The electronic device may obtain a perceptual importance values (e.g., weight) for respective perceptual frequency bands. This weighting can originate from the relative bit assignment in the core audio codec, or from an alternative psychoacoustical model. The electronic device may apply the perceptual importance values to weight band energies of both signals in a similar manner. Total perceptual weighted energy is calculated for the element signal 309. Activity detection block 310 is used to find the non-silent segments (or non-silent time frames) of the element signal and at those time frames, the perceptual energy of the frequency bands is compared against the perceptual energy of the sum signal, and averaged over time, and then summed over frequency bands 311. The final importance measure is calculated as the weighted sum of the two factors (total element perceptual energy, and relative energy average compared to sum signal) 312. This process occurs for each n audio element in the mix, resulting in a set of n importance measures, each corresponding to a particular audio element.

[0037] Aspects of the example of Figure 3, such as the use of Fourier transforms 304 and the total perceptual energy determination 309, are techniques as described in U.S. Patent Application Publication No. 2025 / 0046321, which techniques are incorporated herein by reference. However, the example of Figure 3 relates to mix presentations that include multiple audio elements, not just to single-channel data objects, and as a result, the example of Figure 3 introduces additional techniques such as metadata accounting and per-audio-element decorrelating transforms 301 in order to take advantage of correlations among audio elements to determine bitrate allocations for a complete mix presentation.

[0038] Operation S230 of the example method of Figure 2 includes allocating, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements, for instance using bitrate allocation engine 115 of the example of Figure 1. To obtain the final assigned bitrate per audio element, an embodiment of the disclosure utilize an interactive algorithm based on perceptual measures , and three hyperparameters: 1) total available bitrate for all audio elements , 2) low limit bitrate per type of audio element, and 3) maximum bitrate per type of audio element. Low limit bitrate and Maximum bitrate vary depending of the type of the audio element (and as a function of the codec and the total rate), in an embodiment of the disclosure. For example, if the quality for stereo saturates at around 128 kbit / s, assigning more rate yields diminishing returns, while potentially harming other elements. This is different for multichannel or HOA (Higher-Order Ambisonics) elements. The limits can be heuristically assigned. Then, in an embodiment of the disclosure, an iterative bit reservoir loop assigns the final rates. In an embodiment of the disclosure, bitrates corresponding to at least two of the plurality of the audio elements may be different from each other.

[0039] Operation S240 of the example method of Figure 2 includes generating an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrates for each of the plurality of audio elements. For instance, Figure 1 illustrates using audio codec instances 120 (where each instance may be used to encode a particular audio signal in the mix presentation) to collectively generate the coded mix presentation 125 for the input audio elements 104. As described above, the bitrate allocation is based on the perceptual importance of each audio element in the mix relative to the mix as a whole (e.g., based on an importance measure for each of the n audio elements), and in an embodiment of the disclosure on hyperparameters such as the overall available bit rate and per-element minimum and maximum bitrates (which may vary based on the element).

[0040] In an embodiment of the disclosure, each audio element signal Sn is the entire temporal audio signal for particular track, and bitrate allocation is determined based on this entire signal. Likewise, each entire signal is encoded by the codec in a given instance. In an embodiment of the disclosure, the decorrelating transform (e.g., operation 301) occurs for each full temporal signal, such that decorrelations are determined across that entire signal. Other embodiments may perform bitrate allocation and encoding on a subset of the entire signal, including on a frame-by-frame basis. Figure 4 illustrates an example embodiment in which bitrate allocation and encoding are performed on a frame-by-frame basis. Each audio element signal Sn may first be processed using metadata 401 for that audio element. Then, in the example of Figure 4, framing analysis 401 determines how long each subsegment, or frame, of the audio element signal will be. Framing information 403 is also synchronized with and used by each codec instance 404 to encode each respective frame. Each frame's worth of information for each audio signal is decorrelated, for instance as described in Figure 3, and bit allocation 404 is done dynamically once per frame, outputting the rate per audio-element frame . This, along with the framing sync information 403, is used as the input for frame processing with the audio codec 404. The codec can also optionally re-use the frequency-domain transform output (e.g. MDCT) from 401 to reduce latency and processing.

[0041] Dynamic bitrate allocation 402 can be one of many options, that may include lookback or lookahead. In case a perceptual method as in the example of Figure 3 is used, operation 402 includes both perceptual importance calculation and bit allocation. As a result, unlike conventional encoding in which bitrate and coding decisions are made on the basis of entire input signal (e.g., a whole track), the example of Figure 4 can dynamically make bitrate and encoding determinations on a frame-by-frame basis, optimizing the correlations within and between audio-element signals on a dynamic basis, rather than assigning bitrate statically to each particular audio-element in a track.

[0042] In an embodiment of the disclosure, codec resource allocation can also be affected by higher-level, content-aware analysis. For instance, in an embodiment of the disclosure there may be metadata, e.g., from a content creator and / or a machine-learning analysis or classification that can affect allocation. This metadata may be time varying, in an embodiment of the disclosure. An audio element may have a specified interactive level change in playback, which can be manually indicated during content creation. This can happen, for example, when some users require accessibility (e.g., boosted dialogue). A content creator can also annotate the audio element content type generally, which can help with selecting the appropriate codec rate and other settings, in case the codec operates more efficiently for certain types of content than for others (e.g. for speech).

[0043] Figure 5 illustrates an example embodiment in which metadata from a content creator and / or from a classifier is used to adjust bitrate determinations for an audio element. Metadata 501 comes from the content creation process, while metadata 502 comes from a non-context aware classifier model 502 and metadata 503 comes from a context-aware classifier 503 that takes into account scene information corresponding to the audio. These classifiers can take into account both the audio content as well as the rendering and mixing metadata 501, in an embodiment of the disclosure.

[0044] The metadata can be used to adjust hyperparameters for individual audio elements 506 during bitrate allocation 506 after a final importance measure has been computed in 507. These element-specific hyperparameters 504 override global hyperparameters 505 if metadata is present for that particular elements, resulting in content-aware allocation 508.

[0045] Classifiers 502 and 503 can be based on machine learning or can be simpler knowledge-based audio processing units. For example, a simple frequency-weighted transient detector can complement the energy-based analysis of Figure 3: the intensity of the detected transients typically leads to greater bitrate being allocated to the corresponding mix presentation audio element, in order to avoid audible distortion.

[0046] In an embodiment of the disclosure, in situations where the end user rendering is known, pre-rendering can be performed at the encoder, which may lead to a reduction in the number of mix presentation audio elements or the channels within each element, thus allowing for greater bitrate to the remaining elements or channels. In such embodiment, the decoder communicates the end-user reproduction device system and algorithms to the encoder (e.g. in a streaming situation). In some cases, this may mean changing the mix presentation audio element type (i.e. from Ambisonics to stereo).

[0047] The techniques described herein result in high-quality compression of mix presentation audio signals that contain multiple audio elements, thereby improving the efficiency of content transmission while still retaining audio quality (e.g., based on perceptual importance).

[0048] Figure 6 illustrates an example computer system 600. In an embodiment of the disclosure, one or more computer systems 600 perform one or more operations of one or more methods described or illustrated herein. A computing system or computing systems 600 may be referred to as an electronic device or electronic devices, and various operations may be performed by one or more electronic devices. In an embodiment of the disclosure, one or more computer systems 600 provide functionality described or illustrated herein. In an embodiment of the disclosure, software running on one or more computer systems 600 performs one or more operations of one or more methods described or illustrated herein or provides functionality described or illustrated herein. An embodiment of the disclosure include one or more portions of one or more computer systems 600. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

[0049] This disclosure contemplates any suitable number of computer systems 600. This disclosure contemplates computer system 600 taking any suitable physical form. As example and not by way of limitation, computer system 600 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 600 may include one or more computer systems 600; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 600 may perform without substantial spatial or temporal limitation one or more operations of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 600 may perform in real time or in batch mode one or more operations of one or more methods described or illustrated herein. One or more computer systems 600 may perform at different times or at different locations one or more operations of one or more methods described or illustrated herein, where appropriate.

[0050] In an embodiment of the disclosure, computer system 600 includes a processor 602, memory 604, storage 606, an input / output (I / O) interface 608, a communication interface 610, and a bus 612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0051] In an embodiment of the disclosure, processor 602 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 602 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 604, or storage 606; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 604, or storage 606. In an embodiment of the disclosure, processor 602 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 602 including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 604 or storage 606, and the instruction caches may speed up retrieval of those instructions by processor 602. Data in the data caches may be copies of data in memory 604 or storage 606 for instructions executing at processor 602 to operate on; the results of previous instructions executed at processor 602 for access by subsequent instructions executing at processor 602 or for writing to memory 604 or storage 606; or other suitable data. The data caches may speed up read or write operations by processor 602. The TLBs may speed up virtual-address translation for processor 602. In an embodiment of the disclosure, processor 602 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 602 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 602 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 602. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0052] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.

[0053] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0054] The processor may include various processing circuitry and / or multiple processors.　 For example, as used herein, including the claims, the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when "a processor", "at least one processor", and "one or more processors" are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner.　 At least one processor may execute program instructions to achieve or perform various functions.

[0055] In an embodiment of the disclosure, memory 604 includes main memory for storing instructions for processor 602 to execute or data for processor 602 to operate on. As an example and not by way of limitation, computer system 600 may load instructions from storage 606 or another source (such as, for example, another computer system 600) to memory 604. Processor 602 may then load the instructions from memory 604 to an internal register or internal cache. To execute the instructions, processor 602 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 602 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 602 may then write one or more of those results to memory 604. In an embodiment of the disclosure, processor 602 executes only instructions in one or more internal registers or internal caches or in memory 604 (as opposed to storage 606 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 604 (as opposed to storage 606 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 602 to memory 604. Bus 612 may include one or more memory buses, as described below. In an embodiment of the disclosure, one or more memory management units (MMUs) reside between processor 602 and memory 604 and facilitate accesses to memory 604 requested by processor 602. In an embodiment of the disclosure, memory 604 includes random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 604 may include one or more memories 604, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0056] In an embodiment of the disclosure, storage 606 includes mass storage for data or instructions. As an example and not by way of limitation, storage 606 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 606 may include removable or non-removable (or fixed) media, where appropriate. Storage 606 may be internal or external to computer system 600, where appropriate. In an embodiment of the disclosure, storage 606 is non-volatile, solid-state memory. In an embodiment of the disclosure, storage 606 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 606 taking any suitable physical form. Storage 606 may include one or more storage control units facilitating communication between processor 602 and storage 606, where appropriate. Where appropriate, storage 606 may include one or more storages 606. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0057] In an embodiment of the disclosure, I / O interface 608 includes hardware, software, or both, providing one or more interfaces for communication between computer system 600 and one or more I / O devices. Computer system 600 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 600. As an example and not by way of limitation, an I / O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device or a combination of two or more of these. An I / O device may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 608 for them. Where appropriate, I / O interface 608 may include one or more device or software drivers enabling processor 602 to drive one or more of these I / O devices. I / O interface 608 may include one or more I / O interfaces 608, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0058] In an embodiment of the disclosure, communication interface 610 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 600 and one or more other computer systems 600 or one or more networks. As an example and not by way of limitation, communication interface 610 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 610 for it. As an example and not by way of limitation, computer system 600 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 600 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 600 may include any suitable communication interface 610 for any of these networks, where appropriate. Communication interface 610 may include one or more communication interfaces 610, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0059] In an embodiment of the disclosure, bus 612 includes hardware, software, or both coupling components of computer system 600 to each other. As an example and not by way of limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 612 may include one or more buses 612, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0060] Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0061] In an embodiment of the disclosure, the method may include obtaining a mix presentation including a plurality of audio elements. The method may include determining, for each of the plurality of the audio elements in the mix presentation, a relative perceptual importance. The method may include allocating, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements. Bitrates corresponding to at least two of the plurality of the audio elements are different. The method may include generating an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.

[0062] In an embodiment of the disclosure, the determining, for each of the plurality of the audio elements in the mix presentation, a relative perceptual importance may comprise generating a decorrelated transform for each of the plurality of the audio elements.

[0063] In an embodiment of the disclosure, the method may include masking, for each of the plurality of the audio elements, the decorrelated transform based on a collective sum of decorrelated transforms corresponding to the plurality of the audio elements.

[0064] In an embodiment of the disclosure, the allocating of the bitrate for each of the plurality of the audio elements may be further based on a total bitrate for the mix presentation.

[0065] In an embodiment of the disclosure, the allocating of the bitrate for each of the plurality of the audio elements may be further based on one or more of a minimum bitrate for at least one audio element or a maximum bitrate for at least one audio element.

[0066] In an embodiment of the disclosure, the allocating of the bitrate for each of the plurality of the audio elements may be further based on metadata for the plurality of the audio elements.

[0067] In an embodiment of the disclosure, the metadata may be defined by one or more of a content creator of the mix presentation or a classification of the mix presentation.

[0068] In an embodiment of the disclosure, the method may include generating a series of frames representing the mix presentation. The method may include allocating the bitrate to each of the plurality of the audio elements on a frame-by-frame basis.

[0069] In an embodiment of the disclosure, an electronic device may comprise at least one processor comprising processing circuitry, and at least one memory including one or more instructions. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to obtain a mix presentation including a plurality of audio elements. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to determine, for each of the plurality of the audio elements in the mix presentation, a relative perceptual importance. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to allocate, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements. Bitrates corresponding to at least two of the plurality of the audio elements are different. The at least one memory may include one or more instructions, executed by the at least one processor individually or collectively, to cause the electronic device to generate an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.

[0070] In an embodiment of the disclosure, in the determining the relative perceptual importance, the one or more instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to generate a decorrelated transform for each of the plurality of the audio elements.

[0071] In an embodiment of the disclosure, the one or more instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to mask, for each of the plurality of the audio elements, the decorrelated transform based on a collective sum of decorrelated transforms corresponding to the plurality of the audio elements.

[0072] In an embodiment of the disclosure, the allocating the bitrate for each of the plurality of the audio elements may be further based on a total bitrate for the mix presentation.

[0073] In an embodiment of the disclosure, the allocating of the bitrate for each of the plurality of the audio elements may be further based on one or more of a minimum bitrate for at least one audio element or a maximum bitrate for at least one audio element.

[0074] In an embodiment of the disclosure, the allocating of the bitrate for each of the plurality of the audio elements may be further based on metadata for the plurality of the audio elements.

[0075] In an embodiment of the disclosure, one or more computer-readable storage media storing instructions that are operable when executed by one or more processors, may cause the electronic device to perform the method.

[0076] In an embodiment of the disclosure, a method may be provided. The method may include accessing a mix presentation input audio comprising a plurality of audio elements. The method may include determining, for each audio element in the mix presentation, a relative perceptual importance of that audio element to the mix presentation. The method may include allocating, based on the determined perceptual importances, a bitrate for each audio element. The method may include generating an encoded mix presentation audio stream by encoding each audio element according to the assigned bitrates.

[0077] In an embodiment of the disclosure, the method may include generating a decorrelated transform of each audio element.

[0078] In an embodiment of the disclosure, the method may include masking each decorrelated transform based on a collective sum of each of the decorrelated transforms.

[0079] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on a total bitrate for the mix presentation.

[0080] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on one or more of (1) a minimum bitrate for at least one audio element or (2) a maximum bitrate for at least one audio element.

[0081] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on metadata for that audio element.

[0082] In an embodiment of the disclosure, the metadata may be defined by one or more of (1) a content creator of the mix presentation or (2) a classification of the mix presentation.

[0083] In an embodiment of the disclosure, the method may include generating a series of frames representing the mix presentation and allocating bitrate to each audio element on a frame-by-frame basis.

[0084] In an embodiment of the disclosure, a system may comprise one or more non-transitory computer readable storage media storing instructions, and one or more processors coupled to the one or more non-transitory computer readable storage media and operable to execute the instructions. The one or more processor may be operable to execute the instructions to access a mix presentation input audio comprising a plurality of audio elements. The one or more processor may be operable to execute the instructions to determine, for each audio element in the mix presentation, a relative perceptual importance of that audio element to the mix presentation. The one or more processor may be operable to execute the instructions to allocate, based on the determined perceptual importances, a bitrate for each audio element. The one or more processor may be operable to execute the instructions to generate an encoded mix presentation audio stream by encoding each audio element according to the assigned bitrates.

[0085] In an embodiment of the disclosure, the one or more processor may be operable to execute the instructions to generate a decorrelated transform of each audio element.

[0086] In an embodiment of the disclosure, the one or more processor may be operable to execute the instructions to mask each decorrelated transform based on a collective sum of each of the decorrelated transforms.

[0087] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on a total bitrate for the mix presentation.

[0088] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on one or more of (1) a minimum bitrate for at least one audio element or (2) a maximum bitrate for at least one audio element.

[0089] In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on metadata for that audio element.

[0090] In an embodiment of the disclosure, the metadata may be defined by one or more of (1) a content creator of the mix presentation or (2) a classification of the mix presentation.

[0091] In an embodiment of the disclosure, the one or more processor may be operable to execute the instructions to generate a series of frames representing the mix presentation and allocating bitrate to each audio element on a frame-by-frame basis.

[0092] In an embodiment of the disclosure, the one or more non-transitory computer-readable storage media storing instructions that are operable when executed by one or more processors to access a mix presentation input audio comprising a plurality of audio elements, determine, for each audio element in the mix presentation, a relative perceptual importance of that audio element to the mix presentation, allocate, based on the determined perceptual importances, a bitrate for each audio element, and generate an encoded mix presentation audio stream by encoding each audio element according to the assigned bitrates.

[0093] In an embodiment of the disclosure, the one or more non-transitory computer-readable storage media may store instructions that are operable when executed by one or more processors to generating a decorrelated transform of each audio element.

[0094] In an embodiment of the disclosure, the one or more non-transitory computer-readable storage media may store instructions that are operable when executed by one or more processors that are operable to execute the instructions to mask each decorrelated transform based on a collective sum of each of the decorrelated transforms. In an embodiment of the disclosure, the allocating of a bitrate for each audio element may be further based on a total bitrate for the mix presentation.

[0095] Herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Moreover, "and" is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated otherwise by context.

[0096] This disclosure contemplates a system that includes one or more computer readable storage media storing instructions; and one or more processors coupled to the one or more computer readable storage media and operable to execute the instructions to perform certain functions includes embodiments in which those functions are performed by a single processor, embodiments in which those functions are performed by multiple processors that each perform all the functions, and embodiments in which those functions are performed by multiple processors (e.g., in separate computing devices) where each processor performs at least one function but less than all recited functions. The computer-readable media may be non-transitory.

[0097] A "non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0098] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend.

Claims

1.A method comprising:obtaining a mix presentation including a plurality of audio elements (S210);determining, for each of the plurality of the audio elements in the mix presentation, a perceptual importance (S220);allocating, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements (S230), wherein bitrates corresponding to at least two of the plurality of the audio elements are different; andgenerating an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements (S240).2.The method of Claim 1, wherein the determining, for each of the plurality of the audio elements in the mix presentation, the perceptual importance comprises generating a decorrelated transform for each of the plurality of the audio elements.3.The method of Claim 2, further comprising:masking, for each of the plurality of the audio elements, the decorrelated transform based on a collective sum of decorrelated transforms corresponding to the plurality of the audio elements.4.The method of Claim 3, wherein the allocating of the bitrate for each of the plurality of the audio elements is further based on a total bitrate for the mix presentation.5.The method of Claim 4, wherein the allocating of the bitrate for each of the plurality of the audio elements is further based on one or more of a minimum bitrate for at least one audio element or a maximum bitrate for at least one audio element.6.The method of any one of Claims 4 to 5, wherein the allocating of the bitrate for each of the plurality of the audio elements is further based on metadata for the plurality of the audio elements.7.The method of Claim 6, wherein the metadata is defined by one or more of a content creator of the mix presentation or a classification of the mix presentation.8.The method of any one of Claims 3 to 7, further comprising:generating a series of frames representing the mix presentation; andallocating the bitrate to each of the plurality of the audio elements on a frame-by-frame basis.9.An electronic device comprising:at least one processor (602) comprising processing circuitry;at least one memory (604) including one or more instructions, executed by the at least one processor (602) individually or collectively, to cause the electronic device to:obtain a mix presentation including a plurality of audio elements;determine, for each of the plurality of the audio elements in the mix presentation, a perceptual importance;allocate, based on the determined perceptual importance, a bitrate for each of the plurality of the audio elements, wherein bitrates corresponding to at least two of the plurality of the audio elements are different; andgenerate an encoded mix presentation audio stream by encoding each of the plurality of the audio elements according to the allocated bitrate for each of the plurality of the audio elements.10.The electronic device of Claim 9, wherein, in the determining the perceptual importance, the one or more instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:generate a decorrelated transform for each of the plurality of the audio elements.11.The electronic device of Claim 10, wherein the one or more instructions, when executed by the at least one processor individually or collectively, cause the electronic device to:mask, for each of the plurality of the audio elements, the decorrelated transform based on a collective sum of decorrelated transforms corresponding to the plurality of the audio elements.12.The electronic device of Claim 11, wherein the allocating the bitrate for each of the plurality of the audio elements is further based on a total bitrate for the mix presentation.13.The electronic device of Claim 12, wherein the allocating of the bitrate for each of the plurality of the audio elements is further based on one or more of a minimum bitrate for at least one audio element or a maximum bitrate for at least one audio element.14.The electronic device of any one of Claims 12 to 13, wherein the allocating of the bitrate for each of the plurality of the audio elements is further based on metadata for the plurality of the audio elements.15.One or more computer-readable storage media storing instructions that are operable when executed by one or more processors, cause the electronic device to perform the method of any one of Claims 1 to 8.

Images