Electronic device and control method thereof

Abstract

An electronic device according to an embodiment of the present disclosure may: activate a text-to-speech (TTS) function; acquire a first audio signal generated in response to playback of video content; acquire a second audio signal generated in response to the activation of the TTS function; in response to the activation of the TTS function, classify the first audio signal into a first audio object and a second audio object; determine a first weight for the first audio object and a second weight for the second audio object; and synthesize the first audio object to which the first weight has been assigned, the second audio object to which the second weight has been assigned, and the second audio signal and output same. The first audio object may be a signal of a similar type to the second audio signal compared to the second audio object, and the first weight and the second weight may be different.

Description

Electronic device and control method thereof

[0001] One embodiment disclosed in this document relates to an electronic device and a method for controlling the same.

[0002] With the advancement of electronic technology, electronic devices providing various functions are being developed. For example, technology for separating audio objects from audio data is being developed, and in particular, techniques for separating audio objects such as human voice from audio data using various deep learning technologies have recently been developed.

[0003] In addition, speech synthesis technology known as TTS (text-to-speech) is being utilized in various technological fields, including conversational personal assistants, artificial intelligence speakers, and robots, along with speech separation technology.

[0004] During the playback of video content on an electronic device, a situation may occur where voice is output with overlap due to the activation of the TTS function. For example, when a user is watching video content, if the electronic device receives a message and outputs the message content as voice via the activation of the TTS function, the voice of the content and the voice of the message may overlap, which may reduce the user's immersion in the content. Furthermore, the user may be unable to understand the content of the message output as voice due to the activation of the TTS function. Therefore, there is a need for balancing between the voice signal output from the content and the voice signal output by the TTS function. This can be referred to as audio ducking technology. Audio ducking can be used to automatically reduce the volume of one audio signal in response to another audio signal.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] An electronic device according to one embodiment of the present disclosure may provide audio ducking technology for harmoniously balancing sound between voice signals when outputting a plurality of voice signals.

[0007] An electronic device according to one embodiment of the present disclosure may include one or more processors and a memory for storing instructions. When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device may enable a text-to-speech (TTS) function, acquire a first audio signal generated in response to the playback of video content, acquire a second audio signal generated in response to the activation of the TTS function, classify the first audio signal into a first audio object and a second audio object in response to the activation of the TTS function, determine a first weight for the first audio object and a second weight for the second audio object, the first weight and the second weight are different, assign the first weight to the first audio object and assign the second weight to the second audio object, synthesize the first audio object to which the first weight was assigned, the second audio object to which the second weight was assigned, and the second audio signal into a synthesized audio signal, and output the synthesized audio signal. The first audio object and the second audio signal may correspond to signals of the same type, and the second audio object may correspond to signals of a different type from the first audio object and the second audio signal.

[0008] A control method for an electronic device according to one embodiment of the present disclosure may include: an operation of activating a text-to-speech (TTS) function; an operation of acquiring a first audio signal generated in response to the playback of video content; an operation of acquiring a second audio signal generated in response to the activation of the TTS function; an operation of classifying the first audio signal into a first audio object and a second audio object in response to the activation of the TTS function; an operation of determining a first weight for the first audio object and a second weight for the second audio object, wherein the first weight and the second weight are different, and the first weight is assigned to the first audio object and the second weight is assigned to the second audio object; an operation of synthesizing the first audio object to which the first weight is assigned, the second audio object to which the second weight is assigned, and the second audio signal into a synthesized audio signal; and an operation of outputting the synthesized audio signal. The first audio object and the second audio signal may correspond to signals of the same type, and the second audio object may correspond to signals of a different type from the first audio object and the second audio signal.

[0009] An electronic device according to one embodiment of the present disclosure can output a plurality of voice signals, separate audio objects for each voice signal, and adjust and output sound for highly relevant audio objects.

[0010] An electronic device according to one embodiment of the present disclosure can increase the user's immersion when watching video by assigning different weights to each separated audio object and outputting them.

[0011] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be determined in various ways without departing from the spirit and scope of this disclosure.

[0012] FIG. 1 is a block diagram briefly illustrating the functional configuration of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram illustrating in detail the functional configuration of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 3 is a functional block diagram according to one embodiment of the present disclosure for an electronic device to separate some of a plurality of audio signals by audio object and to synthesize and output the separated audio objects.

[0015] FIG. 4 illustrates the process of an electronic device performing object-specific audio ducking on an audio signal according to one embodiment of the present disclosure.

[0016] FIG. 5 illustrates the process of an electronic device performing object-specific audio ducking on an audio signal according to one embodiment of the present disclosure.

[0017] FIG. 6 is a schematic control flow diagram for an electronic device to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0018] FIG. 7 is a control flow diagram for an electronic device to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0019] FIG. 8 is a control flow diagram for an electronic device to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0020] FIG. 9 is a control flow diagram for an electronic device to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0021] FIG. 10 illustrates an exemplary scenario in which an electronic device performs object-specific audio ducking according to one embodiment of the present disclosure.

[0022] FIG. 11 illustrates an exemplary scenario in which an electronic device performs object-specific audio ducking according to one embodiment of the present disclosure.

[0023] FIG. 12 illustrates an exemplary scenario in which an electronic device performs object-specific audio ducking according to one embodiment of the present disclosure.

[0024] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order).

[0025] Hereinafter, in this document, "front-back direction," "left-right direction," and "up-down direction" may be used based on the illustrated drawings, and the shape and position of each component are not limited by this.

[0026] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components.

[0027] FIG. 1 is a block diagram briefly illustrating the functional configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0028] FIG. 2 is a block diagram illustrating in detail the functional configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0029] Referring to FIGS. 1 and 2, an electronic device (100) according to one embodiment of the present disclosure may include a memory (110) and a processor (120). The memory (110) may be configured to store or remember programs and / or data for controlling each component of the electronic device (100). The processor (120) may be configured to generate control signals for controlling each component of the electronic device (100) based on the programs and / or data stored in the memory (110) and information obtained from other components.

[0030] According to one embodiment, the electronic device (100) may further include, in addition to memory and a processor (120), a microphone (130), a communication interface (140), a sensor (150), a user interface (160), a speaker (170), and a display (180). However, this is exemplary, and in carrying out the present disclosure, new configurations other than the above configurations may be added or some configurations may be omitted.

[0031] According to one embodiment, at least one instruction regarding the electronic device (100) may be stored in the memory (110). For example, an operating system (OS) for operating the electronic device (100) may be stored in the memory (110). For example, various software programs or applications for the electronic device (100) to operate according to various embodiments of the present disclosure may be stored in the memory (110). At least some of the applications stored in the memory (413) may be downloaded from an external server via wireless communication. At least some of the applications stored in the memory (413) may be stored in the memory (413) from the time of shipment for the basic functions of the electronic device (100).

[0032] According to one embodiment, the memory (110) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0033] According to one embodiment, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110), and the processor (120) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (120), and data reading, recording, modification, deletion, and / or updating by the processor (120) may be performed.

[0034] According to one embodiment, the term memory (110) in this document may be used to include memory, ROM (read only memory, ROM) within the processor (120), RAM (random access memory, RAM), or a memory card (e.g., a micro SD card or a memory stick) mounted in the electronic device (100).

[0035] According to one embodiment, a plurality of TTS (text to speech) databases and a plurality of weight sets may be stored in the memory (110), and voice data, text data, and / or a plurality of parameter information according to various embodiments of the present disclosure may be stored.

[0036] According to one embodiment, the artificial intelligence model described below may be implemented in software and stored in memory (110), and the processor (120) may execute the software stored in memory (110) to control the speech recognition, speech extraction (or classification), and speech synthesis processes according to the present disclosure.

[0037] According to one embodiment, the processor (120) may be connected to one or more components included in the electronic device (100) to control the overall operation of the electronic device (100). The processor (120) may include one or more processors.

[0038] According to one embodiment, when the method according to the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by the method according to the present disclosure, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0039] According to one embodiment, the processor (120) may be implemented as a single-core processor including one core, or as one or more multi-core processors including a plurality of cores (e.g., homogeneous multi-core or heterogeneous multi-core). When one or more processors (120) are implemented as multi-core processors, each of the plurality of cores included in the multi-core processor may include memory placed inside the processor, such as cache memory or on-chip memory, and a common cache shared by the plurality of cores may be included in the multi-core processor. Additionally, each of the plurality of cores included in the multi-core processor (or some of the plurality of cores) may independently read and execute program instructions for implementing the method according to one embodiment of the present disclosure, or all (or some) of the plurality of cores may be linked together to read and execute program instructions for implementing the method according to one embodiment of the present disclosure.

[0040] According to one embodiment, when the method according to the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by the method according to the present disclosure, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0041] According to one embodiment, one or more processors (120) may refer to a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) will be described below using the expression "processor (120)."

[0042] According to one embodiment, the processor (120) can be implemented in various types. For example, the processor (120) can be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP).

[0043] In one embodiment, one or more processors (120) may be controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory (110). For example, if one or more processors (120) are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. Predefined operation rules or artificial intelligence models may be created through learning. For example, being created through learning means that a predefined operation rule or artificial intelligence model configured to perform a desired characteristic (or purpose) is created by a basic artificial intelligence model being learned using a number of learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and can perform neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0044] According to one embodiment, the microphone (130) can receive a user voice in response to a user's speech, and the received user voice can correspond to a control command for controlling the operation of the electronic device (100).

[0045] According to one embodiment, the communication interface (140) can perform communication with an external device or server. For example, the communication interface (140) may include at least one of a WiFi chip, a Bluetooth chip, a wireless communication chip, and an NFC chip. The communication interface (140) may be implemented as a communication circuit.

[0046] According to one embodiment, the communication interface (140) establishes a communication connection with an external device or server and can receive a user voice signal from the external device or server. For example, the user voice may be received not only through the microphone (130) but also through the communication interface (140).

[0047] According to one embodiment, the sensor (150) may be configured to detect various forms of information. For example, the sensor (150) may include a touch sensor that detects a user's touch, and the sensor (150) may include various sensors such as a motion sensor, a temperature sensor, a humidity sensor, an illuminance sensor, etc.

[0048] According to one embodiment, the user interface (160) can receive user interactions to control the overall operation of the electronic device (100). For example, the user interface (160) may be implemented with a configuration such as a camera, a microphone (130), a remote control signal receiver, etc. For example, the user interface (160) may be implemented in a form included in the display (180) as a touch screen.

[0049] According to one embodiment, the speaker (170) can output voice. For example, the processor (120) can control the speaker (170) to output voice. For example, the speaker (170) can output output voice corresponding to the acquired text. For example, the speaker (170) can output various notification sounds or voice messages, in addition to various audio data processed by the processor (120).

[0050] According to one embodiment, the display (180) can output an image. And, the processor (120) can control the display (180) to output an image. For example, the processor (120) can control the display (180) to display text information corresponding to the output voice according to the present disclosure.

[0051] Although not illustrated, the electronic device (100) may further include a camera. For example, the camera may be configured to capture still images or video. For example, the camera may capture a still image at a specific point in time. For example, the camera may capture still images in succession.

[0052] According to one embodiment, when a voice signal is output, the electronic device (100) can separate the voice signal into a plurality of audio object signals. For example, the electronic device (100) can separate the voice signal into an object indicating a voice component and an object indicating a background music component. For example, the electronic device (100) can separate the voice signal based on an artificial intelligence learning model.

[0053] According to one embodiment, while voice is output according to the playback of video (or audio) content, an electronic device (100) may simultaneously output additional voice according to the activation of a TTS function. For example, while a voice signal according to the playback of video content is being output, the electronic device (100) may output a voice signal by means of a TTS function for a message received by the electronic device (100). For example, when the electronic device (100) plays video content containing foreign language (e.g., English) dialogue, Korean subtitles corresponding to the foreign language dialogue are displayed on a screen (e.g., display (180)), and the Korean subtitles displayed on the display (180) may be output together as voice by means of a TTS function.

[0054] According to one embodiment, when a plurality of voice signals are output simultaneously by an electronic device (100), the output plurality of voice signals may overlap with each other, and this may act as a factor that interferes with the user's viewing immersion. Below, a control method will be described in which, in a situation where a plurality of voice signals are output simultaneously by an electronic device (100), some of the voice signals are separated into a plurality of audio objects, different weights are assigned to each separated audio object, and then the two are synthesized and output.

[0055] FIG. 3 is a functional block diagram according to one embodiment of the present disclosure, for an electronic device (e.g., the electronic device (100) of FIG. 1) to separate some of a plurality of audio signals by audio object and to synthesize and output the separated audio objects.

[0056] The components illustrated in FIG. 3 are illustrated for the purpose of explaining a control operation in which an electronic device (100) separates some of a plurality of audio signals by object, assigns weights to the separated objects, and then synthesizes and outputs them. The components illustrated in FIG. 3 may be implemented by a processor (e.g., processor (120) of FIG. 1 and 2), memory (e.g., memory (110) of FIG. 1 and 2), speaker (e.g., speaker (170) of FIG. 2), and / or display (e.g., display (180) of FIG. 2).

[0057] The embodiment of FIG. 3 can be optionally combined with the embodiments of FIG. 1 and FIG. 2.

[0058] Referring to FIG. 3, a situation is assumed in which a screen is displayed through a display (180) by video content in an electronic device (100), and sound is output through a speaker (170). For example, an audio signal input to the electronic device (100) by video content will be referred to as a first audio signal. The first audio signal may be implemented by synthesizing one or more audio objects, such as, for example, an object corresponding to background sound, an object corresponding to a person's voice, and / or an object corresponding to sound effects.

[0059] For example, it will be assumed that the audio output from the video content includes foreign language dialogue, and Korean subtitles corresponding to the foreign language dialogue are displayed on the display (180). For example, the electronic device (100) can output Korean subtitles as voice by activating the TTS function.

[0060] For example, while video content is being output, the electronic device (100) can output a notification or received message to the display (180). For example, the electronic device (100) can output the notification or message as voice by enabling the TTS function.

[0061] According to one embodiment, the electronic device (100) can output Korean subtitles or notifications (or messages) as voice when the TTS function is activated, and the audio signal input to the electronic device (100) will be referred to as the second audio signal.

[0062] Hereinafter, an electronic device (100) according to the present disclosure will describe a control process in which, when a first audio signal and a second audio signal are input simultaneously, the first audio signal is separated (or classified) into one or more audio objects, and after assigning different weights to the separated audio objects, the second audio signal is output.

[0063] According to one embodiment, the electronic device (100) may include an audio signal input (310), an audio signal processor (320), an audio signal output (330), and a TTS generator (340).

[0064] According to one embodiment, the audio signal input unit (310) may be configured to acquire an audio signal. For example, the audio signal input unit (310) may receive a first audio signal generated by video content playback.

[0065] According to one embodiment, the TTS generation unit (340) can generate a second audio signal by activating the TTS function. For example, the TTS generation unit (340) can generate a second audio signal by activating the subtitle reading function, or generate a second audio signal for outputting a notification displayed on the display (180) of the electronic device (100) as voice.

[0066] According to one embodiment, the audio signal processing unit (320) may be configured to process an audio signal. For example, the audio signal processing unit (320) may be configured to process (or process) a first audio signal obtained by the audio signal input unit (310), or to process (or process) a second audio signal generated by the TTS generation unit (340). For example, the audio signal processing unit (320) may include an audio object divider (321), an audio signal analyzer, and a gain determiner.

[0067] According to one embodiment, the audio object separation unit (321) may be configured to classify an audio signal into one or more audio objects according to the components of the audio signal and to separate the classified audio objects. For example, the audio object separation unit (321) may classify and / or separate a first audio signal into one or more audio objects.

[0068] According to one embodiment, when the first audio signal includes background music, voice, and sound effects, the audio object separation unit (321) classifies the first audio signal into a background music object, a voice object, and a sound effect object, respectively, and can separate each object.

[0069] According to one embodiment, the audio object separation unit (321) can classify and separate the first audio signal using a machine learning model (e.g., an artificial intelligence model). For example, the audio object separation unit (321) can classify the first audio signal by object based on an artificial intelligence model included in the processor (120) and separate each classified object. The following description outlines the operation of the audio object separation unit (321) classifying or separating the first audio signal by object using a machine learning model. However, the operations described below are merely exemplary, and the audio object separation unit (321) can separate the first audio signal in various ways.

[0070] According to one embodiment, the audio object separation unit (321) can perform audio preprocessing on the first audio signal. For example, the audio object separation unit (321) can sequentially convert a preset number of time-axis audio data among the audio signals into the frequency domain.

[0071] According to one embodiment, the audio object separation unit (321) can convert the audio data into the frequency domain based on a fast Fourier transform (FFT). However, it is not limited thereto, and any method capable of converting audio data into the frequency domain is acceptable.

[0072] According to one embodiment, the audio object separation unit (321) can obtain encoded data by encoding audio data converted into the frequency domain. For example, the audio object separation unit (321) can obtain encoded data by inputting audio data converted into the frequency domain into a first layer of a neural network model.

[0073] According to one embodiment, the audio object separation unit (321) can obtain query data, key data, and value data from the encoding data. For example, the audio object separation unit (321) can obtain query data, key data, and value data by inputting the encoding data into a second layer of a neural network model.

[0074] According to one embodiment, the audio object separation unit (321) can obtain attention weights and context data based on query data, key data, and value data. For example, the audio object separation unit (321) can obtain scored query data by inputting query data into a third layer of a neural network model, obtain attention weights by performing an element-wise product of the scored query data and key data, and obtain context data by performing an element-wise product of the attention weights and value data.

[0075] According to one embodiment, the audio object separation unit (321) can obtain an object separation mask based on context data and query data. For example, the processor (120) can obtain an object separation mask by inputting context data and query data into the fourth layer of a neural network model.

[0076] According to one embodiment, the audio object separation unit (321) can separate the first audio signal into individual audio objects based on an acquired object separation mask. For example, the audio object separation unit (321) can separate each audio object by applying the acquired object separation mask to the original spectrogram.

[0077] According to one embodiment, the audio object separation unit (321) can separate a first audio signal into a first audio object and a second audio object. For example, the first audio object may correspond to a voice component included in the first audio signal, and the second audio object may correspond to a component of the first audio signal excluding the first audio object. For example, the second audio object may include background sound, instrumental sound, and / or sound effect. The first audio object may correspond to a voice signal, and the second audio object may correspond to a signal excluding the voice signal. For convenience of explanation, it has been assumed in this document that the first audio signal is separated into two objects, a first audio object and a second audio object; however, this is merely illustrative, and the first audio signal may be separated into three or more objects.

[0078] According to one embodiment, the audio signal analysis unit (323) can analyze the first audio signal and / or the second audio signal. For example, the audio signal analysis unit (323) may be configured to analyze the magnitude of the first audio signal and / or the second audio signal, or to analyze the playback timing.

[0079] For example, the audio signal analysis unit (323) can analyze the size of each object of the first audio signal separated by the audio object separation unit (321).

[0080] For example, the audio signal analysis unit (323) can analyze the playback time of the second audio signal. For example, the audio signal analysis unit (323) can identify the time when the playback of the second audio signal begins and the time when the playback of the second audio signal ends.

[0081] According to one embodiment, the weight determining unit (325) can determine a weight for the first audio signal and / or the second audio signal. For example, the weight determining unit (325) can determine a weight to be assigned to each audio object separated by each object for the first audio signal.

[0082] For example, the weight determination unit (325) may determine different weights for each audio object. For example, the weight determination unit (325) may determine different weights for the first audio object and the second audio object. For example, it may decide to assign a first weight to the first audio object. For example, it may decide to assign a second weight to the second audio object. For example, the first weight and the second weight may be different from each other.

[0083] According to one embodiment, the weight determination unit (325) may determine different weights to be assigned to the first audio object and the second audio object, respectively, by considering overlap with the second audio signal. For example, the weight determination unit (325) may determine different weights to be assigned to the first audio object and the second audio object, respectively, by considering the user's immersion when overlapped with the second audio signal.

[0084] According to one embodiment, the weight determining unit (325) may assign a relatively small weight to a first audio object similar to a second audio signal. For example, the first weight assigned by the weight determining unit (325) may be relatively smaller than the second weight.

[0085] According to one embodiment, the weight determination unit (325) may assign a third weight to the second audio signal. For example, the weight determination unit (325) may assign a third weight to the second audio signal by considering the size of the first audio object. For example, the third weight may be determined in correspondence with the size of the first audio object. For example, the third weight in the section where the size of the first audio object is large may be greater than the third weight in the section where the size of the first audio object is small.

[0086] According to one embodiment, each audio signal of the audio signal output unit (330) can be synthesized and the synthesized audio signal can be output. For example, by the audio signal output unit (330), a first audio object with a first weight, a second audio object with a second weight, and a second audio signal with a third weight can be synthesized, and the synthesized audio signal can be output.

[0087] According to one embodiment, the electronic device (100) can improve the immersion of video content by separating a first audio signal by object and assigning a weight to each audio signal by considering the association with a second audio signal for the separated audio object. Hereinafter, the technology of the electronic device (100) of the present disclosure separating an audio signal by object and assigning a weight to each object will be referred to as "object-specific audio ducking" technology.

[0088] FIG. 4 illustrates a process in which an electronic device (e.g., the electronic device (100) of FIG. 1) performs object-specific audio ducking on an audio signal according to one embodiment of the present disclosure.

[0089] FIG. 5 illustrates the process of an electronic device (100) performing object-specific audio ducking on an audio signal according to one embodiment of the present disclosure.

[0090] FIGS. 3 and FIGS. 4 can be understood as illustrating an example of an operation in which an electronic device (100) assigns weights to a first audio signal and a second audio signal on an object-by-object basis and outputs them.

[0091] Referring to FIGS. 3 and 4, the electronic device (100) can separate a first audio signal by object. For example, the first audio signal can be separated into a first audio object and a second audio object by an audio object separation unit (321) (e.g., the audio object separation unit (321) of FIG. 3). For example, the first audio object may correspond to a voice signal, and the second audio object may correspond to an audio signal excluding the voice signal.

[0092] According to one embodiment, the electronic device (100) can acquire a second audio signal. For example, the electronic device (100) can acquire the second audio signal by a TTS generator (340) (e.g., the TTS generator (340) of FIG. 3) in response to the activation of the TTS function. For example, the TTS generator (340) can acquire the second audio signal when a character is displayed on a display (e.g., the display (180) of FIG. 2) by the activation of the TTS function. The second audio signal may correspond to a voice signal of the TTS function.

[0093] According to one embodiment, a first audio object and a second audio object may each be assigned a weight. For example, a weight determining unit (325) (e.g., the weight determining unit (325) of FIG. 3) may assign a weight to each of the first audio object and the second audio object. For example, the first audio object may be assigned a first weight by the weight determining unit (325). For example, the second audio object may be assigned a second weight by the weight determining unit (325).

[0094] According to one embodiment, the first weight and the second weight may be different from each other. For example, the first weight and the second weight may be determined by considering the second audio signal.

[0095] According to one embodiment, the first weight may be determined to be a relatively smaller value with respect to the second weight.

[0096] According to one embodiment, the first weight and the second weight may be determined by considering the interval in which the second audio signal exists. For example, an audio signal analysis unit (e.g., the audio signal analysis unit (323) of FIG. 3) may identify the start time and the end time of the second audio signal. For example, the audio signal analysis unit (323) may identify the start identifier and the end identifier (351) included in the second audio signal, and identify the start time and the end time of the second audio signal by the identifiers (351).

[0097] According to one embodiment, the electronic device (100) may assign a first weight and a second weight to each of the first audio object and the second audio object during the interval in which the second audio signal exists. For example, the electronic device (100) may assign a first weight and a second weight to each of the first audio object and the second audio object only when the TTS sound is played.

[0098] According to one embodiment, the electronic device (100) may assign a first weight and a second weight to each of the first audio object and the second audio object, regardless of whether the second audio signal is present. For example, the electronic device (100) may adjust the size of the first audio object and the second audio object even in sections where the TTS sound is not present.

[0099] According to one embodiment, the electronic device (100) may assign a weight to the second audio signal. For example, the weight determining unit (325) may assign a third weight to the second audio signal.

[0100] Below, a control flow diagram for a method for an electronic device (100) to perform an object-specific audio ducking technique will be described.

[0101] FIG. 6 is a schematic control flow diagram for an electronic device (e.g., the electronic device (100) of FIG. 1) to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0102] The embodiment of FIG. 6 can be optionally combined with the embodiments of FIG. 1 to 5.

[0103] Referring to FIG. 6, the electronic device (100) can activate the TTS function in step 610. For example, the electronic device (100) can activate the TTS function by receiving input from a user. For example, the electronic device (100) can activate the TTS function when a predetermined event occurs. The predetermined event may be, for example, an event set by a user, or an event stored in advance in memory (e.g., memory (110) of FIG. 1).

[0104] For example, the electronic device (100) can activate the TTS function by an event in which subtitles are output when playing video content. For example, the electronic device (100) can activate the TTS function when displaying a notification on the display (180). For example, the electronic device (100) can activate the TTS function when running a specific application.

[0105] According to one embodiment, the electronic device (100) can acquire an audio signal in step 620. For example, the electronic device (100) can acquire a first audio signal and a second audio signal.

[0106] According to one embodiment, the electronic device (100) can classify the audio signal in step 630. For example, the electronic device (100) can classify the first audio signal by object. For example, the electronic device (100) can classify the first audio signal into a first audio object corresponding to a voice signal and a second audio object corresponding to the first audio signal excluding the first audio object.

[0107] According to one embodiment, the electronic device (100) may assign weights to the classified audio signals in step 640. For example, the electronic device (100) may assign weights to the first audio object, the second audio object, and the second audio signal, respectively, which are included in the first audio signal. For example, the electronic device (100) may assign a first weight to the first audio object. For example, the electronic device (100) may assign a second weight to the second audio object. For example, the electronic device (100) may assign a third weight to the second audio signal.

[0108] According to one embodiment, the electronic device (100) can synthesize an audio signal in step 650. For example, the electronic device (100) can synthesize a first audio object with a first weight, a second audio object with a second weight, and a second audio signal with a third weight.

[0109] According to one embodiment, the electronic device (100) can output a synthesized audio signal in step 660. For example, the electronic device (100) can output the synthesized audio signal through an audio signal output unit (e.g., the audio signal output unit (330) of FIG. 3).

[0110] FIG. 7 is a control flow diagram for an electronic device (e.g., the electronic device (100) of FIG. 1) to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0111] FIG. 8 is a control flow diagram for an electronic device (100) to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0112] FIG. 9 is a control flow diagram for an electronic device (100) to perform an object-specific audio ducking technique according to one embodiment of the present disclosure.

[0113] The control flow diagrams illustrated in FIGS. 7 to 9, respectively, are illustrated to explain the differences in detailed operations when the electronic device (100) performs an object-specific audio ducking technique. For example, FIG. 7 illustrates a control flow diagram in which the electronic device (100) does not assign a separate weight to the second audio signal (e.g., voice generated by a TTS function), and FIGS. 8 and 9 illustrate a control flow diagram in which the electronic device (100) assigns a separate weight to the second audio signal. However, FIGS. 8 and 9 are illustrated differently depending on whether the section in which the second audio signal is played is considered when assigning weights to each of the first audio object and the second audio object included in the first audio signal. Below, the differences in detailed operations described above will be explained in detail.

[0114] Some of the operations shown in FIGS. 7 to 9 may be omitted, or the same operations may be repeated, and the order of operations may be changed as needed.

[0115] The embodiments of FIGS. 7 to 9 can be optionally combined with the embodiment of FIG. 6.

[0116] Referring to FIG. 7, the electronic device (100) can enable the TTS function in operation 710. For example, operation 710 may correspond to operation 610 of FIG. 6.

[0117] According to one embodiment, the electronic device (100) can acquire a first audio signal and a second audio signal in operation 720. For example, operation 720 may correspond to operation 620 of FIG. 6.

[0118] According to one embodiment, the electronic device (100) can classify the first audio signal obtained in operation 730 into a first audio object and a second audio object. For example, operation 730 may correspond to operation 630 of FIG. 6.

[0119] According to one embodiment, the electronic device (100) can determine a weight for each of the first audio object and the second audio object in operation 740. For example, operation 740 may correspond to operation 640 of FIG. 6.

[0120] According to one embodiment, the electronic device (100) may determine different weights for a first audio object and a second audio object. For example, the first weight may be relatively smaller than the second weight. As a result, the first audio object corresponding to the voice signal may be output at a lower volume than the second audio object corresponding to the background sound or sound effect.

[0121] According to one embodiment, the electronic device (100) may assign a first weight to a first audio object in operation 750. The electronic device (100) may assign a second weight to a second audio object.

[0122] According to one embodiment, the electronic device (100) can synthesize a first audio object, a second audio object, and a second audio signal in operation 760. For example, operation 760 may correspond to operation 650 of FIG. 6.

[0123] According to one embodiment, the electronic device (100) can output an audio signal synthesized in operation 770. For example, operation 770 may correspond to operation 660 of FIG. 6.

[0124] Referring to FIG. 8, the electronic device (100) can enable the TTS function in operation 810. For example, operation 810 may correspond to operation 610 of FIG. 6 and operation 710 of FIG. 7.

[0125] According to one embodiment, the electronic device (100) can obtain a first audio signal and a second audio signal in operation 820. For example, operation 820 may correspond to operation 620 of FIG. 6 and operation 720 of FIG. 7.

[0126] According to one embodiment, the electronic device (100) can classify a first audio signal obtained in operation 830 into a first audio object and a second audio object. For example, operation 830 may correspond to operation 630 of FIG. 6 and operation 730 of FIG. 7.

[0127] According to one embodiment, the electronic device (100) can determine a weight for each of the first audio object and the second audio object in operation 840. For example, operation 840 may correspond to operation 640 of FIG. 6 and operation 740 of FIG. 7.

[0128] According to one embodiment, the electronic device (100) may determine a weight for a second audio signal based on the strength of a first audio object in operation 850. For example, the electronic device (100) may consider the strength of the first audio object when determining a third weight for the second audio signal. Here, the signal strength of the first audio object may refer to the signal strength of the first audio object in a state where the first weight is not applied (assigned). For example, the electronic device (100) may determine a third weight in correspondence with the signal strength (e.g., amplitude) of the first audio object corresponding to the voice signal included in the first audio signal. For example, the signal strength of the first audio object may have a positive correlation with the third weight.

[0129] According to one embodiment, the electronic device (100) may assign weights to a first audio object, a second audio object, and a second audio signal in operation 860. For example, operation 860 may correspond to operation 640 of FIG. 6 and operation 750 of FIG. 7.

[0130] According to one embodiment, the electronic device (100) may assign a first weight to a first audio object, assign a second weight to a second audio object, and assign a third weight to a second audio signal.

[0131] According to one embodiment, the electronic device (100) can synthesize a first audio object, a second audio object, and a second audio signal in operation 870. For example, operation 870 may correspond to operation 650 of FIG. 6 and operation 760 of FIG. 7.

[0132] According to one embodiment, the electronic device (100) can output an audio signal synthesized in operation 880. For example, operation 880 may correspond to operation 660 of FIG. 6 and operation 770 of FIG. 7.

[0133] Referring to FIG. 9, the electronic device (100) can enable the TTS function in operation 910. For example, operation 910 may correspond to operation 610 of FIG. 6, operation 710 of FIG. 7, and operation 810 of FIG. 8.

[0134] According to one embodiment, the electronic device (100) can acquire a first audio signal and a second audio signal in operation 920. For example, operation 920 may correspond to operation 620 of FIG. 6, operation 720 of FIG. 7, and operation 820 of FIG. 8.

[0135] According to one embodiment, the electronic device (100) can classify a first audio signal obtained in operation 930 into a first audio object and a second audio object. For example, operation 930 may correspond to operation 630 of FIG. 6, operation 730 of FIG. 7, and operation 830 of FIG. 8.

[0136] According to one embodiment, the electronic device (100) can determine a weight for each of the first audio object and the second audio object in operation 940. For example, operation 940 may correspond to operation 740 of FIG. 7 and operation 840 of FIG. 8.

[0137] According to one embodiment, the electronic device (100) may, in operation 950, apply a weight to the second audio signal based on the intensity of the first audio object. For example, operation 950 may correspond to operation 640 of FIG. 6, operation 740 of FIG. 7, and operation 840 of FIG. 8.

[0138] According to one embodiment, the electronic device (100) can determine whether a second audio signal is being played in operation 961.

[0139] According to one embodiment, when the electronic device (100) determines that the second audio signal is being played, in operation 963, the playback start time and playback end time of the second audio signal can be obtained. For example, the electronic device (100) can obtain the playback start time and playback end time of the second audio signal by using an audio signal analysis unit (e.g., the audio signal analysis unit (323) of FIG. 3) to identify a playback start identifier and a playback end identifier (e.g., the start / end identifier (351) of FIG. 4) for the second audio signal.

[0140] According to one embodiment, the electronic device (100) can assign weights to each of the first audio object and the second audio object during the period from the start of playback of the second audio signal to the end of playback in operation 970.

[0141] According to one embodiment, the electronic device (100) may apply a weight to the second audio signal in operation 980. For example, the electronic device (100) may apply a third weight to the second audio signal in correspondence with the signal strength of the first audio object. For example, operation 980 may correspond to operation 850 of FIG. 8.

[0142] According to one embodiment, if the electronic device (100) determines that the second audio signal is not being played, it may assign a weight to the second audio signal in operation 965. For example, if the electronic device (100) determines that the second audio signal is not being played, it may omit operations 963 and 970.

[0143] According to one embodiment, the electronic device (100) can synthesize a first audio object, a second audio object, and a second audio signal in operation 990. For example, operation 990 may correspond to operation 650 of FIG. 6, operation 760 of FIG. 7, and operation 870 of FIG. 8.

[0144] According to one embodiment, the electronic device (100) can output an audio signal synthesized in operation 1000. For example, operation 1000 may correspond to operation 660 of FIG. 6, operation 770 of FIG. 7, and operation 880 of FIG. 8.

[0145] FIG. 10 illustrates an exemplary scenario in which an electronic device (100) (e.g., the electronic device (100) of FIG. 1) performs object-specific audio ducking according to one embodiment of the present disclosure.

[0146] FIG. 11 illustrates an exemplary scenario in which an electronic device (100) performs object-specific audio ducking according to one embodiment of the present disclosure.

[0147] FIGS. 10 and FIGS. 11 illustrate an exemplary embodiment in which object-specific audio ducking is performed while video content is being played on a display of an electronic device (100) (e.g., the display (180) of FIG. 2).

[0148] The embodiments of FIGS. 10 and 11 can be optionally combined with the embodiments of FIGS. 1 to 9.

[0149] Referring to FIGS. 10 and 11, the video content played on the display (180) may be multimedia content containing various types of information. For example, the video content may include audio, video, and / or animation.

[0150] According to one embodiment, the audio signal may include a voice component (1010; 1110) based on dialogue of a character in the content and a background sound component (1020; 1120). The audio signal may correspond to the first audio signal described above.

[0151] According to one embodiment, as the voice component (1010; 1110) is a foreign language and the subtitle function is activated, the electronic device (100) can display subtitles corresponding to the voice component (1010; 1110) as text (1030; 1130) on the display (180).

[0152] According to one embodiment, the electronic device (100) can generate a TTS voice corresponding to the text (1030; 1130) in response to the activation of the TTS function. The TTS voice may correspond to the aforementioned second audio signal. For example, the second audio signal may correspond to the voice signal of the TTS function.

[0153] According to one embodiment, when a voice component (1010; 1110) included in the first audio signal and a TTS voice are superimposed and output, the viewing immersion may be reduced. Accordingly, the electronic device (100) can perform object-specific audio ducking.

[0154] According to one embodiment, the electronic device (100) can separate a first audio object corresponding to a voice component (1010; 1110) and a second audio object corresponding to a background sound component (1020; 1120) for a first audio signal.

[0155] According to one embodiment, the electronic device (100) may assign weights to a first audio object and a second audio object. For example, the electronic device (100) may assign a first weight to the first audio object and a second weight to the second audio object.

[0156] According to one embodiment, the electronic device (100) may determine a first weight and a second weight by considering the correlation with a second audio signal. For example, the electronic device (100) may determine the first weight to be a smaller value than the second weight in order to output a smaller signal magnitude of the first audio object, which is a more disruptive element that interferes with the user's immersion when superimposed with TTS voice.

[0157] According to one embodiment, the electronic device (100) may obtain the start time and end time of the second audio signal and assign a first weight and a second weight to the section where the second audio signal is output. However, if the first weight and the second weight are assigned only to the section where the second audio signal is output, the magnitude variation of the voice component output corresponding to whether or not the second audio signal is output may not be consistent. For example, depending on whether the TTS voice is output, a phenomenon may occur where the voice component is output at a low or high level. Accordingly, when the second audio signal is generated by a voice component (e.g., dialogue), the electronic device (100) may assign a first weight and a second weight by considering the start time and end time when the second audio signal is generated by the voice component.

[0158] According to one embodiment, the electronic device (100) may assign a third weight to a second audio signal. For example, the electronic device (100) may determine the third weight by considering the signal magnitude of the first audio object.

[0159] For example, the electronic device (100) may determine a third weight for the second audio signal to be small for a section where the voice component (1010; 1110) is output small (e.g., when a person in the video whispers or mutters). For example, the electronic device (100) may determine a third weight for the second audio signal to be large for a section where the voice component (1010; 1110) is output large (e.g., when a person in the video shouts or gets angry).

[0160] Referring to FIG. 10, a background sound component (1020) can be output from time t1, a voice component (1010) can be output from time t2 after a delay of d1 has elapsed from time t1, and a TTS voice generated by text (1030) can be output from time t2' after a delay of d2 has elapsed from time t2.

[0161] Referring to FIG. 11, a background sound component (1120) can be output from time t3, a voice component (1110) can be output from time t4 after a delay of d3 has elapsed from time t3, and a TTS voice generated by text (1130) can be output from time t4' after a delay of d4 has elapsed from time t4.

[0162] According to one embodiment, the electronic device (100) separates the first audio signal by object and assigns different first and second weights to each object considering the second audio signal, and assigns a third weight to the second audio signal considering the magnitude of the voice component (1010; 11110) included in the first audio signal, thereby increasing the recognition of the TTS voice and providing an immersive viewing environment.

[0163] FIG. 12 illustrates an exemplary scenario in which an electronic device (100) performs object-specific audio ducking according to one embodiment of the present disclosure.

[0164] FIG. 12 illustrates an exemplary embodiment in which object-specific audio ducking is performed while video content is being played on a display (e.g., the display (180) of FIG. 2) of an electronic device (100). For example, it illustrates an embodiment in which information (e.g., incoming message or notification) displayed on the display (180) is output as a TTS voice while video content is being played.

[0165] Referring to FIG. 12, the video content played on the display (180) may be multimedia content containing various types of information. For example, the video content may include audio, video, and / or animation. For example, the video content may be video content in which a band consisting of a singer and musicians plays instruments and sings.

[0166] According to one embodiment, the audio signal may include a voice component and a background sound component. The audio signal may correspond to the first audio signal described above. The first audio signal may include, for example, a voice component corresponding to a singer's vocals and a background sound corresponding to an instrumental sound.

[0167] According to one embodiment, the electronic device (100) may output information displayed on the display (180) as a TTS voice in response to the activation of the TTS function. For example, the received information may include a message received by the electronic device (100) from an external device or information generated by the electronic device (100). The TTS voice may correspond to the aforementioned second audio signal.

[0168] According to one embodiment, when a voice component included in a first audio signal and a TTS voice are superimposed and output, the sense of viewing immersion may be reduced. Accordingly, the electronic device (100) can perform object-specific audio ducking.

[0169] According to one embodiment, the electronic device (100) can separate a first audio signal into a first audio object corresponding to a voice component and a second audio object corresponding to a background sound component.

[0170] According to one embodiment, the electronic device (100) may assign weights to a first audio object and a second audio object. For example, the electronic device (100) may assign a first weight to the first audio object and a second weight to the second audio object.

[0171] According to one embodiment, the electronic device (100) may determine a first weight and a second weight by considering the correlation with the second audio signal. For example, the electronic device (100) may determine the first weight to be smaller than the second weight in order to output a smaller signal magnitude of the first audio object, which is a more disruptive element that interferes with the user's immersion when superimposed with TTS voice. However, the electronic device (100) may determine the first weight and the second weight by considering the correlation between the first audio object included in the first audio signal and the second audio signal. For example, the electronic device (100) may determine the first weight and the second weight to be substantially the same by considering that the first audio signal is music composed of vocal voice and instrumental sound.

[0172] According to one embodiment, the electronic device (100) may obtain the start time and end time of the second audio signal and assign a first weight and a second weight to the section where the second audio signal is output. However, if the first weight and the second weight are assigned only to the section where the second audio signal is output, the magnitude variation of the voice component output corresponding to whether the second audio signal is output may not be consistent. For example, depending on whether the TTS voice is output, a phenomenon may occur where the voice component is output at a low or high level. Accordingly, the electronic device (100) may assign a first weight and a second weight by considering the start time and end time when the second audio signal is generated by the voice component.

[0173] According to one embodiment, the electronic device (100) may assign a third weight to a second audio signal. For example, the electronic device (100) may determine the third weight by considering the signal magnitude of the first audio object.

[0174] According to one embodiment, the electronic device (100) may determine a third weight by considering the association between a first audio object included in a first audio signal and a second audio signal. The electronic device (100) may determine a third weight by considering that the first audio signal is music composed of vocal voice and instrumental sound.

[0175] Referring to FIG. 12, a background sound component can be output from time t5, a voice component can be output from time t6 after a delay of d5 has elapsed from time t5, and a TTS voice generated by a received message (1220) can be output from time t6' after a delay of d6 has elapsed from time t6.

[0176] According to one embodiment, the electronic device (100) separates the first audio signal by object and assigns different first and second weights considering the second audio signal to each object, and assigns a third weight considering the voice component included in the first audio signal to the second audio signal, thereby increasing the recognition of the TTS voice and providing an immersive viewing environment.

[0177] An electronic device (100) according to one embodiment of the present disclosure may provide an object-specific audio ducking technique that separates audio signals by object and determines weights by object considering TTS voice.

[0178] An electronic device (100) according to one embodiment of the present disclosure can output a plurality of voice signals, separate audio objects for each voice signal, and adjust and output sound for audio objects with high relevance.

[0179] An electronic device (100) according to one embodiment of the present disclosure can increase the user's immersion when watching video by assigning different weights to each separated audio object and outputting them.

[0180] An electronic device (100) according to one embodiment of the present disclosure can improve the level of recognition for a TTS signal by performing object-specific audio ducking in consideration of the TTS voice.

[0181] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0182] An electronic device according to one embodiment of the present disclosure (e.g., the electronic device (100) of FIG. 1) may include one or more processors (120) and a configured memory (110) for storing instructions. When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) enables a text-to-speech (TTS) function (610; 710; 810; 910; 910), acquires a first audio signal generated in response to video content being played (620; 720; 820; 920), acquires a second audio signal generated in response to the activation of the TTS function (620; 720; 820; 920), classifies the first audio signal into a first audio object and a second audio object in response to the activation of the TTS function (630; 730; 830; 930), determines a first weight for the first audio object and a second weight for the second audio object (650; 740, 750; 840; 940), the first weight and the second weight are different, and the first weight is assigned to the first audio object, the second weight is assigned to the second audio object, the first audio object to which the first weight is assigned, the second audio object to which the second weight is assigned, and the second audio signal are combined into a combined audio signal, and the combined audio signal is output (660, 670; 760, 770; 870, 880; 990, 1000). The first audio object may be a signal of a type similar to the second audio signal compared to the second audio object. For example, the first audio object and the second audio signal may correspond to signals of the same type.The second audio object may correspond to a signal of a different type from the first audio object and the second audio signal. For example, the first audio object and the second audio signal may correspond to voice signals, and the second audio object may correspond to a signal excluding voice signals.

[0183] In an electronic device (100) according to one embodiment of the present disclosure, the first audio object may correspond to a voice signal among the audio signals.

[0184] In an electronic device (100) according to one embodiment of the present disclosure, the second audio object may correspond to a signal among the audio signals excluding the first audio object.

[0185] In an electronic device (100) according to one embodiment of the present disclosure, the first weight may be smaller than the second weight.

[0186] In an electronic device (100) according to one embodiment of the present disclosure, the instructions, when executed individually or collectively by one or more processors (120), may cause the electronic device (100) to determine a third weight for the second audio signal (640; 750; 850; 950).

[0187] In an electronic device (100) according to one embodiment of the present disclosure, when the instructions are executed individually or collectively by one or more processors (120), the electronic device (100) may be able to determine the third weight based on the signal strength of the first audio object (850; 950).

[0188] In an electronic device (100) according to one embodiment of the present disclosure, the third weight may have a positive correlation with the signal strength of the first audio object.

[0189] In an electronic device (100) according to one embodiment of the present disclosure, when the instructions are executed individually or collectively by one or more processors (120), the electronic device (100) may obtain a first time point at which the output of the second audio signal is initiated and a second time point at which the output of the second audio signal is terminated (963), and may assign the first weight and the second weight to each of the first audio object and the second audio object that are output during the first time point and the second time point (970).

[0190] In an electronic device (100) according to one embodiment of the present disclosure, the instructions, when executed individually or collectively by one or more processors (120), may cause the electronic device (100) to assign the second weight (980) to the second audio object output during the first time point and the second time point.

[0191] In an electronic device (100) according to one embodiment of the present disclosure, the memory (110) may be configured to store a neural network model obtained by learning the relationship between a plurality of sample audio signals and the plurality of sample audio objects.

[0192] A control method for an electronic device (100) according to one embodiment of the present disclosure comprises: an operation of activating a text-to-speech (TTS) function (610; 710; 810; 910); an operation of acquiring a first audio signal generated in response to the playback of video content (620; 720; 820; 920); an operation of acquiring a second audio signal generated in response to the activation of the TTS function (620; 720; 820; 920); an operation of classifying the first audio signal into a first audio object and a second audio object in response to the activation of the TTS function (630; 730; 830; 930); an operation of determining a first weight for the first audio object and a second weight for the second audio object (650; 740, 750; 840; 940); wherein the first weight and the second weight are different, and the first It may include the first audio object to which weights are assigned, the second audio object to which second weights are assigned, and the operation of synthesizing the second audio signal into a synthesized audio signal and outputting the synthesized audio signal (660, 670; 760, 770; 870, 880; 990, 1000). The first audio object may be a signal of a type similar to the second audio signal compared to the second audio object.

[0193] In a control method for an electronic device (100) according to one embodiment of the present disclosure, the first audio object may correspond to a voice signal among the audio signals.

[0194] In a control method for an electronic device (100) according to one embodiment of the present disclosure, the second audio object may correspond to a signal among the audio signals excluding the first audio object.

[0195] In a control method of an electronic device (100) according to one embodiment of the present disclosure, the first weight may be smaller than the second weight.

[0196] A control method of an electronic device (100) according to one embodiment of the present disclosure may include an operation (640; 750; 850; 950) for determining a third weight for the second audio signal.

[0197] A control method for an electronic device (100) according to one embodiment of the present disclosure may include an operation (850; 950) of determining the third weight based on the signal strength of the first audio object.

[0198] In a control method for an electronic device (100) according to one embodiment of the present disclosure, the third weight may have a positive correlation with the signal strength of the first audio object.

[0199] A control method for an electronic device (100) according to one embodiment of the present disclosure may include an operation (963) of obtaining a first time point at which the output of the second audio signal is initiated and a second time point at which the output of the second audio signal is terminated, and an operation (970) of applying the first weight to the first audio object output during the first time point and the second time point.

[0200] A control method for an electronic device (100) according to one embodiment of the present disclosure may include an operation (980) of applying the second weight to the second audio object output during the first time point and the second time point.

[0201] In a control method of an electronic device (100) according to one embodiment of the present disclosure, the operation (640) of classifying the first audio signal may include the operation of classifying the first audio signal into the first audio object and the second audio object from a neural network model obtained by learning the relationship between a plurality of sample audio signals and the plurality of sample audio objects.

Claims

1. In an electronic device (100), One or more processors (120); and It includes a memory (110) for storing instructions, When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) causes: Enable TTS (text to speech) function (610; 710; 810; 910; 910), A first audio signal generated in response to the playback of video content is obtained (620; 720; 820; 920), and A second audio signal generated in response to the activation of the above TTS function is obtained (620; 720; 820; 920), and In response to the activation of the above TTS function, the first audio signal is classified into a first audio object and a second audio object (630; 730; 830; 930), A first weight for the first audio object and a second weight for the second audio object are determined (650; 740, 750; 840; 940), and the first weight and the second weight are different, The first weight is assigned to the first audio object, and the second weight is assigned to the second audio object, The first audio object to which the first weight is assigned, the second audio object to which the second weight is assigned, and the second audio signal are combined into a synthesized audio signal, and The above synthesized audio signal is output (650, 660; 760, 770; 870, 880; 990, 1000), An electronic device (100) in which the first audio object and the second audio signal correspond to signals of the same type, and the second audio object corresponds to signals of a different type from the first audio object and the second audio signal.

2. In Paragraph 1, The first audio object is an electronic device (100) corresponding to a voice signal.

3. In Paragraph 2, The above second audio object is an electronic device (100) corresponding to a signal excluding voice signals.

4. In Paragraph 1, The electronic device (100), wherein the first weight is smaller than the second weight.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) causes, An electronic device (100) for determining a third weight for the second audio signal (640; 750; 850; 950).

6. In Paragraph 5, When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) causes, An electronic device (100) that determines the third weight based on the signal strength of the first audio object (850; 950).

7. In Paragraph 6, The above third weight is an electronic device (100) having a positive correlation with the signal strength of the first audio object.

8. In Paragraph 5, When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) causes, A first time point at which the output of the second audio signal is initiated and a second time point at which the output of the second audio signal is terminated are obtained (963), An electronic device (100) that assigns the first weight and the second weight to each of the first audio object and the second audio object output during the first time point and the second time point (970).

9. In Paragraph 8, When the above instructions are executed individually or collectively by the one or more processors (120), the electronic device (100) causes, An electronic device (100) that assigns the second weight (980) to the second audio object output during the first time point and the second time point.

10. In Paragraph 1, An electronic device (100) configured such that a neural network model obtained by learning the relationship between a plurality of sample audio signals and a plurality of sample audio objects is stored in the memory (110).

11. A method for controlling an electronic device (100), Action to enable TTS (text to speech) function (610; 710; 810; 910); An operation to acquire a first audio signal generated in response to the playback of video content (620; 720; 820; 920); An operation to acquire a second audio signal generated in response to the activation of the above TTS function (620; 720; 820; 920); In response to the activation of the above TTS function, an operation (630; 730; 830; 930) of classifying the above first audio signal into a first audio object and a second audio object; An operation to determine a first weight for the first audio object and a second weight for the second audio object (650; 740, 750; 840; 940); the first weight and the second weight are different, and The operation of assigning the first weight to the first audio object and assigning the second weight to the second audio object; The operation of synthesizing the first audio object to which the first weight is assigned, the second audio object to which the second weight is assigned, and the second audio signal into a synthesized audio signal; and the operation of outputting the synthesized audio signal; (660, 670; 760, 770; 870, 880; 990, 1000), and A method in which the first audio object and the second audio signal correspond to signals of the same type, and the second audio object corresponds to signals of a different type from the first audio object and the second audio signal.

12. In Paragraph 11, The above first audio object corresponds to a voice signal, a method.

13. In Paragraph 12, The above second audio object is a method corresponding to a signal excluding the voice signal.

14. In Paragraph 11, A method in which the first weight is smaller than the second weight.

15. In Paragraph 11, A method further comprising an operation (640; 750; 850; 950) for determining a third weight for the second audio signal.

Images