Electronic device and control method therefor

Abstract

An electronic device and a control method therefor are disclosed. The electronic device comprises at least one memory for storing instructions, and at least one processor including processing circuity, wherein, when executed individually or collectively by the at least one processor, the instructions can instruct the electronic device to: obtain quality information including a score indicating the level of quality of an input voice signal when the input voice signal corresponding to a user speech is received; identify a registered voice signal for the user stored in the memory when the score is less than a first threshold value; obtain, on the basis of the registered voice signal, a reference voice signal indicating a voice characteristic of the user when the registered voice signal is identified; and obtain an output voice signal for the input voice signal on the basis of the reference voice signal and an input text corresponding to the input voice signal.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device, and specifically, to an electronic device capable of enhancing a reference voice signal used for synthesizing a user's voice and a method for controlling the same.

[0002] Recently, with the advancement of artificial intelligence-related technologies, personalized Text-to-Speech (TTS) technologies capable of synthesizing speech that reflects the unique characteristics of a user's voice are advancing. In particular, regarding personalized speech synthesis, a technology (so-called Zero-shot TTS or Voice Cloning) is attracting attention. This technology synthesizes (generates) an output speech that reflects the unique characteristics of the user's voice based on the input voice (reference voice) without the need for pre-registering the user's voice, training a neural network model using the registered voice, or undergoing a fine-tuning process.

[0003] In zero-shot TTS, high-quality output speech can be obtained when the quality of the input user voice (especially the initial user voice) is high. However, zero-shot TTS has a limitation in that it is difficult to obtain high-quality output speech when the quality of the input user voice is low.

[0004] For example, if the input user voice is too short, it is difficult to express the user's timbre; if the input user voice is the result of unstable speech, the output voice may also be unstable; and if sounds other than the input user voice are included, it is difficult to expect high-quality output voice.

[0005] The above limitations exist not only in zero-shot TTS but also in technologies that use a reference voice for the user, such as voice conversion (VC) that converts the user's voice into the voice of another user.

[0006] The aspects according to the present disclosure are intended to provide an electronic device and a method for controlling the same that can obtain a high-quality output voice signal by enhancing a reference voice signal used for synthesizing a user's voice, thereby solving at least the problems described above.

[0007] Additional aspects will be disclosed in part in the following description, and in part will be apparent from the description or can be learned through the practice of the presented embodiments.

[0008] According to one aspect of the present disclosure, an electronic device comprises at least one processor including at least one memory for storing instructions and a processing circuit, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device acquires quality information including a score indicating a level of quality of the input voice signal when an input voice signal corresponding to a user's utterance is received, and if the score is less than a first threshold value, identifies a registered voice signal for the user stored in the memory, and if the registered voice signal is identified, acquires a reference voice signal indicating the voice characteristics of the user based on the registered voice signal, and acquires an output voice signal for the input voice signal based on an input text corresponding to the input voice signal and the reference voice signal.

[0009] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the quality information based on at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, or the amount of frequency change of the voice signal.

[0010] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the input voice signal as the reference voice signal if the score is greater than or equal to the first threshold value.

[0011] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to obtain the reference voice signal by combining the input voice signal and the registered voice signal if the score is less than the first threshold value and greater than or equal to the second threshold value which is smaller than the first threshold value.

[0012] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the registered voice signal as the reference voice signal if the score is less than the second threshold value.

[0013] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may remove noise included in the input voice signal based on the registered voice signal if the score is less than the second threshold value, and obtain the reference voice signal by combining the noise-removed input voice signal and the registered voice signal.

[0014] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may identify the registered voice signal among the plurality of registered voice signals based on the similarity between the plurality of registered voice signals and the input voice signal.

[0015] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may acquire a plurality of scores representing the quality of each of the plurality of registered voice signals, acquire a plurality of registered texts corresponding to each of the plurality of registered voice signals, and identify the registered voice signal among the plurality of registered voice signals based on the similarity, the plurality of scores representing the quality of each of the plurality of registered voice signals, and the type of the plurality of registered texts.

[0016] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may acquire the input voice signal as the reference voice signal if the registered voice signal is not identified.

[0017] The memory stores registration information including the plurality of registered voice signals and the plurality of scores, and when the instructions are executed individually or collectively by the at least one processor, the electronic device may update the registration information by adding the reference voice signal to the plurality of registered voice signals and adding a score indicating the level of the quality of the input voice signal to the plurality of scores.

[0018] When the above instructions are executed individually or collectively by the at least one processor, the electronic device can acquire text of a first language corresponding to the input voice signal, and the input text can be acquired by translating the text into a second language.

[0019] According to one aspect of the present disclosure, a method for controlling an electronic device comprises the steps of: receiving an input voice signal corresponding to a user’s utterance, acquiring quality information including a score indicating a level of quality of the input voice signal; identifying a registered voice signal for the user among a plurality of registered voice signals stored in memory if the score is less than a first threshold value; acquiring a reference voice signal indicating a voice characteristic of the user based on the registered voice signal if the registered voice signal is identified; and acquiring an output voice signal for the input voice signal based on an input text corresponding to the input voice signal and the reference voice signal.

[0020] The step of acquiring the quality information may include acquiring the quality information based on at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, and the amount of frequency change of the input voice signal.

[0021] The above method may further include the step of acquiring the input voice signal as the reference voice signal if the score is greater than or equal to the first threshold value.

[0022] The step of acquiring the reference voice signal may include the step of acquiring the reference voice signal by combining the input voice signal and the registered voice signal if the score is less than the first threshold value and greater than or equal to the second threshold value which is smaller than the first threshold value.

[0023] According to one aspect of the present disclosure, a non-transient computer-readable recording medium comprising a program for executing a method for controlling an electronic device comprises: a method for controlling the electronic device comprising, when an input voice signal corresponding to a user’s utterance is received, a step of obtaining quality information including a score indicating a level of quality of the input voice signal; when the score is less than a first threshold value, a step of identifying a registered voice signal for the user among a plurality of registered voice signals stored in the memory; when the registered voice signal is identified, a step of obtaining a reference voice signal indicating a voice characteristic of the user based on the registered voice signal; and a step of obtaining an output voice signal for the input voice signal based on an input text corresponding to the input voice signal and the reference voice signal.

[0024] The above and other aspects, features, and advantages according to specific embodiments of the present disclosure will become more apparent from the following description, which is described together with the accompanying drawings.

[0025] FIG. 1 is a block diagram showing an exemplary configuration of an electronic device according to one or more embodiments of the present disclosure,

[0026] FIG. 2 is a drawing showing an operation performed by a plurality of modules and inputs / outputs according to one or more embodiments of the present disclosure,

[0027] FIG. 3 is a drawing for explaining one or more embodiments related to an operation when the score is greater than or equal to a first threshold value and an operation when a registered voice signal corresponding to an input voice signal is not identified.

[0028] FIG. 4 is a drawing for illustrating one or more embodiments related to an operation of removing noise from an input voice signal,

[0029] FIG. 5 is a drawing for illustrating one or more embodiments related to the operation of acquiring input text,

[0030] FIG. 6 is a drawing for illustrating one or more embodiments related to the operation of updating a registration information database,

[0031] FIG. 7 is a block diagram showing an exemplary configuration of an electronic device according to one or more embodiments of the present disclosure, and,

[0032] FIG. 8 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0033] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0034] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0035] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0036] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0037] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0038] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) A, (2) B, or (3) both A and B.

[0039] The terms “first,” “second,” “first,” or “second” used in this disclosure may refer to corresponding components regardless of importance or order and are used to distinguish components from one another without limiting them.

[0040] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0041] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0042] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0043] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0044] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0045] The various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0046] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0047] FIG. 1 is a block diagram showing the configuration of an electronic device (100) according to one or more embodiments of the present disclosure, and FIG. 2 is a drawing showing a plurality of modules and inputs / outputs according to one or more embodiments of the present disclosure.

[0048] The ‘electronic device (100)’ may refer to a device capable of enhancing a reference voice signal used for synthesizing a user’s voice. For example, the electronic device (100) may be implemented as a user terminal such as a smartphone or tablet PC, or as a server or cloud computing device. There are no particular limitations on the type of electronic device (100) according to the present disclosure.

[0049] As illustrated in FIG. 1, the electronic device (100) may include memory (110) and a processor (120).

[0050] At least one instruction regarding an electronic device (100) may be stored in the memory (110). Additionally, an operating system (O / S) for operating the electronic device (100) may be stored in the memory (110). Furthermore, various software programs or applications for operating the electronic device (100) may be stored in the memory (110) according to various embodiments of the present disclosure. Additionally, the memory (110) may include semiconductor memory such as flash memory or magnetic storage media such as a hard disk.

[0051] Specifically, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110), and the processor (120) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (120), and reading, writing, modifying, deleting, updating, etc. of data by the processor (120) may be performed.

[0052] Meanwhile, in the present disclosure, the term memory (110) may be used to include memory (110), ROM, RAM, or a memory card (e.g., micro SD card, memory stick) mounted in the processor (120).

[0053] In one embodiment, the memory (110) may store an input voice signal, a registered voice signal, a reference voice signal, and an output voice signal according to the present disclosure. The memory (110) may store information regarding registration information, input text, registered text, scores, and threshold values. The memory (110) may store information regarding a plurality of modules and information regarding a neural network model according to the present disclosure.

[0054] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (110), and the information stored in the memory (110) may be updated as it is received from an external device or input by a user.

[0055] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to the configuration of the electronic device (100) including a memory (110), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above.

[0056] The processor (120) can be implemented in various ways. For example, the processor (120) can be implemented as at least one of an Application Specific Integrated Circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware Finite State Machine (FSM), or a Digital Signal Processor (DSP). Meanwhile, in this disclosure, the term processor (120) may be used to include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a MPU (Micro Processor Unit), etc.

[0057] In one embodiment, the processor (120) can enhance a reference voice signal representing the voice characteristics of the user. Specifically, the processor (120) can enhance the reference voice signal used for voice synthesis by using a voice signal input according to the user's speech and a previously stored registered voice signal.

[0058] The user's 'voice characteristics' may collectively refer to the unique features of the user's voice and may include, for example, characteristics regarding the user's timbre, intonation, pronunciation, prosody, style, etc. 'Enhancing the reference voice signal' may refer to acquiring a high-quality voice signal that effectively represents the user's voice characteristics, and the term 'enhancement' may be replaced with terms such as 'improvement' or 'augmentation'.

[0059] The processor (120) can enhance a reference voice signal using a plurality of modules. As illustrated in FIG. 2, the plurality of modules according to one embodiment may include a quality evaluation module (1110), a speaker recognition module (1120), a quality enhancement module (1130), and a voice synthesis module (1140).

[0060] For convenience of explanation, the following description will be based on the premise that all of the multiple modules are implemented by the processor (120) of the electronic device (100); however, various embodiments according to the present disclosure may be similarly applied even when at least one of the multiple modules is implemented by an external device. Hereinafter, with reference to FIG. 2 in conjunction with FIG. 1, various embodiments in which the processor (120) implements multiple modules will be described.

[0061] When the processor (120) receives an input voice signal corresponding to a user's speech, it can obtain quality information including a score indicating the quality level of the input voice signal. The 'input voice signal' may refer to a voice signal generated by a user's speech and input into the electronic device (100). Specifically, the processor (120) can receive an input voice signal corresponding to a user's speech through a microphone included in the electronic device (100). The processor (120) may also receive an input voice signal from an external device through a communication interface (130) included in the electronic device (100).

[0062] As illustrated in FIG. 2, the processor (120) can input an input voice signal to the quality evaluation module (1110) to perform a quality evaluation process of the input voice signal. The 'quality evaluation module (1110)' can evaluate the quality of the input voice signal. Specifically, when an input voice signal is input, the quality evaluation module (1110) can obtain quality information. 'Quality information' may refer to the level or degree of quality of the input voice signal. The quality information may include a score indicating the level of quality of the input voice signal. Additionally, the quality evaluation module (1110) can identify whether the obtained score is greater than or equal to a preset first threshold value and obtain the identification result.

[0063] For the sake of convenience of explanation, scores included in quality information will be used as examples for quality evaluation indicators below; however, the present disclosure is not limited thereto, and any information that can refer to the quality of an input voice signal, such as 'priority' or 'quality level', may be used in place of the score.

[0064] The quality evaluation module (1110) can obtain quality information based on at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, or the amount of frequency change of the voice signal.

[0065] In one embodiment, the quality evaluation module (1110) can obtain a score based on the length of the input voice signal (e.g., the duration of the input voice signal). For example, the quality evaluation module (1110) can obtain a higher score when the length of the input voice signal is greater than or equal to a threshold length (e.g., an input voice signal threshold length value) compared to when the length of the input voice signal is less than the threshold length.

[0066] In one embodiment, the quality evaluation module (1110) can obtain a score based on the length of a voice part included in the input voice signal (e.g., the duration of the voice part). For example, the quality evaluation module (1110) can obtain a higher score when the length of the voice part included in the input voice signal is greater than or equal to a threshold length compared to when the length of the voice part included in the input voice signal is less than a threshold length.

[0067] In one embodiment, the quality evaluation module (1110) can obtain a score based on the length of the noise portion included in the input voice signal (e.g., a noise threshold length value). For example, the quality evaluation module (1110) can obtain a higher score when the length of the noise portion included in the input voice signal is greater than or equal to the threshold length compared to when the length of the noise portion included in the input voice signal is less than the threshold length.

[0068] For example, the process of distinguishing between the voice portion and the noise portion in an input voice signal can be performed through a VAD (Voice Activity Detection) process. Specifically, the quality evaluation module (1110) can distinguish between the voice portion and the noise portion in the input voice signal by extracting features for each segment based on the energy, frequency characteristics, signal-to-noise ratio (SNR), etc. of the input voice signal, and identifying whether the extracted features exceed a specific criterion (e.g., threshold value).

[0069] Acquiring a score according to the above embodiments takes into account that if the length of the input voice signal is too short (e.g., the duration of the input voice signal is short), the length of the voice part included in the input voice signal is too short (e.g., the duration of the voice part is short), or the length of the noise part included in the input voice signal is too long (e.g., the duration of the noise part is too long), it may be difficult to accurately represent the user's voice characteristics, and consequently, it may be difficult to acquire an output voice signal that reflects the user's voice characteristics.

[0070] In one embodiment, the quality evaluation module (1110) can obtain a score based on the frequency change amount of the voice signal. For example, the quality evaluation module (1110) can obtain a higher score when the frequency change amount of the voice signal is below a threshold level (e.g., frequency change amount threshold value) compared to when the frequency change amount of the voice signal is above the threshold level. In particular, when the frequency change amount of the voice signal is above the threshold level, such as when the pitch indicating the pitch of the sound according to the voice signal changes rapidly or vibrates, the quality evaluation module (1110) can evaluate the quality of the input voice signal as low.

[0071] The quality evaluation module (1110) can obtain a score based on a combination of two or more of the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, and the amount of frequency change of the voice signal.

[0072] In one embodiment, the quality evaluation module (1110) identifies whether the length of the input voice signal is greater than or equal to a threshold length, whether the length of the voice portion included in the input voice signal is greater than or equal to a threshold length, whether the length of the noise portion included in the input voice signal is greater than or equal to a threshold length, and whether the amount of frequency change of the voice signal is less than a threshold level, and may assign a higher score as the number of cases corresponding to the above four cases of the input voice signal increases. For example, the quality evaluation module (1110) may assign a score of 4 if the input voice signal corresponds to all four cases above, and assign a score of 1 if the input voice signal corresponds to only one of the above four cases.

[0073] In one embodiment, the quality evaluation module (1110) assigns a score to at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, or the amount of frequency change of the voice signal, and can obtain a final score based on a weighted sum for each score. The weights for each score used in the weighted sum may be changed according to the settings of the user or developer. For example, the highest weight may be assigned to the length of the voice portion included in the input voice signal, and the lowest weight may be assigned to the amount of frequency change of the voice signal.

[0074] If the score is less than a preset first threshold, the processor (120) can identify a registered voice signal for the user in memory (110). The 'first threshold' may refer to a value that serves as a criterion for determining whether to perform a quality enhancement process for the input voice signal, and may be changed according to the settings of the user or developer. That is, if the score representing the quality of the input voice signal is greater than or equal to the first threshold, it may be a case where a quality enhancement process for the input voice signal is not required, and if the score representing the quality of the input voice signal is less than the first threshold, it may be a case where a quality enhancement process for the input voice signal is required. An embodiment related to the case where the score is greater than or equal to the first threshold is described in detail with reference to FIG. 3.

[0075] 'Registered voice signal' may refer to a voice signal registered as a voice signal for a specific user. Specifically, the registered voice signal may be used as a term to refer to a voice signal when it is identified as a voice signal corresponding to a specific user's speech and information about the voice signal is stored in memory (110).

[0076] Information regarding a registered voice signal (hereinafter referred to as "registration information") may be stored in the memory (110) of an electronic device (100) and may also be stored in the memory (110) of an external device. The registration information may include a file corresponding to the registered voice signal, and the file format may be stored in various formats, such as WAV (Waveform Audio File Format) or FLAC (Free Lossless Audio Codec).

[0077] Additionally, the registration information may include a score indicating the quality of the registered voice signal, speech feature information for the registered voice signal, and a registration text corresponding to the registered voice signal. The registration information may be implemented as a registration information database as shown in FIG. 2, and the registration information database may be stored in the memory (110) of the electronic device (100) or in the memory of an external device. An embodiment related to the operation of updating the registration information database will be described in detail with reference to FIG. 6.

[0078] If the registration information includes speech feature information for a registered voice signal, the processor (120) inputs the registered voice signal into a speech feature information module (which may be included in the speaker recognition module (1120) as described below) to obtain speech feature information corresponding to the registered voice signal, and can store the obtained speech feature information together with the registered voice signal in the memory (110) as registration information. Additionally, if there are multiple registered voice signals for a specific user, the processor (120) may calculate the average of the multiple speech feature information corresponding to each of the multiple registered voice signals and store the average of the multiple speech feature information in the memory (110) as registration information.

[0079] If the registration information includes a registration text corresponding to a registration voice signal, the processor (120) can perform voice recognition on the registration voice signal to obtain the registration text corresponding to the registration voice signal, and store the obtained registration text together with the registration voice signal as registration information in memory (110).

[0080] As illustrated in FIG. 2, if the score is less than the first threshold value, the processor (120) inputs an input voice signal to the speaker recognition module (1120) to identify a registered voice signal for a user. The speaker recognition module (1120) can recognize the user, i.e., the speaker, who uttered the voice corresponding to the input voice signal. The speaker recognition module (1120) can identify the user corresponding to the input voice signal, and specifically, can identify the registered voice signal corresponding to the input voice signal by comparing the input voice signal with the registered voice signal stored in the memory (110).

[0081] Since the identification process when there is only one registered voice signal stored in the registration information database is the same as when there are multiple registered voice signals stored in the registration information database, the following description is based on the premise that multiple registered voice signals are stored. An embodiment related to the case where a registered voice signal corresponding to an input voice signal is not identified is described in detail with reference to FIG. 3.

[0082] The speaker recognition module (1120) can identify a registered voice signal for a user (i.e., a user who uttered a voice corresponding to an input voice signal) based on the similarity (or distance) between a plurality of registered voice signals and an input voice signal stored in a registration information database.

[0083] The speaker recognition module (1120) may include a neural network (which may be referred to as a speech feature extraction module, encoder, inbedder, etc.) trained to acquire speech feature information representing the characteristics of an input speech signal. For example, the speaker recognition module (1120) may include a TDNN (Time-Delay Neural Network) model, which is primarily characterized by learning time delay patterns, or an ECAPA-TDNN (Emphasized Channel Attention, Propagation, and Aggregation Time-Delay Neural Network) model, which is based on TDNN but additionally applies channel emphasis, attention mechanisms, etc. However, there are no special restrictions on the type of neural network used in the implementation of the speaker recognition module (1120).

[0084] 'Speech feature information' may include information representing unique user features extracted from speech uttered by the user. Speech feature information may be replaced with terms such as 'feature value', 'feature vector', 'embedding', etc. Speech feature information may be acquired after the input voice signal is acquired, but it may also be acquired in advance before the input voice signal is acquired and stored as registration information in memory (110).

[0085] The speaker recognition module (1120) can acquire speech feature information for each of the plurality of registered voice signals and the input voice signal. The speaker recognition module (1120) can identify the registered voice signal corresponding to the input voice signal among the plurality of registered voice signals by calculating a value representing the similarity (e.g., cosine similarity, Jaccard similarity, Euclidean distance, etc.) between the speech feature information corresponding to each of the plurality of registered voice signals and the speech feature information corresponding to the input voice signal.

[0086] Multiple registered voice signals may be registered voice signals for a single user, or they may be registered voice signals for multiple users. If the multiple registered voice signals are registered voice signals for multiple users, the speaker recognition module (1120) can identify the user corresponding to the input voice signal among the multiple users. Additionally, once the user corresponding to the input voice signal is identified, the speaker recognition module (1120) can identify the registered voice signal corresponding to the input voice signal among the multiple registered voice signals for the identified user.

[0087] For example, the speaker recognition module (1120) can calculate the average of speech feature information corresponding to a registered voice signal for each of the multiple users, and can identify the user corresponding to the input voice signal among the multiple users by identifying the average of the speech feature information for each of the multiple users that has the highest similarity to the speech feature information of the input voice signal. Then, by identifying the speech feature information that has the highest similarity to the speech feature information of the input voice signal among the speech feature information for each of the multiple registered voice signals for the identified user, it can identify the registered voice signal corresponding to the input voice signal among the multiple registered voice signals for the identified user.

[0088] When a user corresponding to an input voice signal is identified, the processor (120) may use the average of the utterance characteristic information of the identified user to obtain an output voice signal. In addition, the processor (120) may use only the utterance characteristic information of the registered voice signal corresponding to the input voice signal among multiple voice signals for the identified user to obtain an output voice signal. If the average of the utterance characteristic information of the identified user is used to obtain an output voice signal, the representative voice characteristics of the user can be reliably implemented, whereas if only the utterance characteristic information of a specific voice signal among multiple voice signals for the identified user is used to obtain an output voice signal, the voice characteristics of the user that best match the prosody, style, etc. of the input voice signal can be implemented.

[0089] Although an embodiment for identifying a registered voice signal for a user based on similarity between a plurality of registered voice signals and an input voice signal has been described above, the speaker recognition module (1120) can identify a registered voice signal for a user by combining at least one of a plurality of scores corresponding to each of the plurality of registered voice signals and a plurality of registered texts corresponding to each of the plurality of registered voice signals, along with similarity.

[0090] In one embodiment, the speaker recognition module (1120) can obtain a plurality of scores representing the quality of each of the plurality of registered voice signals. The speaker recognition module (1120) can obtain a plurality of registered texts corresponding to each of the plurality of registered voice signals. Furthermore, the speaker recognition module (1120) can identify the registered voice signals using at least one of the types of the plurality of scores and the plurality of registered texts, as well as similarity.

[0091] For example, the speaker recognition module (1120) can identify registered voice signals by assigning a high weight to a registered voice signal with a high score among a plurality of registered voice signals. That is, the speaker recognition module (1120) can prioritize identifying registered voice signals of relatively high quality and reduce the frequency of identifying registered voice signals of relatively low quality. This is because it is difficult to expect the effect of high quality improvement when performing a quality improvement process as described below using registered voice signals of low quality.

[0092] For example, if the input text is a question, the speaker recognition module (1120) can assign a high weight to the registered voice signal corresponding to the question among the multiple registered voice signals based on the multiple registered texts, and if the input text is conversational, the speaker recognition module (1120) can identify the registered voice signal by assigning a high weight to the registered voice signal corresponding to the conversational among the multiple registered voice signals based on the multiple registered texts. This is intended to identify the registered voice signal that best matches the prosody and style of the input voice signal and to use it in the quality improvement process.

[0093] In addition, speaker recognition operations can be implemented in various ways depending on the design or configuration of the speech synthesis module (1140). For example, depending on which elements among the elements such as sound quality, speaker similarity, and delay prevention are focused on when designing or configuring the speech synthesis module (1140), it may be determined whether to assign a higher weight to which registered voice signal among a plurality of registered voice signals.

[0094] When a registered voice signal is identified, the processor (120) can obtain a reference voice signal by improving the quality of the input voice signal based on the registered voice signal. The 'reference voice signal' may refer to a voice signal representing the voice characteristics of a user. The reference voice signal can be obtained based on the input voice signal and the registered voice signal, and can be used for the synthesis of the output voice signal.

[0095] As illustrated in FIG. 2, the processor (120) can input an input voice signal and a registered voice signal into a quality enhancement module (1130) to obtain a reference voice signal. The 'quality enhancement module (1130)' may refer to a module capable of enhancing the quality of a voice signal. Specifically, when an input voice signal and a registered voice signal are input, the quality enhancement module (1130) can enhance the quality of the input voice signal based on the registered voice signal.

[0096] In the following description, quality improvement will be expressed as improving the quality of the input voice signal, but this is merely for convenience of explanation. The processor (120) may improve the quality of the registered voice signal to obtain a reference voice signal, or it may obtain a reference voice signal by using both the input voice signal and the registered voice. That is, the quality improvement module (1130) may implement various embodiments as follows to obtain a reference voice signal of superior quality to each of the input voice signal and the registered voice signal by using at least one of the input voice signal and the registered voice signal.

[0097] In one embodiment, if the score is less than a first threshold and greater than or equal to a second threshold that is smaller than the first threshold, the quality enhancement module (1130) can obtain a reference voice signal by combining the input voice signal and the registered voice signal. On the other hand, if the score is less than the second threshold, the quality enhancement module (1130) can obtain the registered voice signal as the reference voice signal.

[0098] The 'second threshold' refers to a value that serves as a criterion for determining whether to use the input voice signal in the quality enhancement process, and may be a value smaller than the first threshold; it may be changed according to the settings of the user or developer. That is, if the score representing the quality of the input voice signal is equal to or greater than the second threshold, it may be desirable to use the input voice signal in the quality enhancement process, and if the score representing the quality of the input voice signal is less than the second threshold, it may not be desirable to use the input voice signal in the quality enhancement process.

[0099] For example, if a score indicating the quality of an input voice signal is less than a first threshold value and thus a quality improvement process needs to be performed, and if the quality of the input voice signal is higher than a second threshold value and thus it is desirable to use the input voice signal in the quality improvement process, the quality improvement module (1130) can obtain a reference voice signal by using the input voice signal together with the registered voice signal. Conversely, if a score indicating the quality of an input voice signal is less than a first threshold value and thus a quality improvement process needs to be performed, but if the quality of the input voice signal is lower than a second threshold value and thus it is not desirable to use the input voice signal in the quality improvement process, the quality improvement module (1130) can obtain the registered voice signal as a reference voice signal by excluding the input voice signal.

[0100] The process of concatenating the input voice signal and the registered voice signal can be performed as follows. For example, the quality enhancement module (1130) can concatenate two signals so that the two signals are continuous. In particular, if the time at which the registered voice signal is acquired is within a threshold period from the time at which the input voice signal is input, the timbre and style of the input voice signal are highly likely to correspond to the timbre and style of the registered voice signal. Therefore, the effect of quality enhancement can be achieved simply by concatenating the input voice signal and the registered voice signal, which have similar timbres and styles, and increasing the length of the reference voice signal.

[0101] Additionally, the quality enhancement module (1130) may further perform normalization to adjust the magnitudes of the two signals to a constant level to prevent quality degradation caused by discontinuities at the boundary between the two signals before or after combining the input voice signal and the registered voice signal, smoothing to mitigate abrupt changes (spikes or noise) in the voice signal, and noise suppression to remove unwanted background noise from the voice signal. If the quality of the registered voice signal is below a certain level or is lower than that of the input voice signal, the quality enhancement module (1130) may exclude the registered voice signal from the quality enhancement process and acquire the input voice signal as a reference voice signal.

[0102] In one embodiment, if the score is less than a second threshold value, noise in the input voice signal may be removed, and a reference voice signal may be obtained by combining the noise-removed input voice signal and the registered voice signal.

[0103] An embodiment related to removing noise included in an input voice signal based on a registered voice signal is described in detail with reference to FIG. 4.

[0104] The quality improvement methods described above are not independent of one another, and it goes without saying that performing them sequentially or comprehensively can contribute to further enhancing the quality of the reference speech signal. Furthermore, in one embodiment, after performing some of the quality improvement methods described above and evaluating the quality of the reference speech signal, if the quality is below a critical level, other parts of the quality improvement methods described above may be performed again.

[0105] The processor (120) can obtain an output voice signal for an input voice signal based on an input text and a reference voice signal corresponding to the input voice signal. The 'output voice signal' is an output voice signal containing information about the input voice signal, and specifically, may refer to a voice signal corresponding to the input text and reflecting the user's voice characteristics.

[0106] *102 As illustrated in FIG. 2, the processor (120) can obtain an output voice signal by inputting an input text and a reference voice signal into a voice synthesis module (1140). Specifically, the processor (120) can obtain speech feature information corresponding to the reference voice signal, and input the speech feature information and the input text into the voice synthesis module (1140) to obtain an output voice signal in which the speech feature information corresponding to the reference voice signal is reflected.

[0107] The 'speech synthesis module (1140)' can synthesize a speech signal by converting input text into speech and may include a neural network model. For example, the neural network model included in the speech synthesis module (1140) may include a model referred to as a so-called speech synthesis model or TTS (Text-to-Speech) model.

[0108] 'Input text' may refer to text corresponding to an input voice signal. Specifically, the input text may be text representing a translation result of the input voice signal, or text entered together with the input voice signal. For example, the input voice signal may be a voice signal representing speech in a first language (e.g., Korean), and the input text may be text representing the result of translating text obtained from speech recognition of the input voice signal into a second language (e.g., English).

[0109] For example, if the input voice signal is a voice signal representing the voice of a first language (e.g., Korean) and the input text is text representing the result of translating text obtained from the voice recognition result of the input voice signal into a second language (e.g., English), the output voice signal may be a voice signal representing the voice of the second language (e.g., English). Consequently, for an input voice signal representing the voice of the first language (e.g., Korean), an output voice signal representing the voice of the second language (e.g., English) may be obtained. An embodiment related to the operation of obtaining an input text corresponding to an input voice signal will be described in detail with reference to FIG. 5.

[0110] When an output voice signal is obtained, the processor (120) may provide the output voice signal. Additionally, the processor (120) may provide text corresponding to the output voice signal along with the output voice signal. An embodiment related to providing the output voice signal will be described in detail with reference to FIG. 7.

[0111] According to the embodiments described above with reference to FIGS. 1 and FIG. 2, the electronic device (100) can enhance a reference voice signal used for voice synthesis by using a voice signal input according to the user's speech and a previously stored registered voice signal. Accordingly, the electronic device (100) can obtain a synthesized voice with a significantly high degree of similarity to the actual user's unique voice.

[0112] FIG. 3 is a diagram for explaining one or more embodiments related to an operation when the score is greater than or equal to a first threshold value and an operation when a registered voice signal corresponding to an input voice signal is not identified.

[0113] In the foregoing, with reference to FIGS. 1 and FIGS. 2, one or more embodiments related to the operation when the score is less than the first threshold value and the operation when a registered voice signal corresponding to the input voice signal is identified have been described.

[0114] As illustrated in FIG. 3, the processor (120) may obtain an output voice signal by using the input voice signal as a reference voice signal without improving the quality of the input voice signal using the quality enhancement module (1130). In one embodiment, if the score is greater than or equal to a first threshold value or if a registered voice signal for a user is not identified, the processor (120) may obtain the input voice signal as a reference voice signal.

[0115] As illustrated in FIG. 3, the processor (120) can input an input voice signal to the quality evaluation module (1110) to perform a quality evaluation operation of the input voice signal. If the score obtained through the quality evaluation module (1110) is greater than or equal to a first threshold value, the processor (120) can obtain the input voice signal as a reference voice signal.

[0116] That is, if the score is greater than or equal to the first threshold value, it may be a case where a quality enhancement process for the input voice signal is not required, so the processor (120) may not perform a quality enhancement process and may treat the input voice signal itself as a reference voice signal. Then, the processor (120) inputs the input voice signal as a reference voice signal to the voice synthesis module (1140) and, together with it, inputs the input text to the voice synthesis module (1140) to obtain an output voice signal.

[0117] If the score obtained through the quality evaluation module (1110) is less than the first threshold value, the processor (120) inputs the input voice signal to the speaker recognition module (1120) as shown in FIG. 3, and can identify the registered voice signal for the user.

[0118] When a registered voice signal for a user is identified through the speaker recognition module (1120), as described above, the processor (120) can obtain a reference voice signal by improving the quality of the input voice signal based on the registered voice signal.

[0119] On the other hand, if a registered voice signal for a user is not identified through the speaker recognition module (1120), the processor (120) can obtain an input voice signal as a reference voice signal, as shown in FIG. 3. Then, the processor (120) can input the input voice signal as a reference voice signal to the voice synthesis module (1140) and, together with it, input text to the voice synthesis module (1140) to obtain an output voice signal.

[0120] According to the embodiments described above with reference to FIG. 3, the electronic device (100) can perform efficient voice synthesis by treating the input voice signal itself as a reference voice signal when a quality enhancement process for the input voice signal is not required, or when a registered voice signal for a user, which is a voice signal for quality enhancement of the input voice signal, is not identified.

[0121] FIG. 4 is a drawing for illustrating one or more embodiments related to removing noise from an input voice signal.

[0122] As described above, when a score indicating the quality of an input voice signal is less than a first threshold value and thus a quality improvement process needs to be performed, but when the quality of the input voice signal is lower than a second threshold value and thus it is not desirable to use the input voice signal in the quality improvement process, the processor (120) can exclude the input voice signal and obtain a registered voice signal as a reference voice signal.

[0123] In one embodiment, when the quality of the input voice signal is lower than the second threshold value and it is not desirable to use the input voice signal in the quality improvement process, the processor (120) can remove noise from the input voice signal using the registered voice signal without excluding the input voice signal, and then perform the quality improvement process as described above.

[0124] Referring to FIG. 4, the processor (120) can input a registered voice signal into a speech feature extraction module (1210) to obtain speech feature information for the registered voice signal.

[0125] The 'utterance feature extraction module (1210)' may refer to a module capable of extracting utterance feature information representing the unique characteristics of a user from an input voice signal. The utterance feature extraction module (1210) may include a neural network model trained to extract the user's utterance features from an input voice signal.

[0126] For example, the speech feature extraction module (1210) may include a TDNN (Time-Delay Neural Network) model, which is primarily characterized by learning time delay patterns, or an ECAPA-TDNN (Emphasized Channel Attention, Propagation, and Aggregation Time-Delay Neural Network) model, which is based on TDNN but additionally applies channel emphasis, attention mechanisms, etc. The speech feature extraction module (1210) may be implemented as a separate module from other modules as shown in FIG. 4, but may also be included in a speaker recognition module (1120) or a noise removal module (1220), etc.

[0127] As described above, 'utterance feature information' may include information representing unique user features extracted from voice uttered by the user. Although utterance feature information extracted from an input voice signal was described in the description of FIGS. 1 and 2, utterance feature information may also be extracted from a registered voice signal (or a reference voice signal which is the voice signal to be registered). Speech feature information may be replaced with terms such as 'feature value', 'feature vector', 'embedding', etc.

[0128] When speech feature information for a registered voice signal is obtained, the processor (120) inputs the input voice signal and the speech feature information for the registered voice signal to a noise removal module (1220) to obtain a noise-removed input voice signal.

[0129] The 'noise removal module (1220)' may refer to a module capable of removing noise from an input voice signal and leaving only the signal corresponding to the user's voice. For example, the noise removal module (1220) may include a neural network model trained to generate mask information to leave only the user's voice from the input voice signal. The noise removal module (1220) may also be referred to by terms such as a speech enhancement module.

[0130] 'Mask information' may represent information for enhancing components corresponding to the user's voice in an input voice signal and suppressing noise or other components corresponding to other users' voices, and may also be referred to by terms such as 'filter information'.

[0131] For example, the mask information may include multiple weights ranging from 0 to 1, and each of the multiple weights may correspond to each of the multiple cells of a spectrogram corresponding to an input voice signal. Each of the multiple weights may indicate that the higher the value, the higher the probability of it corresponding to the user's voice, and that the lower the value, the higher the probability of it corresponding to noise or another user's voice.

[0132] When mask information is obtained, the processor (120) can obtain a noise-removed voice signal by applying the mask information to the input voice signal. Specifically, when mask information is obtained through the noise removal module (1220), the processor (120) can obtain a noise-removed input voice signal by multiplying each of a plurality of cells of a spectrogram corresponding to the input voice signal and each of a plurality of weights of mask information corresponding to each of the plurality of cells.

[0133] For example, among the multiple cells of the spectrogram, the cell corresponding to the mask information weight of 0.8 retains 80% of the original value, so the user's voice can be reflected relatively strongly, and among the multiple cells of the spectrogram, the cell corresponding to the mask information weight of 0.2 retains 20% of the original value, so the user's voice can be reflected relatively weakly.

[0134] For example, after mask information is applied to an input voice signal, the processor (120) can obtain a noise-removed input voice signal by performing an inverse short-time Fourier transform (inverse STFT) to convert the spectrogram into the time domain.

[0135] When a noise-removed input voice signal is obtained through the noise removal module (1220), the processor (120) can obtain a reference voice signal by inputting the registered voice signal and the noise-removed input voice signal to the quality enhancement module (1130). For example, the processor (120) can obtain a reference voice signal with improved quality by combining the noise-removed input voice signal and the registered voice signal using various quality enhancement methods as described above.

[0136] When a reference voice signal is obtained, the processor (120) can obtain an output voice signal by inputting the input text and the reference voice signal to the voice synthesis module (1140) as shown in FIG. 4.

[0137] According to the embodiments described above with reference to FIG. 4, the electronic device (100) can obtain a reference voice signal with improved quality by removing noise from the input voice signal using the registered voice signal, even when it is not desirable to use the input voice signal in the quality improvement process. Accordingly, even when the quality level of the input voice signal is very low, a reference voice signal with significantly improved quality can be obtained.

[0138] FIG. 5 is a drawing for illustrating one or more embodiments related to obtaining input text.

[0139] As described above, 'input text' may refer to text corresponding to an input voice signal, and specifically, may refer to text representing a translation result for an input voice signal.

[0140] In one embodiment, the processor (120) can obtain text of a first language corresponding to an input voice signal. Then, the processor (120) can obtain input text by translating the text into a second language.

[0141] As illustrated in FIG. 5, the processor (120) inputs an input voice signal to a voice recognition module (1310) to obtain original text corresponding to the input voice signal.

[0142] The 'speech recognition module (1310)' may refer to a module capable of obtaining text corresponding to an input speech signal. The speech recognition module (1310) may include a neural network model that may be referred to as a so-called speech recognition model or an ASR model (automatic speech recognition model). For example, the speech recognition module (1310) may include an acoustic model that converts audio signals into phonemes or words, and a language model that identifies words with high probability among candidate words obtained through the acoustic model.

[0143] 'Original text' may refer to text corresponding to the input voice signal and may be used as a term to distinguish it from the translated text as described below.

[0144] As illustrated in FIG. 5, the processor (120) inputs the original text into the translation module (1320) to obtain a translated text corresponding to the original text.

[0145] The 'translation module (1320)' may refer to a module capable of obtaining a translated text corresponding to an input original text. The translation module (1320) may include a neural network model that may be referred to as a so-called NMT (Neural Machine Translation) model, etc. For example, the translation module (1320) may obtain a translated text corresponding to an original text by processing the entire sentence simultaneously through a self-attention mechanism based on a Transformer.

[0146] 'Translated text' may refer to text in which the original text has been translated, and as shown in FIG. 5, the translated text may be input text that is input into the speech synthesis module (1140).

[0147] When a translated text is obtained, the processor (120) inputs the translated text to the speech synthesis module (1140) along with the reference speech signal obtained through the quality enhancement module (1130) to obtain an output speech signal for the input speech signal.

[0148] For example, the input voice signal may be a voice signal representing the voice of a first language (e.g., Korean), and the source text may be text in the first language (e.g., Korean) obtained as a result of voice recognition for the input voice signal. Additionally, the translated text may be text in the second language (e.g., English) obtained as a result of translation for the source text, and the output voice signal may be a voice signal representing the voice of the second language (e.g., English). Consequently, for an input voice signal representing the voice of the first language (e.g., Korean), an output voice signal representing the voice of the second language (e.g., English) can be obtained.

[0149] Although embodiments related to cases where the input text is text representing a translation result of an input voice signal have been described above, the present disclosure is not limited thereto, and the input text may be text entered together with the input voice signal. The term "text entered together with the input voice signal" may refer to text entered as text corresponding to the voice simultaneously with or after the utterance of the voice corresponding to the input voice signal.

[0150] According to the embodiments described above with reference to FIG. 5, the electronic device (100) can obtain a synthetic voice with a significantly high degree of similarity to the actual user's unique voice by reinforcing a reference voice signal representing the speech characteristics of the user in the process of obtaining an output voice signal in which an input voice signal is translated.

[0151] Accordingly, various embodiments according to the present disclosure may be particularly useful in real-time translation and interpretation processes, and when a user's speech continues, such as in real-time translation and interpretation, the electronic device (100) can continuously enhance the reference voice signal to continuously increase the similarity with the actual user's unique voice and obtain a synthesized voice.

[0152] FIG. 6 is a diagram illustrating one or more embodiments related to the operation of updating a registration information database.

[0153] The 'registration information database' may refer to a set of data regarding registration information, that is, information about registration information. The registration information database may include various information about the registration information along with user-specific registration information. For example, as illustrated in the structure diagram (610) of the registration information database in FIG. 6, the registration information database may include at least one user-specific registration information and quality information corresponding to each of the at least one registration information. The registration information database may be stored in the memory (110) of the electronic device (100) or in the memory (110) of an external device.

[0154] 'Registered voice signal' may refer to a voice signal registered as a voice signal for a specific user. The registered voice signal may be used as a term to refer to a voice signal when it is identified as a voice signal corresponding to a specific user's speech and information about the voice signal is stored in memory (110). In particular, the registered voice signal may refer to a reference voice signal obtained through the quality enhancement module (1130) that is registered.

[0155] In one embodiment, the processor (120) may register the acquired reference voice signal as a registered voice signal. As illustrated in FIG. 6, when the user who uttered the voice corresponding to the input voice signal is User 1, reference voice signal 1-1, reference voice signal 1-2, and reference voice signal 1-3 may be included in the registration information database as registered voice signals for User 1.

[0156] In this case, when a new reference voice signal for User 1 is obtained through the quality enhancement module (1130), the processor (120) can input the new reference voice signal into the registration module (1400). The registration module (1400) can update the registration information database based on the new reference voice signal. Since the new reference voice signal is a reference voice signal for User 1 that has already been registered, the registration module (1400) can add the new reference voice signal to the registration information database as reference voice signal 1-4 for User 1.

[0157] Reference voice signals for User 1 and User 2 may be included in the registration information database as registered voice signals for User 1 and User 2, and there may be cases where the reference voice signal for User 3 is not registered. In this case, as shown in FIG. 6, when a new reference voice signal for User 3 is acquired, the processor (120) may add the new reference voice signal to the registration information database as reference voice signal 3-1 for User 3.

[0158] In one embodiment, the processor (120) can add quality information to the registration information database along with the reference voice signal. As illustrated in FIG. 6, when quality information is obtained through the quality evaluation module (1110), the processor (120) can input the quality information along with the reference voice signal to the registration module (1400). The registration module (1400) can match (or pair) the quality information with the reference voice signal corresponding to the same input voice signal, and add the pair of the reference voice signal and quality information to the registration information database.

[0159] Quality information added to the registration information database may be various information that can refer to the quality of the input voice signal, such as a score, priority, quality level, etc. For example, the processor (120) can update the registration information by adding a reference voice signal obtained to a plurality of registered voice signals and adding a score corresponding to the input voice signal to a plurality of scores corresponding to each of the plurality of registered voice signals.

[0160] In one embodiment, the processor (120) may add speech feature information for a registered voice signal to a registration information database along with a reference voice signal. As illustrated in FIG. 6, the processor (120) may add speech feature information for a registered voice signal to a registration information database by matching it with a reference voice signal. For example, if there are multiple registered voice signals for a user (e.g., User 1 and User 2 in FIG. 6), the processor (120) may calculate the average of multiple speech feature information corresponding to each of the multiple registered voice signals and add the average of the multiple speech feature information to the registration information database as well.

[0161] Although not illustrated in FIG. 6, in one embodiment, the processor (120) may match the registered text corresponding to the registered voice signal with the reference voice signal and add it to the registration information database.

[0162] According to the embodiments described above with reference to FIG. 6, the electronic device (100) can not only synthesize a user's voice using a reference voice signal enhanced according to the present disclosure, but can also use the reference voice signal to enhance an input voice signal subsequently obtained by registering the reference voice signal.

[0163] In particular, when an initial input voice signal of a user not registered in the registration information database (e.g., User 3 in Fig. 6) is input, the quality of the output voice signal for that initial input voice signal may not be improved. However, since the initial input voice signal is used as a reference voice signal for speech synthesis and the reference voice signal is added to the registration information database, an output voice signal with improved quality can be provided for the input voice signal following the user's subsequent speech.

[0164] In addition, the electronic device (100) can manage a registration information database by matching quality information and registration text, etc., together with a reference voice signal, thereby performing the operation of identifying a registration voice signal for a user described with reference to FIGS. 1 and FIGS. 2 more effectively and efficiently.

[0165] Furthermore, if speech feature information corresponding to a reference voice signal is extracted in advance and stored in a registration information database, the electronic device (100) can perform the process of acquiring an output voice signal by reflecting the speech feature information more effectively and efficiently.

[0166] FIG. 7 is a block diagram showing in detail the configuration of an electronic device (100) according to one or more embodiments of the present disclosure.

[0167] As illustrated in FIG. 7, an electronic device (100) according to one embodiment of the present disclosure may further include a communication interface (130), an input interface (140), and an output interface (150), in addition to a memory (110) and a processor (120). However, the configurations illustrated in FIG. 1 and FIG. 7 are merely exemplary, and it is understood that in practicing the present disclosure, new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIG. 1 and FIG. 7.

[0168] The communication interface (130) includes a circuit and can perform communication with an external device. Specifically, the processor (120) can receive various data or information from an external device connected through the communication interface (130) and can also transmit various data or information to the external device.

[0169] The communication interface (130) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, or an Ultra-Wide Band (UWB) module. Specifically, the WiFi module and the Bluetooth module can each perform communication in a WiFi manner or a Bluetooth manner. When using a WiFi module or a Bluetooth module, various connection information such as an SSID is first transmitted and received, and then various information is transmitted and received after establishing a communication connection using this information.

[0170] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G (5th Generation). Furthermore, the NFC module can perform communication using the NFC (Near Field Communication) method, which utilizes the 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860~960 MHz, and 2.45 GHz. Additionally, the UWB module can accurately measure the Time of Arrival (ToA), which is the time it takes for a pulse to reach a target, and the Angle of Arrival (AoA), which is the angle of arrival of the pulse at the transmitting device, through communication between UWB antennas. Accordingly, precise distance and location recognition within an error range of tens of centimeters indoors is possible.

[0171] In one embodiment, the processor (120) can receive information regarding an input voice signal, a registered voice signal, registered information, etc. from an external device through a communication interface (130). For example, the processor (120) can acquire an input voice signal by receiving an input voice signal acquired from an external device including a microphone through the communication interface (130).

[0172] Additionally, the processor (120) can control the communication interface (130) to transmit an output voice signal and / or text corresponding to the output voice signal (e.g., translated text) to an external device (e.g., a user terminal such as a smartphone). Accordingly, the user can receive the output voice signal and / or text corresponding to the output voice signal through the speaker and / or display of the external device.

[0173] The input interface (140) includes a circuit, and the processor (120) can receive user commands to control the operation of the electronic device (100) through the input interface (140). Specifically, the input interface (140) may be composed of components such as a microphone, a camera, and a remote control signal receiver. Additionally, the input interface (140) may be implemented in a form included in a display as a touch screen. In particular, the microphone can receive a voice signal and convert the received voice signal into an electrical signal.

[0174] In one embodiment, the processor (120) can receive an input voice signal corresponding to user speech through a microphone.

[0175] The output interface (150) includes a circuit, and the processor (120) can output various functions that the electronic device (100) can perform through the output interface (150). Also, the output interface (150) may include at least one of a display, a speaker, or an indicator.

[0176] The display can output image data under the control of the processor (120). Specifically, the display can output an image stored in the memory (110) under the control of the processor (120). In particular, the display according to one embodiment of the present disclosure may display a user interface stored in the memory (110). The display may be implemented as an LCD (Liquid Crystal Display Panel), OLED (Organic Light Emitting Diodes), etc., and the display may also be implemented as a flexible display, a transparent display, etc. depending on the case. However, the display according to the present disclosure is not limited to a specific type.

[0177] The speaker can output audio data under the control of the processor (120). The indicator can be lit under the control of the processor (120). Specifically, the indicator can be lit in various colors under the control of the processor (120). For example, the indicator can be implemented using LEDs (Light Emitting Diodes), LCDs (Liquid Crystal Display Panels), VFDs (Vacuum Fluorescent Displays), etc., but is not limited thereto.

[0178] In one embodiment, the processor (120) may control the speaker to output an output voice signal. The processor (120) may also control the display to display text (e.g., translated text) corresponding to the output voice signal. The processor (120) may also control the speaker and the display to display text (e.g., translated text) corresponding to the output voice signal while the output voice signal is being output. Accordingly, the user may receive the output voice signal and / or text corresponding to the output voice signal through the speaker and / or display of the electronic device (100).

[0179] FIG. 8 is a flowchart illustrating a method for controlling an electronic device (100) according to one or more embodiments of the present disclosure.

[0180] The electronic device (100) can receive an input voice signal corresponding to a user's speech (S810). When an input voice signal corresponding to a user's speech is received, the electronic device (100) can obtain quality information including a score indicating the quality level of the input voice signal (S820). For example, the electronic device (100) can receive an input voice signal corresponding to a user's speech through a microphone included in the electronic device (100). The electronic device (100) may also receive an input voice signal from an external device through a communication interface (130) included in the electronic device (100).

[0181] In one embodiment, the electronic device (100) can obtain quality information based on at least one of the length of an input voice signal, the length of a voice portion included in the input voice signal, the length of a noise portion included in the input voice signal, or the amount of frequency change of the voice signal.

[0182] If the score is less than a preset first threshold value (S830-Y), the electronic device (100) can identify a registered voice signal for a user in memory (110). In one embodiment, the electronic device (100) can identify a registered voice signal for a user (i.e., a user who uttered a voice corresponding to an input voice signal) based on the similarity between a plurality of registered voice signals stored in a registration information database and an input voice signal.

[0183] When a registered voice signal for a user is identified (S840-Y), the electronic device (100) can obtain a reference voice signal by improving the quality of the input voice signal based on the registered voice signal (S850).

[0184] In one embodiment, if the score is less than a first threshold and greater than or equal to a second threshold that is smaller than the first threshold, the quality enhancement module (1130) can obtain a reference voice signal by combining the input voice signal and the registered voice signal. On the other hand, if the score is less than the second threshold, the quality enhancement module (1130) can obtain the registered voice signal as the reference voice signal.

[0185] The electronic device (100) can obtain an output voice signal for an input voice signal based on an input text corresponding to the input voice signal and a reference voice signal (S860). When the output voice signal is obtained, the electronic device (100) can provide the output voice signal. Additionally, the electronic device (100) may provide text corresponding to the output voice signal along with the output voice signal.

[0186] If the score is greater than or equal to a preset first threshold value (S830-N), the electronic device (100) can acquire an input registration signal as a reference voice signal, and based on the input text corresponding to the input voice signal and the reference voice signal, can acquire an output voice signal for the input voice signal.

[0187] When a registered voice signal for a user is identified (S840-N), the electronic device (100) can obtain the input registered signal as a reference voice signal, and based on the input text corresponding to the input voice signal and the reference voice signal, can obtain an output voice signal for the input voice signal.

[0188] According to the embodiments described above, the electronic device (100) can enhance a reference voice signal used for voice synthesis by using a voice signal input according to the user's speech and a previously stored registered voice signal. Accordingly, the electronic device (100) can obtain a synthesized voice with a significantly high degree of similarity to the actual user's unique voice.

[0189] Although various embodiments of the present disclosure have been described above on the premise of synthesizing a user's voice using a reference voice signal, the various embodiments described above can be similarly applied to technologies that use a reference voice for a user, such as voice conversion (VC) that converts a user's voice into a voice like that of another user.

[0190] The control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided on a non-transitory computer-readable medium.

[0191] Specifically, in a non-transient computer-readable recording medium comprising a program for executing a control method of an electronic device (100), the control method of the electronic device (100) may include: a step of obtaining quality information including a score indicating the quality level of the input voice signal when an input voice signal corresponding to a user's speech is received; a step of identifying a registered voice signal for the user in the memory (110) when the score is less than a preset first threshold value; a step of obtaining a reference voice signal indicating the voice characteristics of the user by improving the quality of the input voice signal based on the registered voice signal when the registered voice signal is identified; and a step of obtaining an output voice signal for the input voice signal based on an input text corresponding to the input voice signal and the reference voice signal.

[0192] Although a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling the electronic device (100) have been briefly described above, this is merely to avoid redundant descriptions, and it is obvious that various embodiments of the electronic device (100) can also be applied to a method for controlling the electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling the electronic device (100).

[0193] The artificial intelligence-related function according to the present disclosure is operated through the processor (120) and memory (110) of the electronic device (100).

[0194] The processor (120) may be composed of one or more processors (120). In this case, the one or more processors (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit), but are not limited to the examples of the processor (120) described above.

[0195] The CPU is a general-purpose processor (120) capable of performing not only general operations but also artificial intelligence operations, and can efficiently execute complex programs through a multi-layer cache structure. The CPU is advantageous for a serial processing method that enables organic linkage between previous and next calculation results through sequential calculations. The general-purpose processor (120) is not limited to the examples described above, except for cases where it is specified as the CPU described above.

[0196] A GPU is a processor (120) for large-scale computations, such as floating-point operations used in graphics processing, and can perform large-scale computations in parallel by integrating a large number of cores. In particular, a GPU may be advantageous for parallel processing methods such as convolution operations compared to a CPU. Additionally, a GPU can be used as a co-processor (120) to complement the functions of a CPU. The processor (120) for large-scale computation is not limited to the examples described above, except for cases where it is specified as the aforementioned GPU.

[0197] The NPU is a processor (120) specialized for artificial intelligence computation using an artificial neural network, and each layer constituting the artificial neural network can be implemented in hardware (e.g., silicon). At this time, since the NPU is designed to be specialized according to the specifications required by the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence computation required by the company. As a processor (120) specialized for artificial intelligence computation, the NPU can be implemented in various forms such as a TPU (Tensor Processing Unit), an IPU (Intelligence Processing Unit), a VPU (Vision Processing Unit), etc. The artificial intelligence processor (120) is not limited to the examples described above, except for cases specified as the aforementioned NPU.

[0198] Additionally, one or more processors (120) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors (120), a memory (110) and a network interface such as a bus for data communication between the processor (120) and the memory (110).

[0199] When a plurality of processors (120) are included in a System on Chip (SoC) included in an electronic device (100), the electronic device (100) can perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the processors (120) among the plurality of processors (120). For example, the electronic device (100) can perform operations related to artificial intelligence by using at least one of a GPU, NPU, VPU, TPU, or hardware accelerator specialized for artificial intelligence operations such as convolution operations or matrix multiplication operations among the plurality of processors (120). However, this is merely one embodiment, and it is obvious that operations related to artificial intelligence can be processed using a CPU or a general-purpose processor (120).

[0200] Additionally, the electronic device (100) can perform operations related to artificial intelligence functions using multi-cores (e.g., dual cores, quad cores, etc.) included in a single processor (120). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations and matrix multiplication operations in parallel using multi-cores included in the processor (120).

[0201] One or more processors (120) control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0202] Here, being created through learning means that a predefined rule of operation or an artificial intelligence model of desired characteristics is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server / system.

[0203] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the layer's operation through the result of the operation of the previous layer and at least one defined operation. Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), Deep Q-Networks, and Transformers; however, the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0204] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0205] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0206] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0207] Each component (e.g., module or program) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration.

[0208] Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0209] As used in this disclosure, the terms “part” or “module” include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A “part” or “module” may be an integrally formed part or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0210] Various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0211] When the above instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0212] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, At least one memory for storing instructions; and At least one processor including a processing circuit; comprising, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When an input voice signal corresponding to a user's utterance is received, quality information including a score indicating the quality level of the input voice signal is obtained, and If the above score is less than the first threshold value, the registered voice signal for the user stored in the memory is identified, and When the above registered voice signal is identified, a reference voice signal representing the voice characteristics of the user is obtained based on the above registered voice signal, and An electronic device that obtains an output voice signal for an input voice signal based on an input text corresponding to the input voice signal and a reference voice signal.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device for obtaining quality information based on at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, or the amount of frequency change of the input voice signal.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that acquires the input voice signal as the reference voice signal when the score is greater than or equal to the first threshold value.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that obtains a reference voice signal by combining the input voice signal and the registered voice signal when the score is less than the first threshold value and greater than or equal to the second threshold value which is smaller than the first threshold value.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that acquires the registered voice signal as the reference voice signal when the score is less than the second threshold value.

6. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, If the above score is less than the above second threshold value, noise included in the input voice signal is removed based on the above registered voice signal, and An electronic device that combines the noise-removed input voice signal and the registered voice signal to obtain the reference voice signal.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that identifies a registered voice signal among a plurality of registered voice signals based on the similarity between the plurality of registered voice signals and the input voice signal.

8. In Paragraph 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A plurality of scores representing the quality of each of the above plurality of registered voice signals are obtained, and A plurality of registered texts corresponding to each of the above plurality of registered voice signals are obtained, An electronic device for identifying a registered voice signal among a plurality of registered voice signals based on the above similarity, the plurality of scores representing the quality of each of the plurality of registered voice signals, and the type of the plurality of registered texts.

9. In Paragraph 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that, if the above registered voice signal is not identified, acquires the above input voice signal as the above reference voice signal.

10. In Paragraph 9, The above memory stores registration information including the plurality of registered voice signals and the plurality of scores, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that updates the registration information by adding the reference voice signal to the plurality of registered voice signals and adding a score indicating the level of the quality of the input voice signal to the plurality of scores.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Obtain text of a first language corresponding to the above input voice signal, and The above input text is an electronic device obtained by translating the above text into a second language.

12. In a method for controlling an electronic device, When an input voice signal corresponding to a user's utterance is received, a step of obtaining quality information including a score indicating the quality level of the input voice signal; If the score is less than a first threshold value, a step of identifying a registered voice signal for the user among a plurality of registered voice signals stored in memory; When the above-mentioned registered voice signal is identified, a step of obtaining a reference voice signal representing the voice characteristics of the user based on the above-mentioned registered voice signal; and A method comprising the step of obtaining an output voice signal for the input voice signal based on an input text corresponding to the input voice signal and the reference voice signal.

13. In Paragraph 12, The step of obtaining the above quality information is, A method comprising the step of obtaining quality information based on at least one of the length of the input voice signal, the length of the voice portion included in the input voice signal, the length of the noise portion included in the input voice signal, and the amount of frequency change of the input voice signal.

14. In Paragraph 12, A method further comprising the step of acquiring the input voice signal as the reference voice signal if the score is greater than or equal to the first threshold value.

15. In Paragraph 12, The step of acquiring the above reference voice signal is, A method comprising the step of obtaining a reference voice signal by combining the input voice signal and the registered voice signal if the score is less than the first threshold value and greater than or equal to the second threshold value which is smaller than the first threshold value.

Images