Audio cloning method based on cloud computing technology and related device

Abstract

The present application provides an audio cloning method based on a cloud computing technology and a related device, which are used for ensuring security of cloned voice and preventing user privacy leakage. The method is applied to a cloud platform. The cloud platform is used for managing infrastructure for running a cloud computing service. The infrastructure comprises at least one data center, and each data center among the at least one data center comprises a plurality of servers. The method comprises: acquiring original audio from a first tenant, encrypting the original audio to obtain encrypted audio, and storing the encrypted audio in infrastructure; acquiring a first text from the first tenant; decrypting the encrypted audio to acquire the original audio; and synthesizing the original audio and the first text to obtain cloned audio, wherein sound features of the cloned audio are the same as sound features of the original audio, and content of the cloned audio is content of the first text.

Description

An audio cloning method and related equipment based on cloud computing technology

[0001] This application claims priority to Chinese Patent Application No. 202411989738.5, filed with the State Intellectual Property Office of China on December 30, 2024, entitled "An Audio Cloning Method and Related Equipment Based on Cloud Computing Technology", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing, and in particular to an audio cloning method and related equipment based on cloud computing technology. Background Technology

[0003] With the rapid development of industries such as virtual characters, audiobooks, and video creation, voice cloning technology is gradually entering people's lives. Voice cloning is a speech algorithm technology that clones the timbre, rhythm, and style of a person's speech, meeting the needs of personalized speech synthesis.

[0004] One method for audio cloning involves adding a digital watermark to the cloned audio file. The digital watermark is embedded into the audio file using a watermark embedding algorithm, minimizing any impact on the original sound quality. By extracting the digital watermark, the copyright of the audio can be identified, thus identifying its source.

[0005] This method, based on digital watermarking, can only identify the source of the cloned audio. No protection measures are in place for the security of the original audio, resulting in a high risk of the original audio being stolen and poor privacy. Summary of the Invention

[0006] This application provides an audio cloning method and related equipment based on cloud computing technology, which is used to ensure the security of the cloned original audio and prevent the leakage of user privacy.

[0007] Firstly, this application provides an audio cloning method based on cloud computing technology. This method is applied to a cloud platform, which manages the infrastructure running cloud computing services. The infrastructure includes at least one data center, where each data center includes multiple servers. The method includes:

[0008] The cloud platform obtains the original audio from the first tenant, which can be understood as the cloned audio. The first tenant is the user's account on the cloud platform, through which the user uploads the original audio. The account can be an account already registered on the cloud platform, or an unregistered guest account, etc., without specific limitations here. The cloud platform encrypts the original audio to obtain encrypted audio, which is then stored in the infrastructure. The cloud platform also obtains the first text from the first tenant; the first text is the content of the cloned audio. The cloud platform decrypts the encrypted audio to obtain the original audio, facilitating the extraction of the original audio's sound features for audio cloning. In other words, the cloud platform synthesizes the original audio and the first text to obtain the cloned audio. The sound features of the cloned audio are identical to those of the original audio, and the content of the cloned audio is the content of the first text.

[0009] In this application, the audio to be cloned is encrypted. For the audio cloning process, the input is encrypted audio, which is then decrypted before cloning, ensuring the security of voice input during the audio cloning process. This also guarantees the security of the cloned voice and prevents the leakage of user privacy.

[0010] In some alternative implementations of the first aspect, the cloud platform determines the acoustic features of the original audio to conform to a first preset specification before synthesizing the original audio and the first text. The first preset specification includes matching the acoustic features with the tenant.

[0011] In this application, the original audio cloned by the cloud platform is the audio that matches the tenant, thus avoiding the cloning of mismatched audio and avoiding infringement of the rights of others.

[0012] In some optional implementations of the first aspect, the original audio is audio that conforms to the first preset definition, meaning there are multiple possibilities for the sound features of the original audio to conform to the first preset definition. The cloud platform matches the sound features of the original audio against a feature library to identify whether the original audio conforms to the first preset definition. If the feature library includes a first sound feature among multiple sound features, and this first sound feature matches the sound features of the original audio, and the feature library also includes an association between the first sound feature and the identifier of the first tenant, this means that the tenant associated with the first sound feature is the first tenant that transmitted the original audio, and the cloud platform determines that the sound features of the original audio conform to the first preset definition.

[0013] In this application, in a scheme where the feature library includes a first sound feature that matches the sound features of the original audio, and the first sound feature is associated with a first tenant that transmitted the original audio, the cloud platform determines that the sound features of the original audio conform to a first preset specification. In other words, the cloud platform determines that the sound features of the original audio in the feature library are sound features associated with the first tenant recorded in the feature library, and the upload tenant of the original audio is the first tenant, which means that no one else's sound features have been stolen. In this scenario, the cloud platform directly performs audio cloning operations without infringing on the interests of others. In some optional implementations of the first aspect, the cloud platform matches the sound features of the original audio in the feature library to identify whether the original audio conforms to the first preset specification. If none of the multiple preset sound features included in the feature library match the sound features of the original audio, the cloud platform can also determine that the sound features of the original audio conform to a second preset specification.

[0014] In this application, if the sound features of the original audio do not match multiple preset sound features in the feature library, it indicates that the sound features of the original audio are not in the feature library. They may be sound features not yet added to the feature library, or sound features associated with a new tenant. In this solution, it can be assumed that the sound features of the original audio match the first tenant who uploaded the original audio, without affecting the interests of the tenants associated in the feature library. Furthermore, this enriches the implementation methods and application scenarios of the technical solution in this application, further enhancing the flexibility of the technical solution.

[0015] In some alternative implementations of the first aspect, the cloud platform can also respond to feature addition instructions for the original audio, adding the original audio sound features to the feature library. It also stores the association between the sound features of the original audio and the first tenant in the feature library.

[0016] In this application, the sound features of the original audio can be added to a feature library, thereby providing protection for the sound features of the original audio.

[0017] In some optional implementations of the first aspect, before synthesizing the original audio and the first text, the cloud platform performs semantic analysis on the first text to determine that the first text conforms to a second preset specification. The second preset specification includes at least one of the following: the content of the text does not contain sensitive words. The second preset specification may also include: the content of the text is coherent and the semantics of the text are clear. In this application, the text synthesized by the cloud platform during the audio cloning process conforms to the second preset specification. The second preset specification has multiple possibilities, enriching the implementation methods and application scenarios of the technical solution of this application.

[0018] In some alternative implementations of the first aspect, the cloud platform obtains a second text from the first tenant before obtaining the first text, the content of which does not conform to the second preset requirement. In this solution, the cloud platform responds to a modification instruction for the second text, modifies the content of the second text, and obtains a first text that conforms to the second preset requirement.

[0019] In this application, for text that does not conform to the second preset requirement, the text content can be modified so that the modified text conforms to the second preset requirement, thereby ensuring that the content after voice cloning conforms to the second preset requirement.

[0020] In some optional implementations of the first aspect, in schemes that meet preset conditions, the cloud platform also adds an audio identifier to the cloned audio. This audio identifier indicates the first tenant, i.e., the source of the cloned audio. The preset conditions include at least one of the following: the sound features of the original audio match a second sound feature in the feature library, and the second sound feature in the feature library is not associated with the first tenant. Alternatively, the first text is text that does not conform to the second preset requirement. Wherein, if the sound features of the original audio match a second sound feature in the feature library associated with the second tenant, it indicates that the original audio uploaded by the first tenant may be someone else's voice, posing a risk of cloning someone else's voice. If the first text does not conform to the second preset requirement, it means that the first text may contain sensitive words, incoherent content, or unclear semantics, resulting in the cloned audio containing sensitive or unclear content, affecting the cloning effect.

[0021] In this application, an audio identifier is added to the cloned audio when preset conditions are met to mark the source of the cloned audio and facilitate the identification of the synthesizer. If the cloned audio infringes on the rights of others, evidence can be obtained based on this identifier. In addition, text that does not meet the second preset requirement can also be synthesized with the original audio to generate cloned audio, further enriching the application scenarios of the technical solution of this application.

[0022] In some optional implementations of the first aspect, a feature library is stored in the infrastructure managed by the cloud platform. The feature library includes multiple preset voice features. For some or all of the multiple preset voice features, the feature library also includes the identifier of the associated tenant. A tenant's identifier can be associated with at least one preset voice feature, and a preset voice feature is associated with one tenant's identifier. For example, in a home account or enterprise account scenario, one account identifier is associated with multiple preset voice features.

[0023] In this application, the association between the preset sound features in the feature library and the tenant's identifier can be varied, which enriches the implementation methods of the technical solution of this application.

[0024] Secondly, this application provides a cloud platform for managing the infrastructure running cloud computing services. The infrastructure includes at least one data center, and each of the at least one data center includes multiple servers. The cloud platform includes:

[0025] The acquisition unit is used to acquire the original audio from the first tenant, encrypt the original audio to obtain encrypted audio, and store the encrypted audio in the infrastructure. It also acquires the first text from the first tenant.

[0026] The processing unit is used to decrypt the encrypted audio to obtain the original audio. The original audio and the first text are then synthesized to obtain a cloned audio. The cloned audio has the same sound characteristics as the original audio, and its content is the same as the first text.

[0027] The cloud platform is used to implement the methods shown in the first aspect or any possible implementation of the first aspect, as detailed above, and will not be repeated here.

[0028] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the method disclosed in the first aspect or any possible implementation of the first aspect.

[0029] Fourthly, this application provides a computer program product containing instructions that, when executed on a processor, implement the method shown in the first aspect or any possible implementation of the first aspect; or, when the instructions are run by a cluster of computer devices, cause the cluster of computer devices to implement the method disclosed in the first aspect or any possible implementation of the first aspect.

[0030] Fifthly, this application provides a computer-readable storage medium storing computer program instructions that, when executed on a processor, implement the method described in the first aspect or any possible implementation of the first aspect; or, when executed by a cluster of computer devices, cause the cluster of computer devices to implement the method disclosed in the first aspect or any possible implementation of the first aspect.

[0031] The beneficial effects shown in any of the second to fifth aspects are similar to those of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0032] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0033] Figure 2 is a flowchart illustrating an audio cloning method based on cloud computing technology provided in an embodiment of this application.

[0034] Figure 3 is a schematic diagram of an interface provided in an embodiment of this application;

[0035] Figure 4 is another schematic diagram of an interface provided in an embodiment of this application;

[0036] Figure 5 is another schematic diagram of an interface provided in an embodiment of this application;

[0037] Figure 6 is a schematic diagram of the audio cloning system provided in an embodiment of this application;

[0038] Figure 7 is another schematic diagram of an interface provided in an embodiment of this application;

[0039] Figure 8 is a structural schematic diagram of an audio cloning device provided in an embodiment of this application;

[0040] Figure 9 is a schematic diagram of a computing device provided in an embodiment of this application;

[0041] Figure 10 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0042] Figure 11 is another schematic diagram of the computing device cluster provided in the embodiment of this application. Detailed Implementation

[0043] This application provides an audio cloning method and related equipment based on cloud computing technology to ensure the security of the cloned original audio and prevent user privacy leakage.

[0044] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0045] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0046] First, please refer to Figure 1, which is a schematic diagram of the system architecture provided in the embodiment of this application.

[0047] As shown in Figure 1, users log in to the cloud service platform via the internet using their registered account and password. Alternatively, users can use a guest account provided by the cloud service platform to experience its cloud services via the internet.

[0048] A cloud service platform, also known as a cloud platform, is used to manage the infrastructure that runs cloud computing services. This infrastructure includes at least one data center located in different regions. For example, Region 1 in Figure 1 includes Cloud Data Center 1 and Cloud Data Center 2, and Region 2 includes Cloud Data Center 3 and Cloud Data Center 4. Each cloud data center has multiple servers, and these servers run business instances (including at least one of virtual machines, containers, and dedicated servers).

[0049] In this embodiment, an audio cloning service is deployed in the business instance. Users purchase cloud services through a cloud service platform via their terminal devices and send a request to the cloud service platform to request the audio cloning service. Alternatively, users can experience the audio cloning service through guest accounts, etc. The specific implementation process of the audio cloning service will be described in detail later.

[0050] In summary, the audio cloning method provided in this application can be applied to servers, virtual machines, cloud service platforms, etc. In the following description, a cloud platform will be used as the execution subject. The implementation process for other types of execution subjects is similar.

[0051] It should be noted that the terminal device mentioned in the embodiments of this application can be a device with wireless transceiver function, or a terminal. Specifically, it can refer to user equipment (UE), access terminal, subscriber unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal devices can also be satellite phones, cellular phones, smartphones, wireless data cards, wireless modems, machine-type communication devices, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, communication devices mounted on high-altitude aircraft, wearable devices, drones, robots, terminals in device-to-device (D2D) communication, terminals in vehicle-to-everything (V2X) communication, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, mixed reality (MR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, and wireless terminals in transportation safety. This application does not limit the scope of wireless terminals, such as those used in smart cities, smart homes, or future communication networks.

[0052] In addition, in the embodiments of this application, there may be more or fewer terminal devices in the system. The number and type of terminal devices are determined according to actual needs, and are not limited here.

[0053] Please refer to Figure 2 below. Figure 2 is a flowchart illustrating the audio cloning method based on cloud computing technology provided in the embodiments of this application.

[0054] 201. The cloud platform obtains the original audio from the first tenant, encrypts the original audio to obtain encrypted audio, and stores the encrypted audio in the infrastructure.

[0055] The first tenant is the user's account on the cloud platform, including accounts already registered on the cloud platform, as well as unregistered accounts such as guest accounts and trial accounts. Users upload raw audio to the cloud platform through this account. The raw audio is the audio whose sound is to be cloned.

[0056] Optionally, the terminal device running the first tenant sends a command to the cloud platform, instructing the cloud platform to retrieve the raw audio. The storage location of the raw audio can be varied: it can be stored on the terminal device, server, cloud, or other devices that have established a communication connection with the cloud platform and can store audio; no specific limitation is made here.

[0057] Optionally, the original audio can be real-time recorded audio. For example, the user operates on the interface displayed on the terminal device and starts real-time recording. Optionally, the original audio can also be preset audio. For example, the aforementioned terminal device sends an instruction to the cloud platform, instructing the cloud platform to retrieve the original audio stored at the target storage location.

[0058] After the cloud platform obtains the original audio, it encrypts it to obtain encrypted audio. Compared to the original audio, the encrypted audio obscures the sound information and / or content information of the original audio.

[0059] In practical applications, there are various possibilities for encrypting the original audio. For example, the original audio can be timbre-converted to change its sound information; or noise can be added to the original audio to obscure its content and timbre. Furthermore, other methods can be used to encrypt the original audio, such as adding a watermark or performing signal calculations, etc., but these are not limited here.

[0060] Optionally, in practical applications, a certain encryption method can be set as the default encryption method, and the cloud platform encrypts the original audio based on the default encryption method. Alternatively, multiple encryption schemes can be provided. The terminal device responds to the user's operation command by sending a command to the server, which instructs the cloud platform to encrypt the original audio based on the encryption method selected by the user.

[0061] Optionally, in scenarios where multiple encryption schemes are provided but the user has not selected one, the cloud platform can encrypt the original audio based on any one of the multiple encryption methods, or it can encrypt the original audio based on the default encryption method.

[0062] It is important to note that the cloud platform stores encrypted audio on its infrastructure, rather than the original audio, which means that the input for audio cloning is also encrypted audio, thus improving the security of audio cloning.

[0063] 202. The cloud platform retrieves the first text from the first tenant.

[0064] The content of the first text is the content of the cloned audio. In some optional implementations, the first text is text that conforms to a second preset specification. That is, after the cloud platform obtains the first text, it performs semantic recognition on the first text to determine that the first text conforms to the second preset specification. The second preset specification includes at least one of the following: the content of the text does not contain sensitive words. The second preset specification may also include: the content of the text is coherent, or the semantics of the text are clear.

[0065] Sensitive words include those pre-defined by the cloud platform, such as words that violate public order and good morals, natural laws, or legal regulations. They may also include tenant-defined words. Text coherence means the text is fluent and conforms to the rules of natural language expression. Semantic clarity means the text is unambiguous (e.g., free of typos, misleading descriptions), thus preventing cloned audio from misleading others.

[0066] Furthermore, there are several possible ways for cloud platforms to perform semantic recognition on the first text:

[0067] Optionally, the cloud platform can perform word segmentation on the first text based on its semantics, obtaining multiple words. These multiple words are then compared with sensitive words in a sensitive word database. In cases where none of the words in the first text match any sensitive words, the content of the first text is determined to be free of sensitive words. The sensitive word database includes multiple sensitive words and can be stored on servers, shared storage spaces across multiple devices, or dedicated databases—any other devices accessible to the cloud platform. The sensitive words included in the database can be user-defined or determined based on system parameters; specific limitations are not specified here.

[0068] Optionally, the cloud platform can perform semantic analysis on the first text, dividing it into multiple sentences and identifying whether each sentence is coherent or unclear. In cases where multiple sentences are coherent or semantically clear, the platform determines whether the first text is coherent or semantically clear.

[0069] The first text that conforms to the second preset requirement, as determined by the cloud platform, may be the initial text obtained by the cloud platform or it may be a modified text. The possible scenarios are explained below:

[0070] In some optional implementations, the first text is the initial text obtained by the cloud platform, which can also be understood as the initial text uploaded by the first tenant. This means that after the cloud platform detects the initial text, it determines that the text conforms to the second preset requirement. It can then directly proceed to the next process without needing to provide information on whether the tenant has modified the text content.

[0071] In some alternative implementations, the first text is the modified text. Specifically, before obtaining the first text, the cloud platform obtains the second text from the first tenant. The cloud platform performs semantic recognition on the second text and determines that the second text does not conform to the second preset specification. In this scheme, the cloud platform also responds to a modification instruction for the second text, modifying the content of the second text to obtain the first text that conforms to the second preset specification.

[0072] Furthermore, there are several possibilities for the second text not conforming to the second preset requirement:

[0073] Optionally, the second text may fail to comply with the second presupposition, including: the second text containing sensitive words. For example, the second text containing words that violate public order and good morals. Another example is that the second text contains advertising slogans that include words such as "best" or "most desirable," which violate relevant provisions of the Advertising Law.

[0074] Optionally, the second text not conforming to the second preset requirement includes: the content of the second text is incoherent. For example, the second text is incomplete in meaning or logically inconsistent. Another example is that the content of the second text jumps between paragraphs or lacks contextual connection.

[0075] Optionally, the second text not conforming to the second preset requirement includes: the second text having unclear semantics, being ambiguous, or being misleading. For example, the second text includes typos, or its logic is flawed. Another example is that the second text includes malicious jokes.

[0076] Optionally, the second text not conforming to the second preset requirement includes: although the second text does not contain sensitive words, it includes homophones of sensitive words. Since the purpose of this application's technical solution is to perform audio cloning, the final result is cloned audio. Words that sound like sensitive words may not be sensitive in the text, but they are indistinguishable from sensitive words in terms of sound. Therefore, if the second text contains homophones of sensitive words, it can also be considered that the second text does not conform to the requirement.

[0077] It should also be noted that the second text may be the initial text uploaded by the first tenant, or it may be a modified version of the initial text that still does not conform to the second preset rule.

[0078] For a second text that does not comply with the regulations, the cloud platform may obtain modification instructions for the second text in several ways.

[0079] Optionally, for any non-compliant second text, the cloud platform sends a notification to the terminal device, indicating that the second text does not conform to the second preset specification. The terminal device receives the notification and prompts the user to modify the text. In response to the user's action, the terminal device sends a modification instruction to the cloud platform for the second text, causing the cloud platform to modify the second text based on this instruction, resulting in a first text that conforms to the second preset specification.

[0080] The following description of this scheme is based on a schematic diagram. Please refer to Figure 3, which is a schematic diagram of the interface provided in an embodiment of this application.

[0081] The terminal device displays the interface shown in Figure 3. The user clicks control 301 to modify the content of the second text until a first text conforming to the second preset specification is obtained, and then sends an instruction to the cloud platform. This instruction instructs the cloud platform to obtain the first text conforming to the second preset specification based on the modification operation.

[0082] It should be noted that Figure 3 is merely a schematic representation of the interface and does not constitute a limitation on the interface displayed in actual applications. Optionally, in actual applications, the interface and controls may also display other content, such as displaying the prompt message "The generated text content is risky, do you want to continue cloning?", and displaying the "No" and "Yes" controls. Optionally, the interface may also display specific content of the second text that does not conform to the second preset regulations, such as displaying sensitive words included in the second text, sentences with unclear meanings in the second text, etc., making it more convenient for users to modify. This application does not limit the specific form of the interface.

[0083] Optionally, in the interface shown in Figure 3, for solutions where the second text does not conform to the second preset specification, the user is given the option to modify the text content. In practical applications, the interface can also only display the default controls, such as only displaying control 301, thereby ensuring that the text of the cloned audio conforms to the second preset specification.

[0084] Optionally, for a second text that does not conform to the specifications, the cloud platform can also obtain a default instruction. This default instruction instructs the modification of the second text's content to obtain a first text that conforms to the second preset specifications. In this scheme, the modification of the second text's content is performed automatically by the cloud platform, without the user's awareness. After obtaining the first text that conforms to the second preset specifications, the cloud platform can display the content of the first text through the terminal device, and can also display the location of the modification, which the user then confirms. Alternatively, after modification, the modification result is returned to the terminal device, and the user confirms it through the terminal device. This ensures that the cloud platform's modifications are confirmed by the user, enhancing the user experience.

[0085] In this embodiment, the text synthesized by the cloud platform during the audio cloning process conforms to a second preset specification. This second preset specification has multiple possibilities, enriching the implementation methods and application scenarios of the technical solution in this application. Furthermore, for text that does not conform to the second preset specification, the text content can be modified to make the modified text conform to the second preset specification, thereby ensuring that the content after voice cloning conforms to the second preset specification.

[0086] 203. The cloud platform decrypts the encrypted audio to obtain the original audio.

[0087] The cloud platform decrypts the encrypted audio to restore the original audio. The method by which the server decrypts the audio corresponds to the method of encryption, which will not be elaborated here.

[0088] 204. The cloud platform synthesizes the original audio and the first text to obtain a cloned audio. The sound features of the cloned audio are the same as those of the original audio, and the content of the cloned audio is the content of the first text.

[0089] The cloud platform restores the original audio and extracts its sound features using a neural network. Based on a cloning model, the sound features of the original audio and the first text are synthesized to obtain cloned audio. The synthesis operation can be understood as adding the sound features of the original audio to the text features of the first text, thus obtaining cloned audio with the content of the first text and the sound features of the original audio.

[0090] Voice features, including voiceprint information, are unique; each user's voiceprint is different. In terms of auditory perception, voiceprint information manifests as timbre, rhythm, and style. For example, tone of voice and dialect can be considered rhythm. Speaking habits and pauses can be considered style. Timbre can be used to distinguish different speakers. In the field of computer science, the voice features obtained by neural networks processing audio can be represented as vectors or tensors.

[0091] In some optional implementations, the sound features of the original audio are sound features that conform to a first preset definition. That is, after the cloud platform obtains the sound features of the original audio, it determines that the sound features of the original audio conform to the first preset definition by matching the sound features of the original audio with multiple preset sound features in the feature library.

[0092] The first pre-defined rule includes: matching sound characteristics with tenants. It emphasizes that the sound characteristics of the tenant uploading audio must match those of the audio uploaded by that tenant. In other words, the sound characteristics of the tenant triggering the cloud platform to retrieve the audio must match those of the audio retrieved by the cloud platform.

[0093] Furthermore, there are several possibilities for the original audio's sound characteristics to conform to the first preset specification:

[0094] In some optional implementations, among the multiple preset sound features included in the feature library, there exists a first sound feature that matches the sound features of the original sound and is associated with the first tenant. In this scheme, the cloud platform matches the sound features of the original audio in the feature library, identifies the first sound feature, and determines that the sound features of the original audio conform to the first preset requirement.

[0095] The association between the first tenant and the first voice feature can also be understood as the feature library including the association between the first voice feature and the identifier of the first tenant. Matching the voice features of the original voice with the first voice feature means either the original voice features are identical to the first voice feature, or the error between the original voice features and the first voice feature is within an error range. The error range can be defined by the user, based on system parameter settings, or set with reference to historical experience, and is determined based on the needs of the actual application; specific details are not limited here.

[0096] Additionally, the first tenant's identifier is used to uniquely identify the first tenant, including account identifiers or device identifiers, etc., without specific limitations here. An account identifier can be a username, user identification code, or other identifier uniquely identifying the account. A device identifier can be a device serial number, internet protocol (IP) address, media access control (MAC) address, or other identifier uniquely identifying the device.

[0097] In this embodiment, where the feature library includes a first sound feature that matches the sound features of the original audio, and the first sound feature is associated with a first tenant that transmitted the original audio, the cloud platform determines that the sound features of the original audio conform to a first preset specification. In other words, the cloud platform determines that the sound features of the original audio in the feature library are sound features associated with the first tenant recorded in the feature library, and the uploader of the original audio is the first tenant, which means that no one else's sound features have been stolen. In this scenario, the cloud platform directly performs audio cloning operations without infringing on the interests of others.

[0098] In some optional implementations, none of the preset sound features included in the feature library match the sound features of the original audio. In this solution, the cloud platform matches the sound features of the original audio against the feature library to determine if the sound features of the original audio conform to a first preset requirement.

[0099] Among these, a mismatch between the preset sound features and the sound features of the original audio includes situations where the sound features of the original audio are different from the preset sound features, or where the error between the sound features of the original audio and the preset sound features is outside the error range. The error range can be defined by the user, based on system parameter settings, or set with reference to historical experience, and is determined based on the needs of the actual application; no specific limitations are specified here.

[0100] In this embodiment, if the sound features of the original audio do not match multiple preset sound features in the feature library, it indicates that the sound features of the original audio are not in the feature library. They may be sound features not yet added to the feature library, or sound features associated with a new tenant. In this solution, assuming that the sound features of the original audio match those of the first tenant who uploaded the original audio will not affect the interests of the tenants associated with it in the feature library. Furthermore, this enriches the implementation methods and application scenarios of the technical solution, further enhancing its flexibility. In summary, in the solution where the original audio cloned on the cloud platform matches the tenant, cloning mismatched audio is avoided, thus preventing infringement of others' rights.

[0101] In this application, the audio to be cloned is encrypted. For the cloning process, the input is encrypted audio, which is then decrypted before cloning, ensuring the security of voice input during the audio cloning process. This also guarantees the security of the cloned voice and prevents user privacy leaks. Furthermore, the original audio is encrypted during the audio input stage, and decrypted during cloning to restore the original audio, facilitating the extraction of its sound features. The synthesized audio features are the same as those of the original audio and the first text, ensuring that the sound features of the cloned audio are identical to those of the original audio, thus guaranteeing the cloning effect.

[0102] In the foregoing embodiments, a scheme was described in which the first text conforms to a second preset specification and the sound features of the original audio conform to the first preset specification. In practical applications, the audio cloning method provided in this application may synthesize a first text that does not conform to the second preset specification, or the sound features of the synthesized original audio may not conform to the first preset specification. The possible situations are described below.

[0103] In some alternative implementations, the first text is text that does not conform to the second preset specification. In this approach, the initial text uploaded by the first tenant is text that does not conform to the second preset specification, or the initial text, even after modification, still does not conform to the second preset specification. To respect user experience, the cloud platform can also synthesize text that does not conform to the second preset specification to obtain cloned audio that includes that text content.

[0104] For example, text that does not conform to the second preset definition might be incoherent text. A user's need might be to clone audio including phrases, word groups, etc. In this approach, text that does not conform to the second preset definition does not affect the user experience.

[0105] For example, text that does not conform to the second preset specification may include text containing typos. In the case of voice cloning, typos may not be ambiguous in terms of auditory perception and may not need to be corrected.

[0106] For example, in the embodiment shown in Figure 3, when a user clicks control 302, the terminal device responds to the operation and sends an instruction to the cloud platform, so that the cloud platform determines that the first text does not conform to the second preset specification.

[0107] In summary, the first text synthesized by the cloud platform can either conform to or not conform to the second preset specification, which enriches the implementation methods and application scenarios of the technical solution of this application.

[0108] In some optional implementations, the sound features of the original audio are sound features that do not conform to the first preset definition, that is, the sound features of the original audio are not associated with the first tenant. Specifically, this means that among the multiple preset sound features in the feature library, there exists a second sound feature that matches the sound features of the original audio but is not associated with the first tenant.

[0109] The statement that the second voice feature is not associated with the first tenant means that the feature library does not include any association between the second voice feature and the identifier of the first tenant. Several possible solutions exist:

[0110] Optionally, the tenant identifier associated with the second sound feature in the feature library is not the identifier of the first tenant. In other words, the second sound feature is associated with another tenant. This means that the first tenant currently uploading or accessing the original audio is not the tenant recorded in the feature library as associated with the sound features of the original audio. There are several possible reasons for this, such as the first tenant cloning someone else's audio, or the first tenant's audio being cloned by someone else.

[0111] Optionally, the tenant identifier associated with the second voice feature may not be included in the feature library. For example, the second voice feature could be the voice of a public figure or another person whose audio is easily accessible. In this approach, the second voice feature is easily accessible, and adding it to the feature library can also provide protection.

[0112] In this application embodiment, the first text synthesized by the cloud platform may not conform to the second preset definition, or the sound features of the synthesized original audio may not conform to the first preset definition. In this case, there is a risk of infringing on the interests of others. For these scenarios, this application embodiment can also provide protective measures to achieve traceability of cloned audio. The following is a detailed description.

[0113] In summary, in a scheme that meets the preset conditions, the cloud platform adds an audio identifier to the cloned audio, which is used to indicate the first tenant. That is, the audio identifier is used to indicate the source of the cloned audio.

[0114] The preset conditions include at least one of the following: the sound features of the original audio do not conform to the first preset definition, or the first text may not conform to the second preset definition. The original audio's sound features not conforming to the first preset definition means that the sound features of the original audio match the second sound features in the feature library, and the second sound features in the feature library are not associated with the first tenant. The second preset definition includes at least one of the following: the text content does not contain sensitive words, the text content is coherent, or the text semantics are clear.

[0115] An audio identifier is added to the cloned audio to trace its synthesizer. This audio identifier can take several forms: it can be a digital watermark associated with the user account (e.g., the first tenant) using the audio cloning service, or with the terminal device using the service. It can also be an account identifier, device identifier, etc., without specific limitations here. An account identifier can be a username, user identification code, or other unique identifier for the account. A device identifier can be a device serial number, internet protocol (IP) address, media access control (MAC) address, or other unique identifier for the device.

[0116] In addition, the feature library includes multiple preset voice features, including user-uploaded voice features, cloud platform-preset voice features, and voice features of public figures. The feature library can be stored on servers, in the cloud, in a dedicated database, or on other devices accessible by the cloud platform; specific locations are not limited here.

[0117] In this application, an audio identifier is added to cloned audio when preset conditions are met to indicate the source of the cloned audio and facilitate the identification of the synthesizer of the cloned audio. If the cloned audio infringes on the rights of others, such as by stealing others' audio or generating cloned audio with non-compliant content, evidence can be obtained based on this identifier.

[0118] In this embodiment of the application, during the process of voice cloning, the cloud platform can also perform different operations based on the different results obtained by matching the sound features of the original audio with the feature library.

[0119] In some optional implementations, in the scheme where the sound features of the original audio are matched with the second sound features in the feature library, the cloud platform outputs first information indicating that the sound features of the original audio do not match the first tenant. For example, please refer to Figure 4, which is a schematic diagram of the interface provided in an embodiment of this application.

[0120] As shown in Figure 4, in the scheme of matching the sound features of the original audio with the second sound features in the feature library, the cloud platform sends the first message to the terminal device, displaying the interface shown in Figure 4. This interface includes the prompt message "The original audio has a risk of infringement," indicating that the sound features of the original audio do not match the first tenant. This could be because the original audio might be the voice of a public figure or a voice registered by another user. Furthermore, the interface includes controls 401 and 402, allowing users performing audio cloning to make their selections only after understanding the infringement risks.

[0121] For example, suppose the original audio uploaded by the first tenant is indeed their own voice, and the user has not registered that voice in the feature database. Based on this prompt, the user can request the registrant of the voice from the service provider to confirm whether their voice has been stolen.

[0122] In some optional implementations, where the sound features of the original audio do not match any of the preset sound features in the feature library, the cloud platform outputs second information indicating whether to add the sound features of the original audio to the feature library. For example, please refer to Figure 5, which is a schematic diagram of the interface provided in an embodiment of this application.

[0123] As shown in Figure 5, in a scenario where the sound features of the original audio do not match those in the feature library, the cloud platform sends a second message to the terminal device, displaying the interface shown in Figure 5. This interface includes the message "No original audio detected in the feature library," indicating that the sound features of the original audio are not included in the feature library. Controls 501 and 502 are displayed, providing the user with a selection option.

[0124] Optionally, when the user clicks control 501, the terminal device responds by sending a feature addition instruction to the cloud platform. This instruction directs the addition of the original audio's sound features to the feature library. The cloud platform responds to the feature addition instruction and adds the original audio's sound features to the feature library. Furthermore, the cloud platform can also associate the original audio's sound features with a first tenant, storing the association between the original audio's sound features and the first tenant's identifier in the feature library, thus achieving the binding of the original audio's sound features to the first tenant.

[0125] In this embodiment of the application, the sound features of the original audio can be added to the feature library, thereby providing protection for the sound features of the original audio.

[0126] In some alternative implementations, if the user account that uploaded the original audio is not the user account bound to the sound feature, the server can also send a notification message to the bound user account or the communication account (such as a mobile phone number, email address, etc.) associated with the bound user account to notify the user that their voice has been used by someone else.

[0127] Optionally, when the user clicks control 502, the terminal device responds by sending a hold instruction to the cloud platform. The hold instruction indicates that the sound features of the original audio should not be added to the feature library. The cloud platform responds with a second instruction, keeping the feature library unchanged.

[0128] In the example shown in Figure 5 above, the scheme of adding the sound features of the original audio to the feature library is triggered during the audio cloning process. In practical applications, this scheme can also be triggered in other scenarios. For example, when a user uploads the original audio, the upload interface displays an option to register the original audio, and the terminal device responds to the user's operation command by sending the aforementioned feature addition command or retention command to the cloud platform. Alternatively, after obtaining the cloned audio, the cloned audio display interface displays an option to register the original audio, and the terminal device responds to the user's operation command by sending a feature addition command or retention command to the cloud platform. In the embodiments of this application, the sound features of the original audio can be added to the feature library based on a single command, providing protection for the sound features in the feature library.

[0129] Please refer to Figure 6 below, which is a schematic diagram of an audio cloning system provided in an embodiment of this application. As shown in Figure 6, the audio cloning system includes an encryption module, a text detection module, and an audio cloning model.

[0130] The encryption module is used to encrypt the acquired raw audio, resulting in encrypted audio. During the training of the audio cloning model, the encryption module can acquire the raw audio periodically or periodically. For example, it can acquire the raw audio once for each training iteration.

[0131] The text detection module is used to detect the initial text and identify whether the initial text conforms to the second preset criteria. It can also derive a first text based on the initial text, wherein the first text includes text conforming to the second preset criteria, or may include text conforming to the second preset criteria.

[0132] The audio cloning model includes a decryption module, a sound feature matching module, and an identifier addition module.

[0133] During inference, the decryption module decodes the encrypted audio and extracts the acoustic features of the original audio from the decoded original audio. During model training, the decryption module learns the acoustic features of the encrypted data and performs backpropagation by calculating the distance loss between the acoustic features and those of the original data. Optionally, the decryption module can be jointly trained with the sound feature matching module.

[0134] The sound feature matching module is used not only to identify whether the sound features of the original audio meet the first preset criteria, but also to store a feature library. Specifically, the sound feature matching module encodes the sound features of the original audio into an original feature vector, calculates the distance between the original feature vector and the feature vectors in the feature library, and if the distance is less than a threshold, it determines that the sound features of the original audio are included in the feature library. This distance can be a cosine distance or the distance of other representation vectors; no specific limitation is made here. Alternatively, it can also identify whether the sound features of the original audio are included in the feature library through other methods, such as calculating the similarity between the original feature vector and the feature vectors in the feature library, and determining that the sound features of the original audio are included in the feature library if the similarity is greater than a threshold. This similarity calculation can be a mirror cosine similarity or the similarity of other representation vectors; no specific limitation is made here. This application does not limit the specific implementation method for determining whether the sound features of the original audio are included in the feature library.

[0135] The identifier addition module is used to add audio identifiers to the generated cloned audio in schemes that meet preset conditions, in order to indicate the source of the cloned audio. The preset conditions are as described above and will not be repeated here.

[0136] The audio cloning system is used to implement the cloud-based audio cloning method provided in the embodiments of this application, as detailed above, and will not be repeated here.

[0137] Please refer to Figure 7, which is a schematic diagram of the interface provided in an embodiment of this application.

[0138] The interface shown in Figure 7 can be called the upload interface, which includes: audio upload area, text upload area, encryption area, and registration area.

[0139] The terminal device responds to the user's operation on the control in the audio upload area, acquires the original audio, and transmits the original audio to the cloud platform. Optionally, the user can click the "Start Recording" control to record in real time. In this scheme, the original audio is the audio recorded in real time. Optionally, the user can also click the ">" control in Figure 7, and the terminal device will display an audio selection area based on this operation. This area includes at least one audio file, which may be stored on the terminal device, the cloud platform, or other devices accessible by the cloud platform.

[0140] The terminal device responds to the user's operation on the control in the text upload area, retrieves the text, and transmits the text to the cloud platform. This text may or may not conform to the second preset criteria.

[0141] The terminal device responds to the user's operation on the encrypted area controls, determines the encryption method for the original audio, and transmits this method to the cloud platform. This allows the cloud platform to encrypt the original audio based on the user-selected encryption method. During the audio cloning process, the cloud platform decrypts the audio using the corresponding decryption method. In the embodiment shown in Figure 7, the encryption method for the original audio is timbre encryption.

[0142] The terminal device responds to the user's operation on the controls in the registration area by sending a feature addition command or a retention command to the cloud platform, causing the cloud platform to add or exclude the timbre features of the original audio from the feature library. In the embodiment shown in Figure 7, the terminal device sends a feature addition command to the cloud platform, and the cloud platform responds to the feature addition command by adding the timbre features of the original audio to the feature library.

[0143] It should be noted that Figure 7 is only an example of the interface and does not constitute a limitation on the interface displayed by the audio cloning method provided in the embodiments of this application. In practical applications, the interface may include more or fewer controls, and there are many possibilities for the arrangement of each control.

[0144] For example, the audio upload area, text upload area, encryption area, and registration area shown in Figure 7 can be displayed sequentially. The interface displays one area at a time, and after the operation of one area is completed, the next area is displayed.

[0145] Please refer to Figure 8 below, which is a schematic diagram of the structure of a cloud platform provided in an embodiment of this application. The cloud platform is used to manage the infrastructure for running cloud computing services. The infrastructure includes at least one data center, and each data center includes multiple servers. As shown in Figure 8, the cloud platform 800 includes an acquisition unit 801 and a processing unit 802.

[0146] In some optional implementations, the acquisition unit 801 is used to acquire the original audio from the first tenant, encrypt the original audio to obtain encrypted audio, and store the encrypted audio in the infrastructure. It also acquires the first text from the first tenant.

[0147] Processing unit 802 is used to decrypt the encrypted audio to obtain the original audio. The original audio and the first text are then synthesized to obtain a cloned audio. The sound characteristics of the cloned audio are the same as those of the original audio, and the content of the cloned audio is the content of the first text.

[0148] In some optional implementations, the processing unit 802 is further configured to determine that the sound features of the original audio conform to a first preset specification, the first preset specification including: the sound features match the tenant.

[0149] In some optional embodiments, the processing unit 802 is specifically configured to: match the sound features of the original audio with a feature library, the feature library including multiple preset sound features, the multiple preset sound features including a first sound feature, the first sound feature matching the sound features of the original audio, and the feature library also including the association between the first sound feature and the identifier of a first tenant. The sound features of the original audio are determined to conform to the first preset definition.

[0150] In some optional implementations, the processing unit 802 is specifically configured to: match the sound features of the original audio with a feature library, the feature library including multiple preset sound features, wherein the sound features of the original audio do not match any of the multiple preset sound features; and determine that the sound features of the original audio conform to a first preset specification.

[0151] In some alternative implementations, the processing unit 802 is also configured to add the sound features of the original audio to a feature library in response to a feature addition instruction for the original audio.

[0152] In some optional implementations, the processing unit 802 is further configured to perform semantic analysis on the first text to determine that the first text conforms to a second preset specification.

[0153] The second presupposition includes at least one of the following: the text does not contain sensitive words, the text is coherent, or the text is semantically clear.

[0154] In some optional implementations, the acquisition unit 801 is further configured to acquire second text from the first tenant, wherein the second text is text that does not conform to a second preset specification. In response to a modification instruction for the second text, the content of the second text is modified to obtain a first text that conforms to the second preset specification.

[0155] In some alternative implementations, the processing unit 802 is further configured to add an audio identifier to the cloned audio if a preset condition is met, the audio identifier being used to indicate the first tenant.

[0156] The preset conditions include at least one of the following: the sound features of the original audio match the second sound features in the feature library, and the second sound features are not associated with the first tenant in the feature library. Alternatively, the content of the first text does not conform to the second preset conditions, which include at least one of the following: the content of the text does not contain sensitive words, the content of the text is coherent, or the semantics of the text are clear.

[0157] Both the acquisition unit 801 and the processing unit 802 can be implemented in software or in hardware. For example, the implementation of the processing unit 802 will be described below. Similarly, the implementation of the acquisition unit 801 can be referenced to the implementation of the processing unit 802.

[0158] As an example of a software functional unit, processing unit 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, processing unit 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0159] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0160] As an example of a hardware functional unit, the processing unit 802 may include at least one computing device, such as a server. Alternatively, the processing unit 802 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0161] The processing unit 802 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing unit 802 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing unit 802 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0162] It should be noted that the acquisition unit 801 and the processing unit 802 respectively implement different steps in the audio cloning method based on cloud computing technology, thereby realizing all the functions of the cloud platform 800. The cloud platform 800 is used to implement the audio cloning method based on cloud computing technology provided in the embodiments of this application, and will not be described in detail here.

[0163] Please refer to Figure 9, which is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate with each other via the bus 903. In practical applications, communication can also be achieved through other means such as wireless transmission; the specific method is not limited here.

[0164] It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0165] Processor 901 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0166] The communication interface 902 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0167] Bus 903 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 903 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 904, processor 901, communication interface 902).

[0168] Memory 904 may include volatile memory, such as random access memory (RAM). Memory 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] The memory 904 stores executable program code, and the processor 901 executes the executable program code to implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby realizing the audio cloning method based on cloud computing technology provided in this application embodiment. That is, the memory 904 stores instructions for executing the audio cloning method based on cloud computing technology provided in this application embodiment.

[0170] This application also provides a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0171] Please refer to Figures 10 and 11, which are schematic diagrams of the structure of the computing device cluster provided in the embodiments of this application.

[0172] As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 904 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the audio cloning method based on cloud computing technology provided in the embodiments of this application.

[0173] In some possible implementations, the memory 904 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the audio cloning method based on cloud computing technology. In other words, a combination of one or more computing devices 904 can jointly execute the audio cloning method based on cloud computing technology.

[0174] It should be noted that the memory 904 in different computing devices 900 within the computing device cluster can store different instructions, which are used to execute certain functions of the cloud platform 800. That is, the instructions stored in the memory 904 of different computing devices 900 can implement the functions of one or more units among the acquisition unit 801 and the processing unit 802.

[0175] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. As shown in Figure 11, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 904 in computing device 900A stores instructions for executing the functions of the acquisition unit 801. Simultaneously, the memory 904 in computing device 900B stores instructions for executing the functions of the processing unit 802.

[0176] The connection method between the computing device clusters shown in Figure 11 can be based on the audio cloning method provided in this application, which separates the processing operation from the operation outside the processing operation. That is, it is considered that the function of the acquisition unit 801 is executed by the computing device 900A, and the function of the processing unit 802 is executed by the computing device 900B.

[0177] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0178] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 10 and 11, and will not be repeated here.

[0179] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to execute the aforementioned audio cloning method based on cloud computing technology.

[0180] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the aforementioned audio cloning method based on cloud computing technology.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An audio cloning method based on cloud computing technology, characterized in that, The method is applied to a cloud platform for managing infrastructure running cloud computing services, the infrastructure including at least one data center, each of the at least one data center including multiple servers, and the method includes: The cloud platform obtains the original audio from the first tenant, encrypts the original audio to obtain encrypted audio, and stores the encrypted audio in the infrastructure; The cloud platform obtains the first text from the first tenant; The cloud platform decrypts the encrypted audio to obtain the original audio. The cloud platform synthesizes the original audio and the first text to obtain a cloned audio. The sound features of the cloned audio are the same as those of the original audio, and the content of the cloned audio is the content of the first text.

2. The method according to claim 1, characterized in that, Before the cloud platform synthesizes the original audio and the first text, the method further includes: The cloud platform determines that the sound features of the original audio conform to a first preset specification, which includes: the sound features match the tenant.

3. The method according to claim 2, characterized in that, The cloud platform determines that the sound features of the original audio conform to the first preset definition, including: The cloud platform matches the sound features of the original audio with a feature library. The feature library includes multiple preset sound features, including a first sound feature. The first sound feature matches the sound features of the original audio. The feature library also includes the association between the first sound feature and the identifier of the first tenant. The cloud platform determines that the sound features of the original audio conform to the first preset definition.

4. The method according to claim 2, characterized in that, The cloud platform determines that the sound features of the original audio conform to the first preset definition, including: The cloud platform matches the sound features of the original audio with a feature library, which includes multiple preset sound features. The sound features of the original audio do not match any of the multiple preset sound features. The cloud platform determines that the sound features of the original audio conform to the first preset definition.

5. The method according to claim 4, characterized in that, The method further includes: The cloud platform responds to the feature addition instruction for the original audio and adds the sound features of the original audio to the feature library.

6. The method according to any one of claims 1 to 5, characterized in that, Before the cloud platform synthesizes the original audio and the first text, the method further includes: The cloud platform performs semantic analysis on the first text and determines that the first text conforms to the second preset definition. The second preset requirement includes at least one of the following: the text content does not contain sensitive words.

7. The method according to claim 6, characterized in that, Before the cloud platform obtains the first text from the first tenant, the method further includes: The cloud platform obtains a second text from the first tenant, where the second text does not conform to the second preset specification. The cloud platform obtains the first text from the first tenant, including: The cloud platform responds to the modification instruction for the second text, modifies the content of the second text, and obtains the first text.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If preset conditions are met, the cloud platform adds an audio identifier to the cloned audio, and the audio identifier is used to indicate the first tenant; The preset conditions include at least one of the following: The sound features of the original audio match the second sound features in the feature library, and the second sound features in the feature library are not associated with the first tenant; or, The first text does not conform to the second preset rule, which includes at least one of the following: the content of the text does not contain sensitive words.

9. A cloud platform, characterized in that, The cloud platform is used to manage the infrastructure running cloud computing services, the infrastructure including at least one data center, each of the at least one data center including multiple servers; the cloud platform includes: The acquisition unit is used to acquire the original audio from the first tenant, encrypt the original audio to obtain encrypted audio, and store the encrypted audio in the infrastructure; The acquisition unit is further configured to acquire the first text from the first tenant; The processing unit is used to decrypt the encrypted audio to obtain the original audio. The processing unit is further configured to synthesize the original audio and the first text to obtain cloned audio, wherein the sound features of the cloned audio are the same as those of the original audio, and the content of the cloned audio is the content of the first text.

10. The cloud platform according to claim 9, characterized in that, The processing unit is further configured to determine that the sound features of the original audio conform to a first preset specification, the first preset specification including: the sound features match the tenant.

11. The cloud platform according to claim 10, characterized in that, The processing unit is specifically used for: The sound features of the original audio are matched in a feature library, which includes multiple preset sound features, including a first sound feature. The first sound feature matches the sound features of the original audio, and the feature library also includes the association between the first sound feature and the identifier of the first tenant. The sound features of the original audio are determined to conform to the first preset definition.

12. The cloud platform according to claim 10, characterized in that, The processing unit is specifically used for: The cloud platform matches the sound features of the original audio with a feature library, which includes multiple preset sound features. The sound features of the original audio do not match any of the multiple preset sound features. The cloud platform determines that the sound features of the original audio conform to the first preset definition.

13. The cloud platform according to claim 12, characterized in that, The processing unit is further configured to respond to a feature addition instruction for the original audio and add the sound features of the original audio to the feature library.

14. The cloud platform according to any one of claims 9 to 13, characterized in that, The processing unit is further configured to perform semantic analysis on the first text and determine that the first text conforms to the second preset specification; The second preset requirement includes at least one of the following: the text content does not contain sensitive words.

15. The cloud platform according to claim 14, characterized in that, The acquisition unit is further configured to acquire second text from the first tenant, wherein the second text is text that does not conform to the second preset specification; The acquisition unit is specifically used to respond to a modification instruction for the second text, modify the content of the second text, and obtain the first text.

16. The cloud platform according to any one of claims 9 to 15, characterized in that, The processing unit is further configured to add an audio identifier to the cloned audio if a preset condition is met, the audio identifier being used to indicate the first tenant; The preset conditions include at least one of the following: The sound features of the original audio match the second sound features in the feature library, and the second sound features in the feature library are not associated with the first tenant; or, The content of the first text does not conform to the second preset rule, which includes at least one of the following: the content of the text does not contain sensitive words.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

Images