Information processing system, information processing method, and storage medium

The information processing system addresses the challenge of suppressing specific sounds in audio generation from video by using a conditional layer and ControlNet architecture, enabling controlled and efficient audio production.

WO2026234885A1PCT designated stage Publication Date: 2026-11-12SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/006364
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-02-20
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Conventional technologies for generating audio from video fail to appropriately suppress the influence of specific sounds, leading to difficulties in generating audio that excludes unwanted noise.

Method used

An information processing system that includes a speech generation model capable of suppressing specific sounds by using a conditional layer to exclude first audio data from the generated second audio data, employing a ControlNet architecture and multimodal diffuse transformers to condition the audio generation process.

Benefits of technology

The system effectively generates audio that suppresses the influence of specific sounds, improving controllability and efficiency in sound production by allowing targeted audio editing and generation of multiple audio sets with desired sounds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026006364_12112026_PF_FP_ABST
    Figure JP2026006364_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An information processing system according to the present disclosure includes a processing circuit. The processing circuit receives video data for which audio data is to be generated, receives first audio data to be suppressed when audio data for the video data is generated, and generates second audio data on the basis of the video data and the first audio data. The second audio data is audio data in which audio included in the first audio data is suppressed.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing system, information processing method, and storage medium

[0001] This disclosure relates to an information processing system, an information processing method, and a storage medium.

[0002] Conventionally, technologies have been provided for generating audio data (hereinafter also referred to as "audio") from video data (hereinafter also referred to as "video"). For example, technologies have been provided for generating (synthesizing) audio based on the V2A (Video-to-Audio) model (for example, Non-Patent Document 1).

[0003] Simian Luo, Chuanhao Yan, Chenxu Hu, Hang Zhao, "Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models", [online] [Retrieved January 16, 2026], Internet <URL: https: / / arxiv.org / pdf / 2306.17203>

[0004] However, there is room for improvement in conventional technology. For example, while conventional technology can generate audio from video, it does not specifically consider sounds that should be excluded from the generated audio, making it difficult to appropriately generate audio that suppresses the influence of specific sounds. Therefore, there is a need to appropriately generate audio that suppresses the influence of specific sounds.

[0005] Therefore, this disclosure proposes an information processing system, an information processing method, and a storage medium that can appropriately generate audio with the influence of specific audio suppressed.

[0006] To solve the above problems, one form of information processing system according to the present disclosure includes a processing circuit which receives video data to be used to generate audio data, receives first audio data to be suppressed when generating audio data based on the video data, generates second audio data based on the video data and the first audio data, and the second audio data is audio data in which the audio contained in the first audio data has been suppressed.

[0007] This figure shows an example of information processing according to the embodiment of this disclosure. This figure shows an overview of the speech generation process. This is a schematic diagram showing an example of a network architecture. This figure shows an example of the configuration of an information processing system according to the embodiment of this disclosure. This is a flowchart of the information processing procedure according to the embodiment of this disclosure. This figure shows an example of experimental results. This figure shows an example of experimental results. This is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0008] Embodiments of this disclosure will be described in detail below with reference to the drawings. Note that these embodiments do not limit the information processing system, information processing method, and storage medium described herein. Furthermore, in each of the following embodiments, the same parts are denoted by the same reference numerals to avoid redundant explanations.

[0009] This disclosure will be described in the following order of items: 1. Embodiments 1-1. Overview of information processing according to the embodiment of this disclosure 1-1-1. Processing example (stepwise processing) 1-1-2. Other examples 1-2. Configuration of the information processing system according to the embodiment 1-3. Procedure for information processing according to the embodiment 1-4. Experimental results 2. Others 3. Hardware configuration

[0010] <1. Embodiments> <1-1. Overview of Information Processing According to the Embodiments of the Disclosure> First, an overview of the information processing according to the embodiments of the Disclosure will be described using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiments of the Disclosure. Specifically, Figure 1 explains the overview of the information processing as an example of generating audio from video data that does not contain audio data.

[0011] The information processing according to the embodiment of this disclosure is realized by an information processing system 1 (see Figure 4) having an information processing device 100. Details of the configuration of the information processing system 1 will be described later, but the information processing device 100 in the information processing system 1 is an example of a computer that performs processing related to speech generation. In Figure 1, the information processing device 100 uses a speech generation model M1, which is an example of a machine learning model, to generate speech for a video MV1, which is the video for which speech data is to be generated.

[0012] The video MV1 in Figure 1 is, for example, a video of a moose walking along the lakeshore. The information processing device 100 uses the voice generation model M1 to generate audio corresponding to the video MV1, which is the video of the moose walking along the lakeshore. The voice generation model M1 is an AI (Artificial Intelligence) model (also simply called a "model") generated using machine learning techniques.

[0013] For example, the speech generation model M1 takes video and audio data to be suppressed as input, and outputs audio data (hereinafter also referred to as "second audio data") which corresponds to the input video and has the audio from the suppressed audio data (hereinafter also referred to as "first audio data") suppressed. The speech generation model M1 also takes specification information indicating the target audio to be generated as input, and outputs second audio data which corresponds to the input video, is the target audio indicated by the specification information, and has the audio from the first audio data suppressed. Note that any audio data can be used for the first audio data. For example, the first audio data is audio data specified by the user.

[0014] In Figure 1, text information is shown as an example of specified information, but the specified information is not limited to text and may be in any format. For example, the specified information may be audio data that shows the image of the sound to be generated. Furthermore, the speech generation model M1 may use any information as input, as long as the video to be generated is included in the input. For example, the speech generation model M1 may accept specified information as arbitrary input. Furthermore, the speech generation model M1 may accept first audio data as arbitrary input.

[0015] <1-1-1. Processing Example (Step-by-Step Processing)> The following describes the processing flow shown in Figure 1. Figure 1 illustrates step-by-step processing as an example, in which the generation process is repeatedly executed to generate audio corresponding to the video in stages. First, in Figure 1, the information processing device 100 executes the first generation process (also called the "first generation process") using the video MV1 and the specified information TX1 (step S11). The specified information TX1 in Figure 1 is the text information (sentence) "Water splashes," which is the specified information that designates the sound of splashing water as the target audio.

[0016] The information processing device 100 inputs the video MV1 and specified information TX1 to the audio generation model M1, and generates audio AD11 by causing the audio generation model M1 to output audio AD11. Thus, in the first generation process, the information processing device 100 generates audio AD11 that includes the sound of splashing water corresponding to the video MV1 of a moose walking along the lakeshore. At the time of the first generation process, there is no first audio data to be suppressed, so the information processing device 100 does not input the first audio data to the audio generation model M1.

[0017] Then, the information processing device 100 performs a second generation process (also called the "second generation process") using the video MV1, the specified information TX2, and the audio AD11 (step S12). The specified information TX2 in Figure 1 is the text information (sentence) "Moose snorts," which is the specified information that designates the sound of a moose snorting as the target sound. In the second generation process, the audio AD11 is input to the audio generation model M1 as the first audio data to be suppressed when generating audio with the sound of a moose snorting as the target sound. In other words, in the second generation process, the audio AD11 generated in the first generation process is used as the first audio data.

[0018] The information processing device 100 inputs the video MV1, the specified information TX2, and the first audio data, audio AD11, to the audio generation model M1, causing the audio generation model M1 to output audio AD12, thereby generating audio AD12. As a result, in the second generation process, the information processing device 100 generates audio AD12 that includes the sound of a moose snuffling, corresponding to the video MV1 of a moose walking along the lakeshore, while suppressing the inclusion of audio AD11 (the sound of splashing water). In this way, the information processing device 100 can suppress the inclusion of sounds other than the sound of a moose snuffling that is to be generated for the video MV1 in the second generation process, and generate audio AD12 that appropriately includes the sound of a moose snuffling corresponding to the video MV1.

[0019] Here, we will explain the overview of the speech generation process using the speech generation model M1 with reference to Figure 2. Figure 2 is a diagram illustrating the overview of the speech generation process. For example, the speech generation process PS1 shown in Figure 2 illustrates the overview of the speech generation process using the speech generation model M1, as an example of the process corresponding to the second generation process.

[0020] As shown in Figure 2, the speech generation model M1 includes a first submodel denoted as "VA-to-A Model" and a second submodel denoted as "VT-to-A Model". For example, the speech generation model M1 has a network architecture as shown in Figure 3. Figure 3 is a schematic diagram showing an example of a network architecture. For example, it is a diagram showing an overview of the network architecture NT1 of the speech generation model M1 which functions as a speech conditional flow estimator.

[0021] For example, in the NT1 network architecture, the number of layers (M) in the control branch is equal to the number of layers (N) in the main branch. 1 +N 2) is less. Also, in network architecture NT1, some layers may share the same control signals. The left part of network architecture NT1 shown in Figure 3 may be a model based on existing methods (e.g., MMAudio). Also, for example, the right part of network architecture NT1 shown in Figure 3 is a model added for conditioning (constraints).

[0022] As shown in Figure 3, the speech generation model M1 employs ControlNet as a multimodal diffuse transformer (MM-DiT) to incorporate speech conditions into the pre-trained MMAudio. The MM-DiT is disclosed, for example, in the following document: Patrick Esser, et al. "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." In Proceedings of the International Conference on Machine Learning, 2024.

[0023] For example, if an architecture extended from MM-DiT is adopted for the speech generation model M1, the ControlNet architecture will also be adapted accordingly. For instance, multiple single-modal transformer blocks are stacked to extract features from conditional speech based on Stable Diffusion 3.5. Stable Diffusion 3.5 is disclosed in the following document, for example: • Stability-AI. sd3.5, 2024<https: / / github.com / Stability-AI / sd3.5>

[0024] As shown in Figure 3, the features extracted in each block are added to the intermediate features of the corresponding block in MMAudio. For example, the speech generation model M1 is an extension of MMAudio to support audio conditioning using ControlNet. For example, the data used to train the speech generation model M1 (training dataset) may be the same as that used for MMAudio. A detailed explanation of the training of the speech generation model M1 is omitted, but for example, an 8-second consecutive audio sample is taken, and the audio sample is split into two 4-second audio segments, one of which is used as a condition and the other as a target. Also, for example, during training, the pre-trained parameters of MMAudio are fixed, and only the parameters of the additional module are updated.

[0025] As described above, the speech generation model M1 has a conditional layer for excluding the first speech data from the second speech data. The speech generation model M1 has a ControlNet architecture. The speech generation model M1 is a model based on MMAudio. Note that the network architecture NT1 shown in Figure 3 is merely an example, and any network architecture may be adopted for the speech generation model M1 as long as it can perform the desired processing.

[0026] Let's return to Figure 2 and continue the explanation. As shown in Figure 2, the first partial model consists of the video in Figure 2 and the Conditional audio x in Figure 2. (1) Intermediate features corresponding to the first audio data are generated from the first audio data. In this way, x (1) This corresponds to the first audio data.

[0027] In the second generation process, the video is video MV1, and the first audio data is audio AD11. Also, as shown in Figure 2, the second partial model generates intermediate features corresponding to the pre-suppression specified information of the first audio data from the video (Video) and the specified information (Text) in Figure 2. In the second generation process, the specified information is specified information TX2, which is "Moose snorts".

[0028] As shown in the audio generation process PS1, the intermediate feature corresponding to the first audio data is set as a suppression target (corresponding to Negation in FIG. 2), so that it is not included in the second audio data to be generated, as shown in Generated audio x in FIG. 2 (2) to generate the second audio data. In this way, x (2) corresponds to the second audio data. In the case of the second generation process, the second audio data is audio AD12 in which the inclusion of audio AD11 (the sound of splashing water) is suppressed, and which includes the sound of a moose snorting.

[0029] The conditioning by the first audio data x described above (1) corresponds to Concept Negation. Note that Concept Negation is disclosed, for example, in the following document (hereinafter also referred to as "Du's document"). ・Yilun Du, et al. "Compositional visual generation with energy based models." In Proceedings of the Advances in Neural Information Processing Systems, 2020.

[0030] For x corresponding to the second audio data to be generated (2) , an audio condition that the concept related to x corresponding to the first audio data (1) must not be included is defined as in Equation (1), and is clearly distinguished from the standard audio condition ε(·).

[0031]

[0032] Based on the relationship between x described above (1) and x (2) , the target distribution of x can be approximated as in Equation (2) using Concept Negation (2) .

[0033]

[0034] As described above, negative conditioning is approximated by being explicitly modeled. Here, it is assumed that Concept Negation has the property shown in Equation (3) based on Du's document.

[0035]

[0036] c in Equation (3) p and c n represent conditional concepts for generating x. When the target distribution as shown in Equation (2) is decomposed using Equation (3) and Bayes' theorem, Equation (4) is derived.

[0037]

[0038] The corresponding guide flow can be expressed as shown in Equation (5).

[0039]

[0040] Here, to further simplify the formulation of the guide flow, let w1′=α, w3′=β, and w2′=α+β. This is based on the results reported in the following document. ・Saksham Singh Kushwaha and Yapeng Tian. "Vintage: Joint video and text conditioning for holistic audio generation." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2025.

[0041] Thereby, the guide flow can be expressed as shown in Equation (6). Except for the last term in Equation (6) (the u in β θ,φ () term), estimation is possible using a typical VT2A model such as MMAudio, and only the last term needs to be modeled for stepwise processing.

[0042]

[0043] For example, the last term in equation (6) corresponds to equation (2) and becomes the training target. Since ε(・) is a standard speech adjustment technique, this term can be modeled by introducing an additional speech reference input to a typical V2A model. A model that generates speech semantically similar to the reference model is needed, which enables training on a standard AV dataset.

[0044] Returning to Figure 1, the explanation continues. The information processing device 100 performs a third generation process (also called the "third generation process") using the video MV1, the specified information TX3, the audio AD11, and the audio AD12 (step S13). The specified information TX3 in Figure 1 is the text information (sentence) "Wind rustles," which specifies the sound of wind rustling as the target sound. In the third generation process, audio AD11 and audio AD12 are suppressed when generating audio with the sound of wind rustling as the target sound.

[0045] In Figure 1, the information processing device 100 generates synthesized speech AD2 by combining speech AD11 and speech AD12 through a synthesis process. The synthesized speech AD2 of speech AD11 and speech AD12 is input to the speech generation model M1 as first speech data to be suppressed when generating speech targeting the sound of rustling wind. That is, in the third generation process, the synthesized speech AD2 of speech AD11 generated in the first generation process and speech AD12 generated in the second generation process is used as first speech data.

[0046] The information processing device 100 inputs the video MV1, specified information TX3, and the first audio data, which is synthesized speech AD2, to the speech generation model M1, and generates speech AD13 by causing the speech generation model M1 to output speech AD13. For example, in the third generation process, Conditional audio x in Figure 2 (1)Synthesized speech AD2 is used as the first audio data. As a result, in the third generation process, the information processing device 100 generates audio AD13 that includes the rustling sound of the wind corresponding to the video MV1 of the moose walking along the lakeshore, while suppressing the inclusion of synthesized speech AD2 (the sound of splashing water and the sound of the moose snorting). In this way, the information processing device 100 can suppress the inclusion of sounds other than the rustling sound of the wind that it wants to generate for the video MV1 in the third generation process, and generate audio AD13 that appropriately includes the rustling sound of the wind corresponding to the video MV1.

[0047] Then, the information processing device 100 generates audio corresponding to the video using the audio generated in the generation process (step S14). In Figure 1, the information processing device 100 generates audio corresponding to the video MV1 using audio AD11 generated in the first generation process, audio AD12 generated in the second generation process, and audio AD13 generated in the third generation process. The information processing device 100 generates synthesized audio AD1 by combining audio AD11, audio AD12, and audio AD13 in a synthesis process. As a result, the information processing device 100 generates synthesized audio AD1 that includes the sound of splashing water, the sound of a moose snorting, and the sound of wind rustling, and corresponds to the video MV1.

[0048] Through the process described above, the information processing system 1 can generate second audio data by using the audio data to be suppressed as first audio data, thereby suppressing the inclusion of audio from the first audio data in the generated second audio data. Therefore, the information processing system 1 can appropriately generate audio with the influence of specific audio suppressed. Furthermore, for example, existing V2A systems did not have the technology to generate individual audio data. On the other hand, through the stepwise processing described above, the information processing system 1 can generate audio data individually for each target to be generated. In addition, the information processing system 1 can perform desired editing using the individually generated audio. In this way, the information processing system 1 can give V2A editability. Therefore, the information processing system 1 can improve the controllability and efficiency in the sound production process.

[0049] As described above, the information processing system 1 can appropriately generate audio data corresponding to video data that does not contain audio data by targeting video data that does not contain audio data and generating audio data corresponding to that video data. Furthermore, for example, the information processing system 1 can appropriately generate audio that suppresses the influence of a specific sound desired by the user by using audio data specified by the user as the first audio data and generating the second audio data. The information processing system 1 can appropriately generate audio that suppresses the influence of a specific sound by inputting the video data and the first audio data into a machine learning model and generating the second audio data using the machine learning model. As described above, in stepwise processing, the information processing system 1 can generate multiple second audio data sets, each containing a different sound, by using the second audio data already generated at the processing stage as the first audio data and generating new second audio data.

[0050] <1-1-2. Other Examples> The process described above is merely one example, and the information processing system 1 may generate audio data using various audio data as the first audio data. For example, in the stepwise process described above, a generation process was performed repeatedly, and audio data generated in a generation process prior to that generation process was used as the first audio data. However, the first audio data does not have to be generated in the generation process.

[0051] For example, the information processing system 1 may use audio data specified by a user instructing the generation of audio data as first audio data to generate second audio data corresponding to the video. For example, the information processing system 1 may use audio data containing noise as first audio data to generate second audio data corresponding to the video, with the noise removed.

[0052] <1-2. Configuration of the Information Processing System According to the Embodiment> Next, the configuration of the information processing system 1, which is an example of an information processing system that performs the information processing according to the embodiment, will be described. Figure 4 is a diagram showing an example of the configuration of the information processing system according to the embodiment of this disclosure. For example, the information processing system 1 shown in Figure 4 is an example of an information processing system that performs the information processing described above.

[0053] As shown in Figure 4, the information processing system 1 includes at least an information processing device 100. The information processing device 100 is connected to other computers via a predetermined communication network N, either by wired or wireless means. Note that the configuration of the information processing system 1 shown in Figure 4 is just one example; any device configuration can be adopted as long as the information processing system 1 includes at least an information processing device 100.

[0054] For example, the information processing system 1 may include multiple information processing devices 100. The information processing system 1 may also include terminal devices used by users who request processing from the information processing devices 100. For example, if the information processing device 100 is a server device, the information processing system 1 includes terminal devices used by users who receive the voice generation service provided by the information processing device 100. In this case, the information processing device 100 receives information such as video data for which the user requests voice generation from the terminal device used by the user, performs voice generation processing based on the received information, and transmits the generated voice and other information to the terminal device used by the user.

[0055] As shown in Figure 4, the information processing device 100 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15. In the example in Figure 4, the information processing device 100 includes an input unit 12 (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and an output unit 13 (for example, a liquid crystal display) that outputs (displays, etc.) various information.

[0056] The communication unit 11 is implemented, for example, by a NIC (Network Interface Card) or a communication circuit. The communication unit 11 is connected to a communication network N (such as the Internet) by wire or wireless connection and transmits and receives information with other devices via the communication network N.

[0057] The input unit 12 receives various operations from the user. The input unit 12 accepts input from the user. The input unit 12 accepts input from the user of information used to generate a mesh model. The input unit 12 may also accept various operations from the user via a keyboard, mouse, or touch panel provided on the information processing device 100.

[0058] The output unit 13 outputs various types of information. For example, the output unit 13 has a function to output information as sound. For example, the output unit 13 has a sound output device (sound output unit) such as a speaker that outputs sound. The output unit 13 outputs the information generated by the generation unit 152 as sound. The output unit 13 outputs the second sound data generated by the generation unit 152 as sound.

[0059] Furthermore, the output unit 13 has a display device (display unit) such as a display and displays various information. The output unit 13 displays information related to processing by the processing unit 153. The output unit 13 displays video with sound, in which the sound generated by the processing unit 153 is added to the video. In addition, the output unit 13 may have functions to output information in various ways, not limited to the sound output function and display function.

[0060] The memory unit 14 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by a storage medium (storage device) such as a hard disk or optical disc. The memory unit 14 has a model information storage unit 141.

[0061] The model information storage unit 141 according to this embodiment stores information about a model. For example, the model information storage unit 141 stores information (model data) that shows the structure of a model (network). The model information storage unit 141 includes items such as "model ID," "purpose," and "model data."

[0062] The "Model ID" indicates identification information for identifying the model. The "Purpose" indicates the purpose of the corresponding model. The "Model Data" indicates the data of the model. For example, the "Model Data" includes various pieces of information that make up the model, such as information about the network included in the model and functions.

[0063] The model ID "M1" that identifies the speech generation model M1 is associated with and stored information indicating that its purpose is "speech generation." This indicates that the speech generation model M1 is a model used for the task of speech generation. In addition, the model ID "M1" that identifies the speech generation model M1 is associated with and stored information indicating that the model data of the speech generation model M1 is model data MDT1.

[0064] For example, model data MDT1 includes information indicating that the speech generation model M1 has a conditional layer for excluding the first speech data from the second speech data. Model data MDT1 includes information indicating that the speech generation model M1 has a ControlNet architecture. Model data MDT1 includes information indicating that the speech generation model M1 is a model based on MMAudio.

[0065] The model information storage unit 141 is not limited to the above and may store various types of information depending on the purpose. For example, the model information storage unit 141 stores parameter information of a model learned (generated) through a learning process.

[0066] Furthermore, the memory unit 14 stores various other types of information. For example, the memory unit 14 stores various types of information used in processing. For example, the memory unit 14 stores the video to be processed. For example, the memory unit 14 stores the first audio data used in processing. For example, the memory unit 14 stores the specified information used in processing. For example, the memory unit 14 stores the information generated by the generation unit 152. For example, the memory unit 14 stores the generated second audio data. The memory unit 14 stores the audio generated by the generation unit 152.

[0067] The control unit 15 has a processing circuit. The control unit 15 is realized by executing a program (for example, the information processing program according to this disclosure) stored inside the information processing device 100 using RAM or the like as a working area, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit). The control unit 15 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0068] As shown in Figure 4, the control unit 15 includes an acquisition unit 151, a generation unit 152, a processing unit 153, and a providing unit 154, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in Figure 4, and other configurations are also acceptable as long as they perform the information processing described later.

[0069] The acquisition unit 151 acquires various types of information. The acquisition unit 151 acquires various types of information from an external information processing device. The acquisition unit 151 acquires various types of information from the storage unit 14. The acquisition unit 151 acquires information received by the input unit 12. The acquisition unit 151 acquires video footage received by the input unit 12.

[0070] The acquisition unit 151 acquires various information from the storage unit 14. The acquisition unit 151 acquires the speech generation model M1 from the model information storage unit 141. The acquisition unit 151 acquires video from the storage unit 14. The acquisition unit 151 receives information via the communication unit 11. The acquisition unit 151 acquires video to be processed.

[0071] The acquisition unit 151 acquires video data to be used for generating audio data. The acquisition unit 151 acquires video data that does not contain audio data. The acquisition unit 151 acquires video data to be used for generating audio data from the storage unit 14. The acquisition unit 151 receives video data to be used for generating audio data from a terminal device used by a user requesting audio generation.

[0072] The acquisition unit 151 acquires first audio data to be suppressed when generating audio data for video data. The acquisition unit 151 acquires first audio data from the storage unit 14. The acquisition unit 151 acquires first audio data specified by the user. The acquisition unit 151 receives first audio data from a terminal device used by a user requesting audio generation.

[0073] The acquisition unit 151 acquires designation information indicating the target audio to be generated when generating audio data from video data. The acquisition unit 151 acquires designation information from the storage unit 14. The acquisition unit 151 acquires text indicating the target audio as designation information. The acquisition unit 151 receives text indicating the target audio from a terminal device used by a user requesting audio generation. The acquisition unit 151 acquires third audio data indicating the target audio as designation information. The acquisition unit 151 receives third audio data indicating the target audio from a terminal device used by a user requesting audio generation.

[0074] The generation unit 152 performs processing related to the generation of information. The generation unit 152 performs generation processing using the information stored in the storage unit 14. The generation unit 152 performs generation processing using the information acquired by the acquisition unit 151.

[0075] The generation unit 152 generates second audio data, which is audio data in which the audio contained in the first audio data has been suppressed, based on the video data and the first audio data. The generation unit 152 generates the second audio data by inputting the video data and the first audio data into a machine learning model.

[0076] The generation unit 152 generates second audio data by inputting video data and first audio data into an audio generation model M1 having a conditional layer for excluding first audio data from second audio data. The generation unit 152 generates second audio data by inputting video data and first audio data into an audio generation model M1 having a ControlNet architecture. The generation unit 152 generates second audio data by inputting video data and first audio data into an audio generation model M1 which is a model based on MMAudio.

[0077] The generation unit 152 generates second audio data containing the target audio based on specified information. The generation unit 152 generates second audio data containing the target audio based on third audio data. The generation unit 152 generates multiple second audio data by repeatedly executing a generation process that generates second audio data based on video data and first audio data. In the repetition of the generation process, the generation unit 152 generates new second audio data by using the second audio data already generated at the time of processing as first audio data, thereby generating multiple second audio data, each containing a different audio.

[0078] The generation unit 152 may generate various types of information. The generation unit 152 may generate various types of information to be displayed. For example, the generation unit 152 may generate content to be provided to the user. In this case, the generation unit 152 generates screen-related information (content) using various conventional technologies related to content generation as appropriate. The generation unit 152 generates content using various conventional technologies related to GUI (Graphical User Interface) as appropriate. For example, the generation unit 152 may generate content using CSS (Cascading Style Sheets), JavaScript (registered trademark), HTML (HyperText Markup Language), or any language capable of describing the information processing such as the information display and operation reception described above.

[0079] The processing unit 153 performs various information processing operations. The processing unit 153 performs processing using the information generated by the generation process of the generation unit 152. The processing unit 153 performs information processing using the information stored in the storage unit 14.

[0080] The processing unit 153 performs information processing using the second audio data generated by the generation unit 152. The processing unit 153 generates an audio-enhanced video in which the audio generated by the generation unit 152 is added to the video. For example, the processing unit 153 generates an audio-enhanced video by adding the second audio data generated by the generation unit 152 to the video data.

[0081] The provisioning unit 154 provides various types of information. The provisioning unit 154 provides various types of information to an external information processing device. The provisioning unit 154 transmits various types of information. The provisioning unit 154 transmits various types of information to an external information processing device. The provisioning unit 154 transmits information stored in the storage unit 14. The provisioning unit 154 transmits information stored in the model information storage unit 141.

[0082] The providing unit 154 transmits the information generated by the generating unit 152. The providing unit 154 transmits the second audio data generated by the generating unit 152 to an external device. For example, the providing unit 154 transmits the second audio data generated by the generating unit 152 to a terminal device used by a user. The providing unit 154 transmits the video generated by the processing unit 153 to an external device. The providing unit 154 transmits the video with sound, in which the audio generated by the processing unit 153 is attached to the video, to an external device. For example, the providing unit 154 transmits the video with sound, in which the audio generated by the processing unit 153 is attached to the video, to a terminal device used by a user.

[0083] <1-3. Information Processing Procedure According to the Embodiment> The information processing procedure according to the embodiment will be described below. First, the information processing procedure according to the embodiment will be described using Figure 5. Figure 5 is a flowchart of the information processing procedure according to the embodiment of this disclosure.

[0084] As shown in Figure 5, the information processing system 1 receives video data to which audio data will be generated (step S101). For example, the information processing device 100 acquires video data to which audio data will be generated.

[0085] Then, the information processing system 1 receives the first audio data to be suppressed when generating audio data based on video data (step S102). For example, the information processing device 100 acquires the first audio data to be suppressed when generating audio data based on video data.

[0086] Then, the information processing system 1 generates second audio data based on the video data and the first audio data (step S103). For example, the information processing device 100 generates second audio data based on the video data and the first audio data.

[0087] <1-4. Experimental Results> From here, we will explain the experimental results using the generation processing method (also called "this method") performed by the information processing system 1 described above, using Figures 6 and 7. Figures 6 and 7 are diagrams showing an example of experimental results. First, before explaining Figures 6 and 7, we will describe the assumptions regarding the experiment. In the experimental results shown below, in order to evaluate stepwise speech generation, the quality of both the synthesized speech and individual speech tracks was evaluated.

[0088] For example, following the prior research in the following document, we evaluated synthesized speech from the perspectives of speech quality, semantic consistency, and temporal consistency: • Ho Kei Cheng, et al. "Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2025.

[0089] Furthermore, the Fréchet distance (hereinafter also referred to as "FD"), the Kullback-Libra distance (hereinafter also referred to as "KL"), and the Inception Score (hereinafter also referred to as "IS") from the following literature were used to evaluate the quality of the generated speech: • Tim Salimans, et al. "Improved techniques for training gans." In Proceedings of the Advances in Neural Information Processing Systems, 2016.

[0090] For the FD calculation, we used PANNs (FDPANNs) from Kong's paper and VGGish (FDVGG) from Gemmeke's paper. For the KL calculation, we used PANNs (KLPANNs) and PaSST (KLPaSST), and for the IS calculation, we used PANNs. • Qiuqiang Kong, et al. "Panns: Large-scale pretrained audio neural networks for audio pattern recognition." IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2020. • Jort F Gemmeke, et al. "Audio set: An ontology and human-labeled dataset for audio events." In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2017.

[0091] Furthermore, the semantic alignment between the input video and synthesized speech was evaluated using the cosine similarity (IB score) between embeddings extracted using ImageBind (hereinafter also referred to as "IB") from the following document: • Rohit Girdhar, et al. "Imagebind: One embedding space to bind them all." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023.

[0092] Furthermore, the temporal consistency between the input video and the generated audio was evaluated using Synchformer (DeSync) from the following literature. Specifically, two 4.8-second segments located at the beginning and end of the video were extracted, and their scores were averaged. • Vladimir Iashin, et al. "Synchformer: Efficient synchronization from sparse cues." In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.

[0093] Furthermore, in evaluating each audio track, we assessed its quality from four perspectives: the degree of separation between audio tracks generated for the same video, audio quality, the degree of synchronization between audio and text, and the degree of synchronization of each audio track with the video. Since different audio components should be represented by separate audio tracks, each generated audio track needs to have differences from the other tracks. To evaluate audio separation, we calculated the similarity between CLAP audio embeds from the following literature for each audio track pair (10 pairs per video): • Yusong Wu, et al. "Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation." In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2023.

[0094] For audio-text synchronization, the similarity between CLAP text embeddings from input prompts (used for generating each audio track) and their corresponding CLAP audio embeddings was calculated. For audio quality and audio-video synchronization, IS scores and IB scores were used, respectively, as with the composite audio evaluation protocol.

[0095] First, let's explain the experimental results shown in Figure 6. For example, Figure 6 shows a quantitative evaluation of a composite speech synthesized from multiple generated audio tracks, and for reference, it shows the results of one-step generation using fused captions. For example, Figure 6 shows a quantitative evaluation of the synthesized speech.

[0096] Our method (Ours in Figure 6) achieved the best results among all methods in all metrics except IS. Furthermore, in the experiment shown in Figure 6, we also evaluated the one-step generation of the baseline model using captions generated by fusing the five captions of each video. While this generation process does not provide each audio track and differs from our goal of stepwise generation, these values ​​represent the highest possible scores for each baseline model.

[0097] Next, we will explain the experimental results shown in Figure 7. For example, Figure 7 shows the quantitative evaluation of individual audio tracks, demonstrating that this method effectively improves the audio separation between multiple tracks while maintaining other scores. For example, Figure 7 shows the objective evaluation of each audio track.

[0098] Standard MMAudio-S-16k using CFG struggles to produce sufficiently separated sounds on each audio track, which is reflected in a low audio separation score. However, high audio quality and AV alignment are achieved. Using negative prompting improves audio separation, but all other scores drop significantly.

[0099] The use of NAG also succeeded in improving audio separation while maintaining high sound quality and AV alignment. The AT alignment score is slightly improved from the standard MMAudio. It is presumed that less contamination of other audio concepts leads to a better AT alignment score.

[0100] As described above, the generation process method (this method) executed by the information processing system 1 can negatively utilize a trained model to ensure that the generated audio does not resemble existing audio, thereby guiding the generation of complementary content. By iterating through this guided generation process, this method generates a composite audio that covers all relevant acoustic events in the video. Thus, it has been demonstrated that this method progressively fills in missing audio, ensuring the separability of the generated audio at each step while improving the final audio quality.

[0101] <2. Others> The above-described process is merely an example, and the information processing device 100 may perform various processes. For example, the information processing device 100 may perform a learning process to learn various models such as the speech generation model M1. In this case, the control unit 15 of the information processing device 100 may include a learning unit that performs the learning process. Also, for example, the storage unit 14 of the information processing device 100 may store data used for the learning process (learning data).

[0102] The learning unit of the information processing device 100 (specifically the control unit 15) executes a learning process to learn various models. The learning unit learns various information based on information from an external information processing device and information stored in the storage unit 14. The learning unit stores the model generated by the learning process in the model information storage unit 141. The learning unit also stores the model with updated parameters from the learning process in the model information storage unit 141.

[0103] The learning unit learns various types of information based on the information acquired by the acquisition unit 151. The learning unit learns (generates) a model. The learning unit learns various types of information, such as the model. The learning unit generates a model through learning. The learning unit learns the model using various machine learning techniques. For example, the learning unit learns the parameters of the model (network). The learning unit learns the model using various machine learning techniques.

[0104] The learning unit generates a speech generation model M1. The learning unit learns the network parameters. For example, the learning unit learns the network parameters of the speech generation model M1.

[0105] The learning unit performs learning processing based on the learning data (training data) stored in the memory unit 14. The learning unit generates a speech generation model M1 by performing learning processing using the learning data stored in the memory unit 14. For example, the learning unit generates a model used for speech generation processing. The learning unit generates the speech generation model M1 by learning the network parameters of the speech generation model M1. Note that the learning method (learning processing) is not limited to the method described above, and any known technology can be applied.

[0106] Furthermore, the processing described in the above-described embodiments may be carried out in various other forms (modifications) besides those described above. For example, the system configuration may be various forms, not limited to the examples described above.

[0107] For example, an information processing system may include a device that performs learning processing (learning device) and a device that performs inference processing using the model learned by the learning device (e.g., an information processing device 100). In this case, the information processing system may include both the learning device and the information processing device 100. The above is just one example, and the information processing system may be implemented in various configurations. For example, the information processing device 100 may be a device that performs both learning processing and inference processing. That is, in an information processing system, the learning device and the inference device may be integrated.

[0108] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0109] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0110] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0111] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur.

[0112] <3. Hardware Configuration> The information processing device 100 and the like according to the embodiments described above are realized by a computer 1000 having a configuration such as that shown in Figure 8. The information processing device 100 will be used as an example for explanation. Figure 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 has a processing circuitry 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0113] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0114] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0115] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores programs for each process of the information processing device 100 according to the embodiment, which is an example of program data 1450.

[0116] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. The communication interface 1500 corresponds to the communication unit 11 of the information processing device 100. For example, the processing circuit 1100 receives data from other devices or transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0117] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0118] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (Organic Electro Luminescence Display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.

[0119] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0120] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the processing circuit 1100 of the computer 1000 functions as a control unit 15 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the information processing program according to this disclosure and various data stored by the storage unit 14. The processing circuit 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside the computer 1000, but may be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0121] Furthermore, this technology can also take the following configurations: (1) An information processing system including a processing circuit, the processing circuit receiving video data to be used to generate audio data, receiving first audio data to be suppressed when generating audio data based on the video data, generating second audio data based on the video data and the first audio data, wherein the second audio data is audio data in which the audio included in the first audio data has been suppressed. (2) The information processing system according to (1), wherein the video data does not include audio data. (3) The information processing system according to (1) or (2), wherein the first audio data is specified by the user. (4) The information processing system according to any one of (1) to (3), wherein the video data and the first audio data are input to a machine learning model to generate the second audio data. (5) The information processing system according to (4), wherein the machine learning model has a conditional layer for excluding the first audio data from the second audio data. (6) The information processing system according to (5), wherein the machine learning model has a ControlNet architecture. (7) The information processing system according to any one of (4) to (6), wherein the machine learning model is a model based on MMAudio. (8) The information processing system according to any one of (1) to (7), which receives designation information indicating a target sound to be generated when generating audio data from video data, and generates the second audio data including the target sound based on the designation information. (9) The information processing system according to (8), which receives text indicating the target sound as the designation information, and generates the second audio data including the target sound based on the text. (10) The information processing system according to (8), which receives third audio data indicating the target sound as the designation information, and generates the second audio data including the target sound based on the third audio data.(11) An information processing system according to any one of (1) to (10), wherein a plurality of second audio data are generated by repeatedly executing a generation process that generates the second audio data based on the video data and the first audio data. (12) An information processing system according to (11), wherein, in the repetition of the generation process, a plurality of second audio data are generated, each containing a different sound, by generating new second audio data using the second audio data already generated at the time of processing as the first audio data. (13) An information processing method, wherein video data to be used to generate audio data is received, first audio data to be suppressed when generating audio data targeting the video data is received, second audio data is generated based on the video data and the first audio data, and the second audio data is audio data in which the sound contained in the first audio data is suppressed. (14) A computer-readable storage medium having stored a program for causing a computer to perform the following processes: receiving video data to be used to generate audio data; receiving first audio data to be suppressed when generating audio data based on the video data; and generating second audio data based on the video data and the first audio data, wherein the second audio data is audio data in which the audio contained in the first audio data has been suppressed.

[0122] 1 Information Processing System 100 Information Processing Device 11 Communication Unit 12 Input Unit 13 Output Unit 14 Storage Unit 141 Model Information Storage Unit 15 Control Unit 151 Acquisition Unit 152 Generation Unit 153 Processing Unit 154 Provision Unit

Claims

1. An information processing system comprising a processing circuit, the processing circuit receiving video data to be used to generate audio data, receiving first audio data to be suppressed when generating audio data based on the video data, generating second audio data based on the video data and the first audio data, the second audio data being audio data in which the audio contained in the first audio data has been suppressed.

2. The information processing system according to claim 1, wherein the video data does not include audio data.

3. The information processing system according to claim 1, wherein the first audio data is specified by the user.

4. The information processing system according to claim 1, wherein the second audio data is generated by inputting the video data and the first audio data into a machine learning model.

5. The information processing system according to claim 4, wherein the machine learning model has a conditional layer for excluding the first audio data from the second audio data.

6. The information processing system according to claim 5, wherein the machine learning model has a ControlNet architecture.

7. The information processing system according to claim 4, wherein the machine learning model is a model based on MMAudio.

8. The information processing system according to claim 1, which receives designation information indicating the target audio to be generated when generating audio data based on the video data, and generates the second audio data including the target audio based on the designation information.

9. The information processing system according to claim 8, which receives text indicating the target audio as the designated information, and generates second audio data including the target audio based on the text.

10. The information processing system according to claim 8, which receives a third audio data indicating the target audio as the designated information, and generates the second audio data including the target audio based on the third audio data.

11. The information processing system according to claim 1, wherein a plurality of second audio data are generated by repeatedly executing a generation process that generates the second audio data based on the video data and the first audio data.

12. The information processing system according to claim 11, wherein, in the repetition of the generation process, a plurality of second audio data, each containing a different voice, are generated by using the second audio data already generated at the time of processing as the first audio data to generate new second audio data.

13. An information processing method comprising: receiving video data to be used to generate audio data; receiving first audio data to be suppressed when generating audio data based on the video data; generating second audio data based on the video data and the first audio data, wherein the second audio data is audio data in which the audio contained in the first audio data has been suppressed.

14. A computer-readable storage medium having a program stored in it that causes a computer to perform the following processes: receiving video data to be used to generate audio data; receiving first audio data to be suppressed when generating audio data based on the video data; and generating second audio data based on the video data and the first audio data, wherein the second audio data is audio data in which the audio contained in the first audio data has been suppressed.