Method and electronic device for generating sound effect for image

Abstract

Disclosed is a method by which an electronic device generates a sound effect for an image, comprising the steps of: acquiring an input image; applying the input image to an image encoding model so as to acquire an image feature vector; selecting, from a first audio feature dictionary in which audio feature vectors of a predetermined category are stored, an audio feature vector having a similarity, with the image feature vector, that is greater than or equal to a threshold value; and applying the selected audio feature vector to a generative AI model so as to generate a sound effect corresponding to the input image.

Description

Method for generating sound effects for an image and electronic device

[0001] The present disclosure relates to a method and electronic device for generating sound effects corresponding to an image.

[0002] As smartphone camera capabilities improve, users are taking many photos using their smartphones. Additionally, users share various photos with other users while using Social Networking Services (SNS) via their smartphones.

[0003] Therefore, recent smartphones are developing various services using photos stored in photo folders or enhancing photo management functions. For example, smartphones provide a function that creates a single video from several photos selected by the user or several photos automatically selected through an internal algorithm and displays it to the user.

[0004] A method for an electronic device to generate a sound effect for an image according to one embodiment of the present disclosure may include: acquiring an input image; applying the input image to an image encoding model to acquire an image feature vector; selecting an audio feature vector from a first audio feature dictionary in which audio feature vectors of a predetermined category are stored, wherein the similarity to the image feature vector is greater than or equal to a threshold value; and applying the selected audio feature vector to a generative AI model to generate a sound effect corresponding to the input image.

[0005] An electronic device according to one embodiment of the present disclosure may include a memory for storing one or more instructions; and at least one processor operably coupled to the memory and comprising a processing circuit. The electronic device may acquire an input image by having at least one processor execute one or more instructions alone or in cooperation. The electronic device may acquire an image feature vector by applying the input image to an image encoding model by having at least one processor execute one or more instructions alone or in cooperation. The electronic device may select an audio feature vector whose similarity to the image feature vector is greater than or equal to a threshold value from a first audio feature dictionary in which audio feature vectors of a predetermined category are stored by having at least one processor execute one or more instructions alone or in cooperation. The electronic device may generate a sound effect corresponding to the input image by applying the selected audio feature vector to a generative AI model by having at least one processor execute one or more instructions alone or in cooperation.

[0006] FIG. 1 is a drawing for explaining an electronic device that provides video according to one embodiment of the present disclosure.

[0007] FIG. 2 is a flowchart for explaining a method for constructing a first audio feature dictionary according to one embodiment of the present disclosure.

[0008] FIG. 3 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to construct a first audio feature dictionary.

[0009] FIG. 4 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to generate a sound effect for an image.

[0010] FIG. 5 is a flowchart illustrating a method for extracting an audio feature vector most similar to an image feature vector according to one embodiment of the present disclosure.

[0011] FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that generates sound effects using a generative AI model.

[0012] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that generates sound effects through batch processing.

[0013] FIG. 8 is a flowchart illustrating a method for generating a plurality of sound effects for an input image according to one embodiment of the present disclosure.

[0014] FIG. 9 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that generates a plurality of sound effects using global feature vectors and local feature vectors.

[0015] FIG. 10 is a flowchart illustrating a method for generating a plurality of sound effects using a feature fusion model according to one embodiment of the present disclosure.

[0016] FIG. 11 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure applying fused audio feature vectors to a generative AI model using a feature fusion model.

[0017] FIG. 12 is a flowchart illustrating a method for generating a sound effect when an audio feature vector most similar to an image feature vector according to one embodiment of the present disclosure is stored in a first audio feature dictionary.

[0018] FIG. 13 is a drawing for explaining a first audio feature dictionary and a second audio feature dictionary according to one embodiment of the present disclosure.

[0019] FIG. 14 is a flowchart illustrating a method for generating sound effects for a plurality of input images of the same category according to one embodiment of the present disclosure.

[0020] FIG. 15 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that generates sound effects for a plurality of input images of the same category.

[0021] FIG. 16 is a flowchart for explaining a method of playing a video according to one embodiment of the present disclosure.

[0022] FIG. 17 is a drawing illustrating the operation of an electronic device according to one embodiment of the present disclosure playing a video with sound effects.

[0023] FIG. 18 is a block diagram illustrating the function of an electronic device according to one embodiment of the present disclosure.

[0024] FIG. 19 is a detailed block diagram for explaining the function of an electronic device according to one embodiment of the present disclosure.

[0025] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0026] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0027] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0028] Throughout the entire disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as described in the disclosure refer to a unit that processes at least one function or operation, and "...part" or "module" may be implemented in hardware or software, or a combination of hardware and software.

[0029] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0030] Unless the context clearly indicates otherwise, the singular forms (e.g., "a," "an," and "the") may be understood to include plural objects. Thus, for example, the description "a component surface" may include cases where it refers to one or more of such surfaces.

[0031] All functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing and may include circuitry such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.

[0032] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0033] FIG. 1 is a drawing for explaining an electronic device (1000) that provides video according to one embodiment of the present disclosure.

[0034] An electronic device (1000) according to one embodiment of the present disclosure may be implemented in various forms. For example, the electronic device (1000) may be a digital camera, a smartphone, a laptop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), etc., but is not limited thereto. The electronic device (1000) may be a wearable device that can be worn by a user. The wearable device may include at least one of an accessory-type device (e.g., a watch, a ring, a wristband, an ankle band, a necklace, glasses, a contact lens), a head-mounted device (HMD), a fabric or clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit), but is not limited thereto. For convenience of explanation, the following description will be given using the example where the electronic device (1000) is a smartphone.

[0035] According to one embodiment of the present disclosure, an electronic device (1000) can generate a video using some photos stored in memory. For example, the electronic device (1000) can generate a video by selecting some photos related to a specific event or a specific time.

[0036] Referring to 101 in FIG. 1, the electronic device (1000) can automatically or via user input select photos related to a trip to Hawaii and generate a video showing a story of a trip to Hawaii using the selected photos. At this time, the electronic device (1000) can generate a video including the selected photos along with specific background music (BGM).

[0037] According to one embodiment of the present disclosure, the electronic device (1000) may provide sound effects that match each photo included in the video. For example, referring to 102 in FIG. 1, the electronic device (1000) may generate sound effects that match each photo using a generative AI model. In this case, according to one embodiment of the present disclosure, the electronic device (1000) may generate sound effects of a predetermined category by inputting audio features of a predetermined category that are closest to the image features of the photo into the generative AI model, rather than directly inputting image features of the photo into the generative AI model.

[0038] When image features of a photograph are directly input into a generative AI model, sound effects that are strange or do not match the photograph may be generated depending on the photograph. However, according to one embodiment of the present disclosure, by inputting audio features of a predetermined category closest to the image features of the photograph into the generative AI model, sound effects of the desired category can be reliably generated. That is, although there may be various types of photographs, by limiting the audio features that can match the image features of the photograph to a specific category, the effect of limiting the category of photographs suitable for sound effect generation can be achieved.

[0039] In addition, the electronic device (1000) can store audio features that produce sound effects well when input into a generative AI model in advance in a database (e.g., audio feature dictionary) and compare them with image features of a photograph, thereby reducing the variables for generating sound effects according to the photograph and generating sound effects of excellent sound quality. The method by which the electronic device (1000) stably generates sound effects will be examined in detail later with reference to FIG. 4.

[0040] Referring to 103 in FIG. 1, an electronic device (1000) according to one embodiment of the present disclosure can generate a video in which the sound effect of each photo is inserted along with background music when a sound effect corresponding to each photo is generated. And when there is a request for playback from a user, the electronic device (1000) can provide a new video playback experience to the user by playing the video in which the sound effect of each photo is inserted along with background music (BGM).

[0041] Below, we will examine in detail with reference to FIG. 2 the operation of an electronic device (1000) pre-constructing an audio feature dictionary containing audio features of a specific category in order to control the category of the generated sound effect.

[0042] FIG. 2 is a flowchart for explaining a method for constructing a first audio feature dictionary according to one embodiment of the present disclosure. FIG. 3 is a diagram for explaining an operation in which an electronic device (1000) according to one embodiment of the present disclosure constructs a first audio feature dictionary (100).

[0043] Referring to FIG. 2, a method for an electronic device (1000) to construct a first audio feature dictionary may include steps S210 through S240. In one embodiment of the present disclosure, steps S210 through S240 may be executed by at least one processor included in the electronic device (1000). The method for an electronic device (1000) to construct a first audio feature dictionary is not limited to that illustrated in FIG. 2, and in one or more embodiments, additional steps not illustrated in FIG. 2 may be included, or some steps may be omitted.

[0044] In step S210, the electronic device (1000) according to one embodiment of the present disclosure can select categories for generating sound effects.

[0045] According to one embodiment of the present disclosure, an electronic device (1000) may select a category corresponding to the type of sound effect to be generated by reflecting system settings or user settings. For example, the electronic device (1000) may select categories such as water sound, wave sound, wind sound, airplane sound, animal sound, etc., but is not limited thereto.

[0046] According to one embodiment of the present disclosure, an electronic device (1000) can analyze photos stored in a photo management application and select categories for generating sound effects. For example, if the electronic device (1000) mainly stores photos of dogs, cats, the sea, fountains, etc. in the photo management application, it can select categories such as dog sounds, cat sounds, wave sounds, water sounds, etc.

[0047] According to one embodiment of the present disclosure, the electronic device (1000) may adjust selected categories by reflecting user input. For example, the electronic device (1000) may provide a list of selected categories to the user through a user interface. At this time, the user may exclude some categories from the list or add other categories.

[0048] In step S220, an electronic device (1000) according to one embodiment of the present disclosure can obtain reference sound effects included in selected categories.

[0049] For example, if wave sounds and dog sounds are selected as categories for generating sound effects, the electronic device (1000) can collect various wave sounds and various dog sounds as reference sound effects.

[0050] The electronic device (1000) may obtain a reference sound effect from an audio file or video file stored in memory, obtain a reference sound effect from an external device (e.g., a cloud server, an audio search server, etc.), or collect a reference sound effect in real time through a microphone.

[0051] In step S230, an electronic device (1000) according to one embodiment of the present disclosure can obtain audio feature vectors by applying reference sound effects to an audio encoding model.

[0052] An audio encoding model according to one embodiment of the present disclosure may be an artificial intelligence model that converts raw audio (e.g., waveform) into feature embeddings. The audio encoding model may also be represented as an audio encoder.

[0053] Referring to FIG. 3, according to one embodiment of the present disclosure, an electronic device (1000) can input reference sound effects (310) of a specific category collected in step S220 into an audio encoding model (320) to convert the reference sound effects (310) into audio feature vectors (330). The audio encoding model (320) can extract features from the reference sound effects (310) and generate audio feature vectors (330) using the extracted features. At this time, the audio encoding model (320) can convert the reference sound effects (310) into a latent space. For example, the electronic device (1000) can convert a 10-second reference sound effect waveform (e.g., 480,000 samples) into a latent feature of much smaller dimensions (e.g., a 512-dimensional feature vector).

[0054] According to one embodiment of the present disclosure, when the categories of the reference sound effects (310) are similar, the corresponding audio feature vectors (330) may be located close together in the latent space. On the other hand, when the categories of the reference sound effects (310) are different, the corresponding audio feature vectors (330) may be located far apart in the latent space.

[0055] Returning to FIG. 2, in step S240, an electronic device (1000) according to one embodiment of the present disclosure can store audio feature vectors in a first audio feature dictionary.

[0056] A first audio feature dictionary according to one embodiment of the present disclosure may be a database in which audio feature vectors of a category for which sound effect generation is desired are stored, but is not limited thereto.

[0057] Referring to FIG. 3, an electronic device (1000) according to one embodiment of the present disclosure may store audio feature vectors (330) corresponding to reference sound effects (310) of a predetermined category together with meta information (e.g., category) in a first audio feature dictionary (100). Additionally, according to one embodiment of the present disclosure, audio feature vectors (330) may be stored in the first audio feature dictionary (100) by category.

[0058] According to one embodiment of the present disclosure, the electronic device (1000) can update the first audio feature dictionary (100) when the categories selected in step S210 are adjusted. For example, when an airplane sound category is added, the electronic device (1000) can add an audio feature vector corresponding to the reference airplane sound to the first audio feature dictionary (100).

[0059] According to one embodiment of the present disclosure, the electronic device (1000) can efficiently utilize memory space by storing a dimensionally reduced audio feature vector in the first audio feature dictionary (100) without storing the original data of the reference sound effect, and can efficiently generate a sound effect that matches the input image by rapidly calculating the similarity with the image feature vector of the input image. In addition, by pre-storing only the audio feature vector that generates a sound effect well when input to a generative AI model in the first audio feature dictionary (100), a sound effect that matches the image can be stably generated.

[0060] Below, with reference to FIG. 4, we will examine in detail how the electronic device (1000) generates sound effects that match the input image.

[0061] FIG. 4 is a flowchart illustrating a method for an electronic device (1000) according to one embodiment of the present disclosure to generate a sound effect for an image.

[0062] Referring to FIG. 4, a method for an electronic device (1000) to generate a sound effect for an image may include steps S410 to S440. In one embodiment of the present disclosure, steps S410 to S440 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to generate a sound effect for an image is not limited to that illustrated in FIG. 4, and in one or more embodiments, additional steps not illustrated in FIG. 4 may be included, or some steps may be omitted.

[0063] In step S410, an electronic device (1000) according to one embodiment of the present disclosure can acquire an input image.

[0064] According to one embodiment of the present disclosure, an electronic device (1000) can acquire at least one input image for generating a video. The electronic device (1000) can automatically select at least one input image by using metadata (tag information) of previously stored images. For example, the electronic device (1000) may select at least one photo containing a specific person from among photos stored in a photo management application as an input image, or may select at least one photo related to a specific event (e.g., travel, sports day, anniversary, holiday) as an input image.

[0065] According to one embodiment of the present disclosure, the electronic device (1000) may receive user input for selecting at least one input image to be created as a video. For example, the user may select at least one photo to create a video from the execution window of a photo management application.

[0066] According to one embodiment of the present disclosure, the input image may be stored in the memory of an electronic device (1000) or may be stored in an external device (e.g., a cloud server).

[0067] In step S420, an electronic device (1000) according to one embodiment of the present disclosure can obtain an image feature vector by applying an input image to an image encoding model.

[0068] An image encoding model according to one embodiment of the present disclosure may be an artificial intelligence model that converts a raw image into a feature embedding. The image encoding model may also be represented as an audio encoder.

[0069] According to one embodiment of the present disclosure, when an input image is applied to an image encoding model, the image encoding model can convert the input image into a latent space. For example, the image encoding model can convert the input image into an image feature vector of the same dimension as the audio feature vectors stored in a first audio feature dictionary (100, see FIG. 3). The image encoding model can extract features from the input image and generate an image feature vector of the same dimension as the audio feature vectors using the extracted features. In this case, the image feature vector may be represented as an embedding vector. The image encoding model can be trained so that the image feature vector effectively preserves the semantic / contextual features of the image.

[0070] According to one embodiment of the present disclosure, when the categories of input images are similar, corresponding image feature vectors may be located close together in the latent space. On the other hand, when the categories of input images are different, corresponding image feature vectors may be located far apart in the latent space.

[0071] In step S430, an electronic device (1000) according to one embodiment of the present disclosure may select an audio feature vector from a first audio feature dictionary (100) in which audio feature vectors of a predetermined category are stored, the audio feature vector having a similarity to an image feature vector greater than or equal to a threshold value.

[0072] According to one embodiment of the present disclosure, since the audio feature vectors stored in the first audio feature dictionary (1000) and the image feature vectors of the input image are vectors of the same dimension, the electronic device (1000) can compare the image feature vectors of the input image with the audio feature vectors stored in the first audio feature dictionary (100).

[0073] According to one embodiment of the present disclosure, an electronic device (1000) can calculate the similarity between an image feature vector and each audio feature vector stored in a first audio feature dictionary (100). For example, the electronic device (1000) can calculate the similarity by measuring the distance between the image feature vector and each audio feature vector in a latent space. According to one embodiment of the present disclosure, the electronic device (1000) can measure the distance between the image feature vector and each audio feature vector using similarity measurement metrics such as cosine similarity, Euclidean distance, and inner product. The closer the similarity value is to 1, the more similar the two vectors may be determined to be.

[0074] According to one embodiment of the present disclosure, an electronic device (1000) can determine whether the similarity between an image feature vector and each audio feature vector is greater than or equal to a threshold value. That is, it can detect audio feature vectors whose similarity to an image feature vector is greater than or equal to a threshold value. The threshold value may be a reference value for determining whether an image feature vector and an audio feature vector are similar. The threshold value may vary according to system settings or policies. For example, the electronic device (1000) may adjust the threshold value to a lower value if it wants to increase the frequency of sound effect generation for an image, and may adjust the threshold value to a higher value if it wants to increase the matching accuracy between an image and a sound effect.

[0075] When comparing an image feature vector with audio feature vectors, if there are multiple audio feature vectors having a similarity greater than a threshold value, the electronic device (1000) can select the audio feature vector closest to the image feature vector. That is, the electronic device (1000) can select the audio feature vector having the maximum similarity among the audio feature vectors having a similarity greater than a threshold value.

[0076] Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) may determine an audio feature vector having maximum similarity to an image feature vector and determine whether the maximum similarity is greater than or equal to a threshold value. If the maximum similarity is greater than or equal to a threshold value, the electronic device (1000) may select the audio feature vector having maximum similarity as a vector for generating sound effects. The operation of the electronic device (1000) selecting the audio feature vector having maximum similarity will be examined further later with reference to FIG. 5.

[0077] In step S440, an electronic device (1000) according to one embodiment of the present disclosure can generate a sound effect corresponding to an input image by applying a selected audio feature vector to a generative AI model.

[0078] According to one embodiment of the present disclosure, a generative AI model may be an artificial intelligence model trained to generate sound effects by receiving an audio feature vector as input. A generative AI model according to one embodiment of the present disclosure may include a latent diffusion-based audio generation model. For example, the generative AI model may be composed of a latent diffusion model (LDM), a variational autoencoder decoder (VAE Decoder), and a vocoder, but is not limited thereto. A diffusion model is a probabilistic generative model that learns the process of gradually contaminating data with noise and then restoring the original data from that noise, and a latent diffusion model (LDM) may be a model that performs the diffusion process in a latent space. A VAE decoder may be a model that restores a latent space (e.g., a latent vector) to an original space (e.g., a Mel-spectrogram). A vocoder, short for voice coder, may be a model that converts an audio signal from an intermediate representation, such as a spectrogram, into a waveform.

[0079] According to one embodiment of the present disclosure, when an audio feature vector is input, a latent diffusion model can generate a new latent vector based on the audio feature vector and random noise, and a VAE decoder can reconstruct the new latent vector into a spectrogram (MEL-spectrogram). Additionally, a vocoder can generate sound effects suitable for an input image by converting the spectrogram into an actual audio waveform.

[0080] According to one embodiment of the present disclosure, instead of directly inputting image feature vectors into a generative AI model, audio feature vectors closest to image feature vectors are searched from a predetermined first audio feature dictionary (100) and input into the generative AI model, thereby reducing variables for generating sound effects according to images and enabling stable sound effect generation. In particular, by managing the first audio feature dictionary (100) in advance, categories suitable for generating sound effects and categories not suitable for generating sound effects can be designated in advance, thereby allowing control over the types of sound effects generated. Additionally, the electronic device (1000) can stably generate sound effects that match images by pre-storing only audio feature vectors that generate sound effects well when input into the generative AI model in the first audio feature dictionary (100).

[0081] FIG. 5 is a flowchart illustrating a method for extracting an audio feature vector most similar to an image feature vector according to one embodiment of the present disclosure.

[0082] Referring to FIG. 5, a method for an electronic device (1000) to extract an audio feature vector most similar to an image feature vector may include steps S510 to S540. In one embodiment of the present disclosure, steps S510 to S540 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to generate a sound effect for an image is not limited to that illustrated in FIG. 5, and in one or more embodiments, additional steps not illustrated in FIG. 5 may be included, or some steps may be omitted.

[0083] In step S510, an electronic device (1000) according to one embodiment of the present disclosure can obtain similarities between an image feature vector obtained by applying an input image to an image encoding model and each audio feature vector stored in an audio feature dictionary.

[0084] Similarity can be calculated as the distance between an image feature vector and each audio feature vector in the latent space. Similarity can be represented as a value between 0 and 1, but is not limited thereto. For example, audio feature vectors that are closer to an image feature vector may have a similarity value closer to 1.

[0085] In step S520, an electronic device (1000) according to one embodiment of the present disclosure may determine whether the maximum similarity among the similarities is greater than or equal to a threshold value. The threshold value may be a value that serves as a criterion for classifying an image feature vector and an audio feature vector into the same category.

[0086] For example, the electronic device (1000) can identify 0.8 as the maximum similarity when the similarity between the image feature vector and the first audio feature vector is 0.3, the similarity between the image feature vector and the second audio feature vector is 0.2, and the similarity between the image feature vector and the third audio feature vector is 0.8. The electronic device (1000) can determine whether the maximum similarity '0.8' is greater than or equal to a threshold value. If the threshold value is 0.7, the maximum similarity 0.8 may be greater than or equal to the threshold value, and if the threshold value is 0.9, the maximum similarity may be less than the threshold value.

[0087] According to one embodiment of the present disclosure, the electronic device (1000) may not generate a sound effect for an input image when the maximum similarity is less than a threshold value (N0 of S520). That is, the electronic device (1000) may not generate a sound effect for an input image when an audio feature vector very similar to an image feature vector is not stored in a first audio feature dictionary. Accordingly, according to one embodiment of the present disclosure, the electronic device (1000) may control the category of the image for which a sound effect is generated by restricting the audio feature vectors stored in the first audio feature dictionary to a specific category.

[0088] In step S530, an electronic device (1000) according to one embodiment of the present disclosure may select an audio feature vector having the maximum similarity among the audio feature vectors stored in the first audio feature dictionary for sound effect generation when the maximum similarity is greater than or equal to a threshold value (Yes in S520).

[0089] For example, if the input image is a photograph of luggage being loaded before an airplane takes off, the audio feature vector with maximum similarity may be an audio feature vector corresponding to the sound effect that occurs during the airplane takeoff. That is, the image feature vector of the photograph of luggage being loaded before the airplane takeoff may be closest to the audio feature vector of the sound effect that occurs during the airplane takeoff.

[0090] In step S540, an electronic device (1000) according to one embodiment of the present disclosure can generate a sound effect corresponding to an input image by applying an audio feature vector having maximum similarity to an image feature vector to a generative AI model.

[0091] According to one embodiment of the present disclosure, a generative AI model can predict a sound effect corresponding to an audio feature vector having maximum similarity while gradually removing noise from random noise. For example, if the input image is a photograph of loading luggage before an airplane takes off, the generative AI model can generate a sound effect that occurs during airplane takeoff based on random noise and an audio feature vector having maximum similarity as a sound effect for the input image (a photograph of loading luggage before an airplane takes off).

[0092] Below, we will examine in detail, with reference to FIG. 6, the operation of an electronic device (1000) generating sound effects for an input image using a latent diffusion-based audio generation model.

[0093] FIG. 6 is a diagram illustrating the operation of an electronic device (1000) according to one embodiment of the present disclosure generating sound effects using a generative AI model.

[0094] Referring to FIG. 6, an electronic device (1000) can acquire an input image (610) for generating sound effects. For example, the electronic device (1000) can acquire a beach photo as the input image (610). The electronic device (1000) can apply the input image to an image encoding model (620) to generate an image feature vector (630). Then, the electronic device (1000) can identify an audio feature vector in the first audio feature dictionary (100) that is most similar to the image feature vector (630) and has a similarity level greater than or equal to a threshold value.

[0095] The first audio feature dictionary (100) can be constructed in advance before acquiring the input image (610). For example, the electronic device (1000) can select categories to be generated as sound effects and collect reference sound effects (310) of the selected categories. The electronic device (1000) can input the collected reference sound effects (310) into an audio encoding model (320) to convert them into audio feature vectors (330) in a latent space. The electronic device (1000) can store the audio feature vectors (330) in the first audio feature dictionary (100). Since the operation of the electronic device (1000) preparing the first audio feature dictionary (100) in advance has been described in FIG. 3, a redundant description will be omitted.

[0096] According to one embodiment of the present disclosure, an electronic device (1000) can determine similarity by comparing the distance between an image feature vector (630) and each audio feature vector stored in a first audio feature dictionary (100). The electronic device (1000) can select the audio feature vector (640) closest to the image feature vector (630). At this time, if the similarity between the image feature vector (630) and the closest audio feature vector (640) is less than a threshold value (threshold similarity), the electronic device (1000) may not generate sound effects for the input image. On the other hand, if the similarity between the image feature vector (630) and the closest audio feature vector (640) is greater than or equal to the threshold value (threshold similarity), the electronic device (1000) may input the audio feature vector (640) into a generative AI model to generate sound effects.

[0097] Meanwhile, according to one embodiment of the present disclosure, since the image feature vector and the audio feature vector of the first audio feature dictionary (100) must be in the same latent space for comparison (matching) to be possible, the image feature vector and the audio feature vector may be vectors of the same dimension. For example, the image feature vector and the audio feature vector may be vectors of 512 dimensions, but are not limited thereto. Meanwhile, the audio encoding model (320) and the image encoding model (620) may be trained together using various audio (sound effect) and image pairs as training data. For example, the audio encoding model (320) and the image encoding model (620) may be trained such that the distance between corresponding audio feature vectors and image feature vectors is close, and the distance between unrelated audio feature vectors and image feature vectors is far apart.

[0098] According to one embodiment of the present disclosure, an electronic device (1000) can generate a sound effect by inputting an audio feature vector (640) closest to an image feature vector (630) into a generative AI model. At this time, the generative AI model may include a latent diffusion model (LDM) (650), a VAE decoder (660), and a vocoder (670). When the audio feature vector (640) is input to the latent diffusion model (650), the latent diffusion model (650) can generate audio (sound effect) by progressively denoising noise from random noise (651) using a noise prediction network (e.g., Denoising U-net) (652). At this time, the audio feature vector (640) is used as a condition, so that the noise prediction network (652) can denoise the noise to generate audio (sound effect) that maintains the category of the audio feature vector (640).

[0099] According to one embodiment of the present disclosure, audio (sound effect) generated through a latent diffusion model (650) can be represented as a three-dimensional vector in latent space. That is, the latent diffusion model (650) can receive an audio feature vector (640) as input and output a three-dimensional vector (Width X Height X Dimension). A VAE decoder (660) can restore a vector sampled in latent space to the original data space. For example, the VAE decoder (660) can convert the three-dimensional vector output from the latent diffusion model (650) into a Mel spectrogram (time-frequency representation). A vocoder (670) may be a model that converts the Mel spectrogram into an actual audio waveform. Thus, the vocoder (670) can generate a sound effect (680) for an input image (610) by converting the Mel spectrogram output from the VAE decoder (660) into an actual audio waveform. For example, the electronic device (1000) can generate a wave sound as a sound effect for an input image (610) (e.g., a picture of a beach).

[0100] According to one embodiment of the present disclosure, the electronic device (1000) may generate sound effects more stably through batch processing of audio feature vectors input to a generative AI model. With reference to FIG. 7, we will examine in detail the operation of the electronic device (1000) generating sound effects through batch processing.

[0101] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that generates sound effects through batch processing.

[0102] In FIG. 7, the operation of identifying the audio feature vector (640) closest (most similar) to the image feature vector (630) of the input image (610) in the first audio feature dictionary (100) is the same as in FIG. 6, so a redundant description will be omitted.

[0103] Referring to FIG. 7, the electronic device (1000) can generate a predetermined number of audio feature vectors (640) by copying the audio feature vector (640) closest to the image feature vector (630). For example, the electronic device (1000) can copy the audio feature vector (640) to create three audio feature vectors (640). The three audio feature vectors (640) may be identical. FIG. 7 describes the case where the predetermined number is three, but it is not limited thereto.

[0104] According to one embodiment of the present disclosure, an electronic device (1000) can generate three candidate sound effects (711, 712, 713) by applying three audio feature vectors (640) to a generative AI model. For example, when the electronic device (1000) inputs three audio feature vectors (640) to a latent diffusion model (650), the latent diffusion model (650) can generate three audio (sound effects) by progressively denoising noise from a plurality of random noises (651) using a noise prediction network (e.g., Denoising U-net) (652). The three audio (sound effects) generated through the latent diffusion model (650) can be represented as three-dimensional vectors in latent space. Since different random noise (651) is used for three audio feature vectors (640), the three-dimensional vectors, which are the output values ​​of the latent diffusion model (650), may differ even if the three audio feature vectors (640) are identical. The VAE decoder (660) can convert the three three-dimensional vectors output from the latent diffusion model (650) into three Mel spectrograms. And the vocoder (670) can generate three candidate sound effects (711, 712, 713) for the input image (610) by converting the three Mel spectrograms output from the VAE decoder (660) into actual audio waveforms.

[0105] According to one embodiment of the present disclosure, an electronic device (1000) can obtain a plurality of audio feature vectors corresponding to a plurality of candidate sound effects by applying each of a plurality of candidate sound effects to an audio encoding model (320). Then, based on the result of comparing a plurality of audio feature vectors with an image feature vector (630), the electronic device (1000) can determine the sound effect corresponding to the audio feature vector with the greatest similarity to the image feature vector (630) among the plurality of sound effects as the sound effect corresponding to the input image (610). For example, the electronic device (1000) can obtain three audio feature vectors (721, 722, 723) by applying each of three candidate sound effects (711, 712, 713) to an audio encoding model (320). The electronic device (1000) can calculate the similarity by comparing the three audio feature vectors (721, 722, 723) with the image feature vector (630). For example, the similarity between the first audio feature vector (721) and the image feature vector (630) may be 0.7, the similarity between the second audio feature vector (722) and the image feature vector (630) may be 0.3, and the similarity between the third audio feature vector (723) and the image feature vector (630) may be 0.5. In this case, since the similarity between the first audio feature vector (721) and the image feature vector (630) is the greatest, the electronic device (1000) may determine the first candidate sound effect (711) corresponding to the first audio feature vector (721) as the final sound effect corresponding to the input image (610).

[0106] According to one embodiment of the present disclosure, an electronic device (1000) can generate multiple candidate sound effects by copying the audio feature vector (640) closest to the image feature vector (630) and inputting it into a generative AI model, and then verifying the relationship between each candidate sound effect and the input image (610) once again to generate an optimal sound effect that fits the input image (610).

[0107] Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) may generate two or more sound effects for a single input image. Hereinafter, with reference to FIG. 8, we will examine in detail how the electronic device (1000) generates a plurality of sound effects.

[0108] FIG. 8 is a flowchart illustrating a method for generating a plurality of sound effects for an input image according to one embodiment of the present disclosure.

[0109] Referring to FIG. 8, a method for an electronic device (1000) to generate a plurality of sound effects for an input image may include steps S810 to S890. In one embodiment of the present disclosure, steps S810 to S890 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to generate a plurality of sound effects for an input image is not limited to that illustrated in FIG. 8, and in one or more embodiments, additional steps not illustrated in FIG. 8 may be included, or some steps may be omitted.

[0110] In step S810, an electronic device (1000) according to one embodiment of the present disclosure can obtain a global feature vector by performing a pooling operation on the entire feature map corresponding to the input image.

[0111] According to one embodiment of the present disclosure, an electronic device (1000) may input an input image into an image encoder to obtain a feature map for the input image. The feature map may be the result of extracting features (e.g., edges, patterns, textures, object shapes, etc.) from the input image. The image encoder may include a convolution layer.

[0112] According to one embodiment of the present disclosure, an electronic device (1000) can obtain a global feature vector by performing a pooling operation on the entire feature map. Pooling may be a downsampling technique that reduces the size of the feature map (e.g., spatial dimension, HXW). Pooling may include, but is not limited to, max pooling, average pooling, and min pooling. Max pooling may select only the largest value within a predetermined area (e.g., 2X2). Average pooling may summarize the values ​​within a predetermined area into an average value. Average pooling may summarize the overall trend. Min pooling may select only the smallest value within a predetermined area (e.g., 2X2). Hereinafter, the electronic device (1000) will be described as an example of performing average pooling.

[0113] According to one embodiment of the present disclosure, since the global feature vector is a feature map compressed into a single representative value, spatial information is lost and only the overall features can be retained.

[0114] In step S820, an electronic device (1000) according to one embodiment of the present disclosure can obtain a plurality of local feature vectors by dividing a feature map corresponding to an input image into window units of a predetermined size and performing a pooling operation.

[0115] According to one embodiment of the present disclosure, an electronic device (1000) may obtain a plurality of local feature vectors by performing local pooling on a feature map. Local pooling may be an operation that reduces the size of a feature map by dividing the feature map into small area units and extracting representative values. Local pooling may include, but is not limited to, max pooling, mean pooling, and min pooling.

[0116] According to one embodiment of the present disclosure, each local feature vector may be a summary that preserves the local features of an input image. Each local feature vector may be different from or identical to one another.

[0117] In step S830, an electronic device (1000) according to one embodiment of the present disclosure can determine whether the first maximum similarity between a global feature vector and audio feature vectors stored in a first audio feature dictionary (100) is greater than or equal to a threshold value.

[0118] According to one embodiment of the present disclosure, an electronic device (1000) can calculate the similarity between a global feature vector and each audio feature vector stored in a first audio feature dictionary (100). For example, the electronic device (1000) can calculate the similarity by measuring the distance between the global feature vector and each audio feature vector in a latent space. According to one embodiment of the present disclosure, the electronic device (1000) can measure the distance between the global feature vector and each audio feature vector using similarity measurement metrics such as cosine similarity, Euclidean distance, and inner product. When the electronic device (1000) calculates the similarity between the global feature vector and each audio feature vector, it can identify the maximum similarity. At this time, the maximum similarity can be expressed as a first maximum similarity. When the first maximum similarity is identified, it can determine whether the first maximum similarity is greater than or equal to a threshold value (threshold similarity).

[0119] Step S830 corresponds to Step S520 of FIG. 5, so a detailed explanation will be omitted.

[0120] According to one embodiment of the present disclosure, if the first maximum similarity between a global feature vector and audio feature vectors is less than a threshold value (No of S830), it can be seen that an audio feature vector similar to the global feature vector is not stored in the first audio feature dictionary (100), so the electronic device (1000) may not generate sound effects associated with the global feature vector.

[0121] In step S840, an electronic device (1000) according to one embodiment of the present disclosure may select a first audio feature vector having a first maximum similarity as a vector for generating sound effects if the first maximum similarity between a global feature vector and audio feature vectors is greater than or equal to a threshold value (Yes in S830).

[0122] In step S850, an electronic device (1000) according to one embodiment of the present disclosure can determine whether a second maximum similarity between a plurality of local feature vectors and audio feature vectors stored in a first audio feature dictionary (100) is greater than or equal to a threshold value.

[0123] According to one embodiment of the present disclosure, an electronic device (1000) can calculate similarities between each local feature vector and each audio feature vector. The electronic device (1000) can identify the maximum similarity among the similarities. The maximum similarity is referred to as the second maximum similarity. The electronic device (1000) can determine whether the second maximum similarity is greater than or equal to a threshold value (threshold similarity).

[0124] In step S860, an electronic device (1000) according to one embodiment of the present disclosure may generate a first sound effect by applying a first audio feature vector to a generative AI model when the first maximum similarity is greater than or equal to a threshold value (S830) and the second maximum similarity is less than a threshold value (No of S850). That is, when the second maximum similarity is less than a threshold value, it can be seen that an audio feature vector similar to each local feature vector is not stored in the first audio feature dictionary (100), so the electronic device (1000) may generate only the first sound effect related to the global feature vector and not generate the second sound effect related to the local feature vectors.

[0125] In step S870, an electronic device (1000) according to one embodiment of the present disclosure may select a second audio feature vector having a second maximum similarity as a vector for generating sound effects if the second maximum similarity is greater than or equal to a threshold value (Yes in S850).

[0126] In step S880, an electronic device (1000) according to one embodiment of the present disclosure can generate a first sound effect and a second sound effect for an input image by applying a first audio feature vector associated with a global feature vector and a second audio feature vector associated with a local feature vector to a generative AI model.

[0127] According to one embodiment of the present disclosure, an electronic device (1000) can generate a first sound effect and a second sound effect by applying a first audio feature vector and a second audio feature vector in parallel to a generative AI model. For example, the electronic device (1000) can input the first audio feature vector and the second audio feature vector into a generative AI model by batch processing. A batch may refer to a group of data samples processed at once.

[0128] For example, if the input image is a picture of a park with rain falling or a cat sitting in a part of the picture, the electronic device (1000) can input a first audio feature vector most similar to a global feature vector into a generative AI model to generate a rain sound as a first sound effect, and input a second audio feature vector most similar to a local feature vector into a generative AI model to generate a cat meow sound as a second sound effect.

[0129] In step S890, an electronic device (1000) according to one embodiment of the present disclosure can synthesize a first sound effect and a second sound effect to generate a sound effect corresponding to an input image.

[0130] According to one embodiment of the present disclosure, a first sound effect and a second sound effect can be synthesized so that they can be output simultaneously. Accordingly, the first sound effect and the second sound effect can be output simultaneously while the input image is subsequently played. For example, while a picture of a cat sitting in a rainy park is played, the sound of rain and the sound of a cat meowing can be output.

[0131] FIG. 8 describes an example in which the electronic device (1000) generates a first sound effect and a second sound effect as sound effects for an input image, but it is not limited thereto. The electronic device (1000) may generate three or more sound effects. For example, the electronic device (1000) may generate a first sound effect, a second sound effect, and a third sound effect by using a first audio feature vector most similar to a global feature vector, a second audio feature vector most similar to local feature vectors, and a third audio feature vector most similar to local feature vectors after the second audio feature vector. For example, if the input image is a photograph of a dog and a cat taken in a rainy park, the electronic device (1000) may generate a rain sound as a first sound effect in relation to the global feature vector, and generate a dog sound and a cat sound as a second sound effect and a third sound effect in relation to the local feature vectors.

[0132] We will look further into the operation in which the electronic device (1000) batch processes the first audio feature vector and the second audio feature vector and inputs them into a generative AI model with reference to FIG. 9.

[0133] FIG. 9 is a diagram illustrating the operation of an electronic device (1000) according to one embodiment of the present disclosure to generate a plurality of sound effects using a global feature vector and local feature vectors.

[0134] Referring to FIG. 9, an electronic device (1000) can acquire an input image (900) for generating sound effects. For example, the electronic device (1000) can acquire a picture of a puppy playing on a beach as the input image (900). The electronic device (1000) can apply the input image to an image encoder (910) to acquire a feature map (901). The image encoder (910) may be a Convolutional Neural Network (CNN). The image encoder (910) can generate the feature map (901) by extracting important features of the input image using a convolutional filter (e.g., 3x3, 5x5 kernel).

[0135] The electronic device (1000) can obtain a global feature vector (902) by performing global pooling (920) on a feature map (901). For example, the electronic device (1000) can obtain a global feature vector (902) representing the average value of the entire feature map by performing pooling only once over the entire area of ​​the feature map. At this time, if the dimension (e.g., 960) of the global feature vector (902) is different from the dimension (e.g., 512) of the audio feature vectors stored in the first audio feature dictionary (100), the electronic device (1000) can input the global feature vector (902) into a linear model (940) to generate a global feature vector (904) that has the same dimension (e.g., 512) as the audio feature vectors.

[0136] Meanwhile, the electronic device (1000) can obtain multiple local feature vectors by performing local pooling (930) on the feature map (901). For example, if the size of the feature map is 8X8, the electronic device (1000) can generate 4X4 local feature vectors (903) using a 2X2 filter. That is, the electronic device (1000) can obtain 16 local feature vectors (903) representing the average feature value of each 2X2 area by performing pooling on each 2X2 area. At this time, if the dimension (e.g., 960) of the local feature vectors (903) is different from the dimension (e.g., 512) of the audio feature vectors stored in the first audio feature dictionary (100), the electronic device (1000) can input the local feature vectors (903) into a linear model (940) to generate local feature vectors (905) that have the same dimension (e.g., 512) as the audio feature vectors.

[0137] The electronic device (1000) can calculate the similarity between the global feature vector (904) and the audio feature vectors stored in the first audio feature dictionary (100), and can calculate the similarity between the local feature vectors (905) and the audio feature vectors stored in the first audio feature dictionary (100) (S950). For example, the electronic device (1000) can identify a first maximum similarity by calculating the similarity between the global feature vector (904) and the audio feature vectors stored in the first audio feature dictionary (100). Additionally, it can identify a second maximum similarity by calculating the similarity between the local feature vectors (905) and the audio feature vectors stored in the first audio feature dictionary (100).

[0138] The electronic device (1000) may select a first audio feature vector (906) having a first maximum similarity as a vector for generating sound effects when the first maximum similarity is greater than or equal to a threshold value (threshold similarity) (S960). The electronic device (1000) may select a second audio feature vector (907) having a second maximum similarity as a vector for generating sound effects (S970). Since the input image (900) is a picture of a puppy playing on the beach, the first audio feature vector (906) may be an audio feature vector related to the sound of waves, and the second audio feature vector (907) may be an audio feature vector related to the sound of a puppy.

[0139] When the first audio feature vector (906) and the second audio feature vector (907) are selected, the electronic device (1000) can input the first audio feature vector (906) and the second audio feature vector (907) into a generative AI model to generate a first sound effect and a second sound effect. The generative AI model may include a latent diffusion model (650), a VAE decoder (660), and a vocoder (670). For example, the electronic device (1000) can input the first audio feature vector (906) and the second audio feature vector (907) into the latent diffusion model (650). At this time, the latent diffusion model (650) can output a first three-dimensional vector corresponding to the first audio feature vector (906) and a second three-dimensional vector corresponding to the second audio feature vector (907) by gradually removing noise from the first random noise and the second random noise sampled from a Gaussian distribution (normal distribution). When the VAE decoder (660) receives a first dimensional vector and a second dimensional vector as input, it can convert the first dimensional vector into a first Mel spectrogram and convert the second dimensional vector into a second Mel spectrogram. When the vocoder (670) receives the first Mel spectrogram and the second Mel spectrogram as input, it can convert the first Mel spectrogram into a first sound effect (first audio waveform) and convert the second Mel spectrogram into a second sound effect (second audio waveform) and output them. The electronic device (1000) can merge the first sound effect and the second sound effect output from the vocoder (670) to generate a final sound effect (908) for the input image (900). Merging the first sound effect and the second sound effect may mean processing so that the first sound effect and the second sound effect can be output simultaneously while the input image (900) is being played.

[0140] For example, since the input image (900) is a picture of a puppy playing on a beach, the electronic device (1000) can generate a wave sound as a first sound effect associated with a global feature vector and a puppy sound as a second sound effect associated with a local feature vector. The electronic device (1000) can also combine the wave sound and the puppy sound to generate a sound effect for the input image (900). Thus, the wave sound and the puppy sound can be output together while the picture of the puppy playing on the beach is being played.

[0141] Meanwhile, according to one embodiment of the present disclosure, if one of the first maximum similarity and the second maximum similarity is less than a threshold value, the electronic device (1000) may generate only one sound effect. For example, if the first maximum similarity is less than a threshold value and the second maximum similarity is greater than or equal to a threshold value, the electronic device (1000) may select only the second audio feature vector (907) having the second maximum similarity as a vector for generating sound effects and input the second audio feature vector (907) into a generative AI model. On the other hand, if the first maximum similarity is greater than or equal to a threshold value and the second maximum similarity is less than a threshold value, the electronic device (1000) may select only the first audio feature vector (906) having the first maximum similarity as a vector for generating sound effects and input the first audio feature vector (906) into a generative AI model. For example, if the input image (900) is a beach photograph containing a seagull flying in the sky, an audio feature vector similar to the global feature vector (904) representing the beach may be stored in the first audio feature dictionary (100) as being related to the sound of waves. On the other hand, an audio feature vector similar to the local feature vector representing the seagull may not be stored in the first audio feature dictionary (100) as being related to the sound of a seagull crying. Instead, if an audio feature vector representing a chicken crying sound is stored in the first audio feature dictionary (100), the electronic device (1000) may extract the audio feature vector representing the chicken crying sound as the audio feature vector most similar to the local feature vector representing the seagull, but the similarity between the local feature vector representing the seagull and the audio feature vector representing the chicken crying sound may be below a threshold value. In this case, the electronic device (1000) may generate wave sounds as sound effects for the input image (900), but may not generate seagull cries as sound effects for the input image (900).

[0142] Meanwhile, according to one embodiment of the present disclosure, if the first audio feature vector (906) closest to the global feature vector (904) and the second audio feature vector (907) closest to one of the local feature vectors (905) are the same, the electronic device (1000) may select the third audio feature vector closest to one of the local feature vectors (905) next to the second audio feature vector (907) as a vector for generating sound effects instead of the second audio feature vector (907). For example, if the input image (900) is a photograph of a puppy playing on a beach, the first audio feature vector (906) may be related to the sound of waves, the second audio feature vector (907) may be related to the sound of waves, and the third audio feature vector may be related to the sound of a puppy. In this case, the electronic device (1000) may select the first audio feature vector (906) and the third audio feature vector as vectors for generating sound effects and apply them to a generative AI model.

[0143] According to one embodiment of the present disclosure, the electronic device (1000) may fuse the first audio feature vector (906) and the second audio feature vector (907) into a single vector before inputting them into a generative AI model. The operation of the electronic device (1000) fusing the first audio feature vector (906) and the second audio feature vector (907) into a single vector and inputting them into a generative AI model will be examined in detail with reference to FIGS. 10 and FIGS. 11.

[0144] FIG. 10 is a flowchart illustrating a method for generating a plurality of sound effects using a feature fusion model according to one embodiment of the present disclosure.

[0145] The operation of the electronic device (1000) selecting the first audio feature vector closest to the global feature vector and the second audio feature vector closest to the local feature vector from the first audio feature dictionary (100) is the same as the description in FIG. 8, so a repeated description will be omitted.

[0146] In step S1010, an electronic device (1000) according to one embodiment of the present disclosure can obtain a fused audio feature vector by applying a first audio feature vector and a second audio feature vector to a feature fusion model.

[0147] According to one embodiment of the present disclosure, the feature fusion model may be an artificial intelligence model trained to fuse a plurality of feature vectors into a single vector while preserving each feature of a plurality of feature vectors. Accordingly, when an electronic device (1000) inputs a first audio feature vector and a second audio feature vector to the feature fusion model, the feature fusion model may output a single fused audio feature vector that effectively preserves each feature of the first audio feature vector and the second audio feature vector.

[0148] In step S1020, the electronic device (1000) can apply the fused audio feature vector to a generative AI model to generate a sound effect corresponding to an input image, in which a first sound effect corresponding to a first audio feature vector and a second sound effect corresponding to a second audio feature vector are synthesized.

[0149] According to one embodiment of the present disclosure, when an electronic device (1000) applies a single fused audio feature vector to a generative AI model compared to applying a first audio feature vector and a second audio feature vector to a generative AI model, the amount of computation of the generative AI model can be reduced by half.

[0150] The operation of the electronic device (1000) fusing the first audio feature vector and the second audio feature vector and inputting them into a generative AI model will be examined in detail with reference to FIG. 11.

[0151] FIG. 11 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure applying fused audio feature vectors to a generative AI model using a feature fusion model.

[0152] Since the operation of the electronic device (1000) selecting the first audio feature vector (906) closest to the global feature vector (904) and the second audio feature vector (907) closest to one of the plurality of local feature vectors (905) as a vector for generating sound effects is the same as described in FIG. 9, we will describe the next operation first.

[0153] Referring to FIG. 11, the electronic device (1000) can fuse the first audio feature vector (906) and the second audio feature vector (907) into a single audio feature vector (1120) using a feature fusion model (1110) before inputting them into a generative AI model for generating sound effects. The feature fusion model (1110) can fuse the first audio feature vector (906) and the second audio feature vector (907) into one while preserving each of their respective features.

[0154] When a fused audio feature vector (1120) is output from a feature fusion model (1110), the electronic device (1000) can input the fused audio feature vector (1120) into a generative AI model to generate a sound effect (1130). The generative AI model may include a latent diffusion model (650), a VAE decoder (660), and a vocoder (670).

[0155] For example, an electronic device (1000) can input a fused audio feature vector (1120) into a latent diffusion model (650). At this time, the latent diffusion model (650) can output a new audio 3D vector by gradually removing noise from random noise sampled from a Gaussian distribution (normal distribution). When the VAE decoder (660) receives the 3D vector, it can convert the 3D vector into a Mel spectrogram. When the vocoder (670) receives the Mel spectrogram, it can convert the Mel spectrogram into an audio waveform to generate a sound effect (1130). At this time, the sound effect (1130) may be a sound effect in which a first sound effect corresponding to the first audio feature vector and a second sound effect corresponding to the second audio feature vector are combined.

[0156] In FIG. 11, the feature fusion model (1110) is described as an example of fusing a first audio feature vector (906) and a second audio feature vector (907), but is not limited thereto. According to one embodiment of the present disclosure, the feature fusion model (1110) may fuse three or more audio feature vectors into a single audio feature vector.

[0157] Meanwhile, the electronic device (1000) may further include a second audio feature dictionary (200, see FIG. 13) in which audio feature vectors of a category not to be generated are stored, in addition to a first audio feature dictionary (100) in which audio feature vectors of a category not to be generated are stored. In this case, the electronic device (1000) can generate sound effects (1130) related to the first audio feature vector (906) and the second audio feature vector (907) in which the first audio feature vector (906) closest to the global feature vector (904) and the second audio feature vector (907) closest to one of a plurality of local feature vectors (905) are stored in the first audio feature dictionary (100). On the other hand, the electronic device (1000) may not generate sound effects (1130) associated with the first audio feature vector (906) and the second audio feature vector (907) when the first audio feature vector (906) and the second audio feature vector (907) are stored in the second audio feature dictionary (200, see FIG. 13). Below, we will examine in detail the operation of the electronic device (1000) managing the first audio feature dictionary (100) and the second audio feature dictionary (200, see FIG. 13) with reference to FIG. 12 and FIG. 13.

[0158] FIG. 12 is a flowchart for explaining a method for generating a sound effect when an audio feature vector most similar to an image feature vector according to one embodiment of the present disclosure is stored in a first audio feature dictionary (100).

[0159] Referring to FIG. 12, a method for an electronic device (1000) to generate a sound effect when an audio feature vector most similar to an image feature vector is stored in a first audio feature dictionary (100) may include steps S1210 to S1240. In one embodiment of the present disclosure, steps S1210 to S1240 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to generate a sound effect when an audio feature vector most similar to an image feature vector is stored in a first audio feature dictionary (100) is not limited to that illustrated in FIG. 12, and in one or more embodiments, additional steps not illustrated in FIG. 12 may be included, or some steps may be omitted.

[0160] In step S1210, an electronic device (1000) according to one embodiment of the present disclosure may generate a first audio feature dictionary (100) in which audio features of a category to be generated are stored, and a second audio feature dictionary in which audio feature vectors of a category not to be generated are stored.

[0161] According to one embodiment of the present disclosure, an electronic device (1000) may generate a first audio feature dictionary (100) in which audio features of a category for which sound effects are desired are stored. For example, the electronic device (1000) may select first categories for which sound effects are desired. The electronic device (1000) may collect first reference sound effects included in the selected first categories. The electronic device (1000) may obtain first audio feature vectors by applying the first reference sound effects to an audio encoding model (320). The electronic device (1000) may generate the first audio feature dictionary (100) by storing the first audio feature vectors in the first audio feature dictionary (100). Since the operation of generating the first audio feature dictionary (100) corresponds to steps S210 to S240 of FIG. 2, a redundant description will be omitted.

[0162] According to one embodiment of the present disclosure, the electronic device (1000) may generate a second audio feature dictionary in which audio feature vectors of categories for which sound effect generation is not desired are stored. For example, the electronic device (1000) may select second categories for which sound effect generation is not desired.

[0163] The electronic device (1000) may select second categories for which sound effects are not to be generated, reflecting system settings or user settings. For example, the electronic device (1000) may select second categories such as crowd sounds, screams, and crying sounds, but is not limited thereto. The electronic device (1000) may also adjust the second categories by reflecting user input.

[0164] The electronic device (1000) can collect second reference sound effects included in second categories. The electronic device (1000) may obtain second reference sound effects from an audio file or video file already stored in memory, obtain second reference sound effects from an external device (e.g., cloud server, audio search server, etc.), or collect second reference sound effects in real time through a microphone.

[0165] The electronic device (1000) can obtain second audio feature vectors by applying second reference sound effects to an audio encoding model (320). The second audio feature vectors may be located in a latent space. The electronic device (1000) can generate a second audio feature dictionary by storing the second audio feature vectors in a second audio feature dictionary.

[0166] FIG. 13 is a drawing for explaining a first audio feature dictionary and a second audio feature dictionary according to one embodiment of the present disclosure.

[0167] Referring to FIG. 13, the first audio feature dictionary (100) can store audio feature vectors related to categories for which sound effects are desired, such as wave sounds, wind sounds, airplane sounds, and dog sounds. The second audio feature dictionary (200) can store audio feature vectors related to categories for which sound effects are not desired, such as crowd sounds (e.g., murmuring sounds, sounds of others talking, etc.), crying sounds, and screaming sounds. The first audio feature dictionary (100) and the second audio feature dictionary (200) can be updated to reflect user preferences / non-preferences for sound effects. Additionally, the user can check the categories included in the first audio feature dictionary (100) and the second audio feature dictionary (200).

[0168] Meanwhile, although not illustrated in FIG. 13, meta-information corresponding to each audio feature vector may be stored together in the first audio feature dictionary (100) and the second audio feature dictionary (200). The meta-information may include category information (label) to which each audio feature vector belongs, dimensions (size), type of value, scaling status, input data, model information, extraction layer, preprocessing method, etc., but is not limited thereto.

[0169] Returning to FIG. 12, in step S1220, an electronic device (1000) according to one embodiment of the present disclosure can determine similarities between all audio feature vectors and image feature vectors stored in a first audio feature dictionary (100) and a second audio feature dictionary (200).

[0170] Similarity can be calculated as the distance between an image feature vector and each audio feature vector in the latent space. Similarity can be represented as a value between 0 and 1, but is not limited thereto. For example, audio feature vectors that are closer to an image feature vector may have a similarity value closer to 1.

[0171] In step S1230, the electronic device (1000) according to one embodiment of the present disclosure can determine whether the maximum similarity is greater than or equal to a threshold value.

[0172] According to one embodiment of the present disclosure, an electronic device (1000) can identify the maximum similarity among all similarities between image feature vectors and all audio feature vectors stored in a first audio feature dictionary (100) and a second audio feature dictionary (200). The electronic device (1000) can determine whether the maximum similarity is greater than or equal to a threshold value (threshold similarity). For example, the electronic device (1000) can determine that the similarity between an image feature vector and a first audio feature vector is 0.95, which is the maximum similarity. The electronic device (1000) can determine that 0.95 is greater than or equal to a threshold value (e.g., 0.8).

[0173] According to one embodiment of the present disclosure, if the maximum similarity is less than a threshold value (No of S1230), the electronic device (1000) may not generate a sound effect for the input image. If the maximum similarity is less than a threshold value, it can be seen that no audio feature vectors similar in category to the image feature vector of the input image are stored in the first audio feature dictionary (100), so the electronic device (1000) may not generate a sound effect for the input image. That is, the electronic device (1000) can control the category of the image for which a sound effect is to be generated by restricting the audio feature vectors stored in the first audio feature dictionary (100) to a specific category.

[0174] In step S1240, an electronic device (1000) according to one embodiment of the present disclosure can determine whether an audio feature vector having maximum similarity to an image feature vector is included in the first audio feature dictionary (100) when the maximum similarity is greater than or equal to a threshold value (Yes in S1230).

[0175] According to one embodiment of the present disclosure, an electronic device (1000) can determine whether an audio feature vector having maximum similarity to an image feature vector is included in a first audio feature dictionary (100) or in a second audio feature dictionary (200).

[0176] According to one embodiment of the present disclosure, an electronic device (1000) may not generate sound effects for an input image when the maximum similarity is greater than or equal to a threshold value (Yes of S1230) and an audio feature vector having maximum similarity to an image feature vector is included in the second audio feature dictionary (200) (No of S1240). That is, since the audio feature vectors included in the second audio feature dictionary (200) are related to categories for which sound effects are not desired, the electronic device (1000) may not generate any sound effects for the input image when an audio feature vector having maximum similarity to an image feature vector (the audio feature vector closest to the image feature vector) is included in the second audio feature dictionary (200).

[0177] In step S1250, an electronic device (1000) according to one embodiment of the present disclosure can generate a sound effect corresponding to an input image by applying an audio feature vector having maximum similarity to an image feature vector to a generative AI model when the maximum similarity is greater than or equal to a threshold value (Yes of S1230) and an audio feature vector having maximum similarity is included in the first audio feature dictionary (100) (Yes of S1240).

[0178] For example, an electronic device (1000) can input an audio feature vector having maximum similarity into a latent diffusion model (650). At this time, the latent diffusion model (650) can output a 3D vector of generated audio by gradually removing noise from random noise sampled from a Gaussian distribution (normal distribution). When the VAE decoder (660) receives the 3D vector, it can convert the 3D vector into a Mel spectrogram. When the vocoder (670) receives the Mel spectrogram, it can convert the Mel spectrogram into an audio waveform to generate a sound effect.

[0179] According to one embodiment of the present disclosure, by managing a second audio feature dictionary (200) for which sound effect generation is not desired, an error may occur when calculating the similarity between an image feature vector and audio feature vectors stored in a first audio feature dictionary (100), thereby preventing the generation of sound effects that the user does not want.

[0180] FIG. 14 is a flowchart illustrating a method for generating sound effects for a plurality of input images of the same category according to one embodiment of the present disclosure.

[0181] Referring to FIG. 14, a method for an electronic device (1000) to generate sound effects for a plurality of input images of the same category may include steps S1410 through S1470. In one embodiment of the present disclosure, steps S1410 through S1470 may be executed by at least one processor included in the electronic device (1000). A method for the electronic device (1000) to construct a first audio feature dictionary is not limited to that illustrated in FIG. 14, and in one or more embodiments, additional steps not illustrated in FIG. 14 may be included, or some steps may be omitted.

[0182] In step S1410, an electronic device (1000) according to one embodiment of the present disclosure can acquire a plurality of input images.

[0183] According to one embodiment of the present disclosure, a plurality of input images may be consecutive image frames, but are not limited thereto.

[0184] According to one embodiment of the present disclosure, the electronic device (1000) may automatically select a plurality of input images to generate a video, or receive a selection of a plurality of input images from a user.

[0185] According to one embodiment of the present disclosure, a plurality of input images may be stored in the memory of an electronic device (1000) or may be stored in an external device (e.g., a cloud server).

[0186] Since step S1410 corresponds to step S410 of FIG. 4, a redundant explanation will be omitted.

[0187] In step S1420, an electronic device (1000) according to one embodiment of the present disclosure can input a plurality of input images into an image encoding model (620) to obtain a plurality of image feature vectors.

[0188] According to one embodiment of the present disclosure, when a plurality of input images are applied to an image encoding model (620), the image encoding model (620) can convert each input image into a latent space. For example, the image encoding model (620) can convert each input image into a vector of the same dimension as the audio feature vectors stored in the first audio feature dictionary (100). The image encoding model (620) can extract features from each input image and generate an image feature vector corresponding to each input image using the extracted features.

[0189] According to one embodiment of the present disclosure, when the categories of a plurality of input images are similar, a plurality of image feature vectors may be located close together in the latent space. On the other hand, when the categories of a plurality of input images are different, a plurality of image feature vectors may be located far apart in the latent space.

[0190] In step S1430, an electronic device (1000) according to one embodiment of the present disclosure can detect a plurality of image feature vectors having a similarity to each of the plurality of image feature vectors greater than or equal to a threshold value.

[0191] According to one embodiment of the present disclosure, an electronic device (1000) can calculate the similarity between each image feature vector and each audio feature vector stored in a first audio feature dictionary (100). For example, the electronic device (1000) can calculate the similarity by measuring the distance between each image feature vector and each audio feature vector in a latent space. According to one embodiment of the present disclosure, the electronic device (1000) can measure the distance between each image feature vector and each audio feature vector using similarity measurement metrics such as cosine similarity, Euclidean distance, and inner product.

[0192] According to one embodiment of the present disclosure, an electronic device (1000) determines audio feature vectors having maximum similarity with each image feature vector and can determine whether the maximum similarity is greater than or equal to a threshold value. If the maximum similarity is greater than or equal to a threshold value, the electronic device (1000) can select the audio feature vectors having maximum similarity as vectors for generating sound effects.

[0193] In step S1440, an electronic device (1000) according to one embodiment of the present disclosure can determine whether the categories of a plurality of audio feature vectors are the same.

[0194] According to one embodiment of the present disclosure, an electronic device (1000) can determine whether the categories of a plurality of audio feature vectors are the same based on meta-information of each audio feature vector stored in a first audio feature dictionary (100). The meta-information may include category information.

[0195] According to one embodiment of the present disclosure, the electronic device (1000) may determine whether the categories are the same by using the similarity between a plurality of audio feature vectors. For example, if the similarity between a plurality of audio feature vectors is greater than or equal to a threshold value, it may be determined that the categories are the same.

[0196] For example, if multiple input images are photographs of a rainy park, the audio feature vectors corresponding to the image feature vectors of the multiple input images are located close to each other in the latent space and may have the same category (e.g., rain sound).

[0197] In step S1450, an electronic device (1000) according to one embodiment of the present disclosure can generate sound effects corresponding to each input image by applying each of the plurality of audio feature vectors to a generative AI model when the categories of the plurality of audio feature vectors are different (No. of S1440).

[0198] For example, if multiple input photos are a beach photo, a forest photo, and a park photo, the audio feature vectors corresponding to the image feature vectors of the multiple input images may be of different categories. Accordingly, the electronic device (1000) can generate sound effects separately for each of the beach photo, forest photo, and park photo.

[0199] In step S1460, an electronic device (1000) according to one embodiment of the present disclosure may select a representative audio feature vector among a plurality of audio feature vectors when the categories of a plurality of audio feature vectors are the same (Yes of S1440).

[0200] According to one embodiment of the present disclosure, an electronic device (1000) can select a representative audio feature vector by comparing similarity (distance) between a plurality of audio feature vectors. For example, if a first audio feature vector is located between a second audio feature vector and a third audio feature vector, the electronic device (1000) can select the first audio feature vector as the representative audio feature vector.

[0201] Additionally, when the electronic device (1000) has a plurality of input images that are consecutive frame images, the electronic device (1000) may select an audio feature vector corresponding to the first frame image as a representative audio feature vector, select an audio feature vector corresponding to an intermediate frame image as a representative audio feature vector, and select an audio feature vector corresponding to the last frame image as a representative audio feature vector.

[0202] In step S1470, an electronic device (1000) according to one embodiment of the present disclosure can generate sound effects corresponding to all of a plurality of input images by applying a representative audio feature vector to a generative AI model.

[0203] For example, an electronic device (1000) can input a representative audio feature vector into a latent diffusion model (650). At this time, the latent diffusion model (650) can output a 3D vector of generated audio by gradually removing noise from random noise sampled from a Gaussian distribution (normal distribution). When the VAE decoder (660) receives the 3D vector, it can convert the 3D vector into a Mel spectrogram. When the vocoder (670) receives the Mel spectrogram, it can convert the Mel spectrogram into an audio waveform to generate a sound effect.

[0204] According to one embodiment of the present disclosure, the electronic device (1000) can adjust the playback length of a sound effect based on the number of input images. For example, as the number of input images increases, the playback length of the sound effect may become longer. If the length of the sound effect for one input image is 2.5 seconds and the number of input images is 3, the playback length of the sound effect may be 7.5 seconds. That is, one sound effect may be output while three input images are being played. The sound effects corresponding to the input images may have the same playback length, but may also be different.

[0205] According to one embodiment of the present disclosure, the electronic device (1000) can adjust the playback length by adjusting the size of the random noise of the latent diffusion model (650) based on the number of input images. As the number of input images increases, the size of the random noise may increase.

[0206] According to one embodiment of the present disclosure, when photos of the same category are consecutive, the computational load of a generative AI model can be reduced by generating a single sound effect overall. Additionally, the electronic device (1000) can generate a sound effect that flows naturally for photos of the same category.

[0207] Below, with reference to FIG. 15, we will examine in more detail the operation of an electronic device (1000) generating sound effects for multiple input images of the same category at once.

[0208] FIG. 15 is a diagram illustrating the operation of an electronic device (1000) according to one embodiment of the present disclosure that generates sound effects for a plurality of input images of the same category.

[0209] Referring to FIG. 15, the electronic device (1000) may acquire a first photograph (1501), a second photograph (1502), and a third photograph (1503) as input images for generating sound effects. The first photograph (1501), the second photograph (1502), and the third photograph (1503) may all be photographs taken at a beach.

[0210] The electronic device (1000) can input the first photo (1501), the second photo (1502), and the third photo (1503) into an image encoding model (620) to obtain the first image feature vector (1510), the second image feature vector (1520), and the third image feature vector (153). The electronic device (1000) can calculate the similarity by comparing each of the first image feature vector (1510), the second image feature vector (1520), and the third image feature vector (153) with the audio feature vectors stored in the first audio feature dictionary (100). The electronic device (1000) can identify a first audio feature vector (1521) having maximum similarity to a first image feature vector (1510), a second audio feature vector (1522) having maximum similarity to a second image feature vector (1520), and a third audio feature vector (1523) having maximum similarity to a third image feature vector (1530).

[0211] The electronic device (1000) can determine whether the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) belong to the same category. For example, the electronic device (1000) can determine whether the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) belong to the same category by using the meta information of the first audio feature vector (1521), the meta information of the second audio feature vector (1522), and the meta information of the third audio feature vector (1523) stored in the first audio feature dictionary (100). Additionally, the electronic device (1000) may determine that the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) belong to the same category if the similarity between them is greater than or equal to a threshold value (S1500).

[0212] The electronic device (1000) can generate sound effects for each audio feature vector when the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) do not belong to the same category (S1550). In FIG. 15, since the first photo (1501), the second photo (1502), and the third photo (1503) are all taken at a beach, the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) may be related to the sound of waves. Therefore, FIG. 15 will be described by giving an example of the case where the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) belong to the same category.

[0213] When the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) belong to the same category, the electronic device (1000) can select a representative audio feature vector among the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523). In FIG. 15, a representative audio feature vector can be selected in which the similarity between the first audio feature vector (1521), the second audio feature vector (1522), and the third audio feature vector (1523) is maximized. For example, the electronic device (1000) can obtain A, B, and C as follows, and if C is the largest, select the third audio feature vector (1523) as the representative audio feature vector.

[0214] A = (Similarity between the first audio feature vector (1521) and the second audio feature vector (1522) + Similarity between the first audio feature vector (1521) and the third audio feature vector (1523)) / 2

[0215] B = (Similarity between the second audio feature vector (1522) and the first audio feature vector (1521) + Similarity between the second audio feature vector (1522) and the third audio feature vector (1523)) / 2

[0216] C = (Similarity between the third audio feature vector (1523) and the first audio feature vector (1521) + Similarity between the third audio feature vector (1523) and the second audio feature vector (1522)) / 2

[0217] The electronic device (1000) can copy the representative audio feature vector (1523). For example, the electronic device (1000) can copy the representative audio feature vector (1523) to create three representative audio feature vectors (1523). In FIG. 15, the case where the number is three is described as an example, but it is not limited thereto.

[0218] According to one embodiment of the present disclosure, an electronic device (1000) can generate three candidate sound effects (1531, 1532, 1533) by applying three representative audio feature vectors (1523) to a generative AI model. For example, when the electronic device (1000) inputs three representative audio feature vectors (1523) into a latent diffusion model (650), the latent diffusion model (650) can generate three audio (sound effects) by progressively denoising noise from a plurality of random noises (651) using a noise prediction network (e.g., Denoising U-net) (652). The three audio (sound effects) generated through the latent diffusion model (650) can be represented as three-dimensional vectors in latent space. Since different random noise (651) is used for three audio feature vectors (640), the three-dimensional vectors, which are the output values ​​of the latent diffusion model (650), may differ even if the three audio feature vectors (640) are identical. The VAE decoder (660) can convert the three three-dimensional vectors output from the latent diffusion model (650) into three Mel spectrograms. And the vocoder (670) can generate three candidate sound effects (1531, 1532, 1533) for the input image (620) by converting the three Mel spectrograms output from the VAE decoder (660) into actual audio waveforms.

[0219] Meanwhile, according to one embodiment of the present disclosure, the electronic device (1000) can adjust the length of three candidate sound effects (1531, 1532, 1533) based on the number of input images belonging to the same category (S1560). For example, the latent diffusion model (650) can adjust the length of the three candidate sound effects (1531, 1532, 1533) by adjusting the size (WXH) of the random noise (651). If it is set to generate a sound effect of 2.5 seconds for one input image, the latent diffusion model (650) can adjust the size of the random noise (651) so that candidate sound effects (1531, 1532, 1533) of 7.5 seconds (2.5X3) are generated.

[0220] The electronic device (1000) can obtain three audio feature vectors (1541, 1542, 1543) by applying each of the three candidate sound effects (1531, 1532, 1533) to an audio encoding model (320). The electronic device (1000) can calculate similarity by comparing the three audio feature vectors (1541, 1542, 1543) with one of the first image feature vector (1510), the second image feature vector (1520), and the third image feature vector (1530) (S1570). In FIG. 15, since the third audio feature vector (1523) is selected as the representative audio feature vector, the case in which the three audio feature vectors (1541, 1542, 1543) are compared with the third image feature vector (1530) will be explained as an example.

[0221] Referring to FIG. 15, the similarity between the fourth audio feature vector (1541) and the third image feature vector (1530) may be 0.7, the similarity between the fifth audio feature vector (1542) and the third image feature vector (1530) may be 0.3, and the similarity between the sixth audio feature vector (1543) and the third image feature vector (1530) may be 0.5. In this case, since the similarity between the fourth audio feature vector (1541) and the third image feature vector (1530) is the greatest, the electronic device (1000) may determine the first candidate sound effect (1531) corresponding to the fourth audio feature vector (1541) as the final sound effect corresponding to the first photo (1501), the second photo (1502), and the third photo (1503).

[0222] According to one embodiment of the present disclosure, the electronic device (1000) can generate sound effects for a first photo (1501), a second photo (1502), and a third photo (1503) of the same category all at once, thereby reducing the computational load of the generative AI model. Additionally, sound effects for the first photo (1501), the second photo (1502), and the third photo (1503) can be played naturally. For example, when the first photo (1501), the second photo (1502), and the third photo (1503) are played, a wave sound effect can be naturally output.

[0223] Below, we will examine in detail the operation of the electronic device (1000) outputting a sound effect with reference to FIG. 16 and FIG. 17.

[0224] FIG. 16 is a flowchart for explaining a method of playing a video according to one embodiment of the present disclosure.

[0225] Referring to FIG. 16, a method for an electronic device (1000) to play a video may include steps S1610 to S1620. In one embodiment of the present disclosure, steps S1610 to S1620 may be executed by at least one processor included in the electronic device (1000). A method for an electronic device (1000) to play a video is not limited to that illustrated in FIG. 16, and in one or more embodiments, additional steps not illustrated in FIG. 16 may be included, or some steps may be omitted.

[0226] In step S1610, an electronic device (1000) according to one embodiment of the present disclosure may receive a playback request for a video including an input image.

[0227] For example, the electronic device (1000) may provide a list of videos to the user. The list of videos may display thumbnail images of multiple videos created by combining multiple photos and adding background music. The electronic device (1000) may receive user input requesting playback of a specific video from the list of videos. For example, the electronic device (1000) may receive a request from the user to play a first video from the list of videos that includes an input image with a sound effect generated.

[0228] In step S1620, the electronic device (1000) according to one embodiment of the present disclosure may output a sound effect at the time when an input image is played while playing a video in response to a playback request.

[0229] According to one embodiment of the present disclosure, an electronic device (1000) can play images along with background music. In this case, if an input image in which a sound effect is generated is played, the electronic device (1000) can output a sound effect at the time the input image is played. For example, if the input image is a picture of a rainy park, the electronic device (1000) can output a rain sound as a sound effect at the time the picture of the rainy park is played.

[0230] According to one embodiment of the present disclosure, when a sound effect is output, the electronic device (1000) may reduce the volume of the background music (BGM) or turn off the sound of the background music completely.

[0231] FIG. 17 is a drawing for explaining the operation of an electronic device (1000) according to one embodiment of the present disclosure playing a video with sound effects.

[0232] According to one embodiment of the present disclosure, an electronic device (1000) can generate a video using a first photograph (1701), a second photograph (1702), a third photograph (1703), a fourth photograph (1704), a fifth photograph (1705), a sixth photograph (1706), a seventh photograph (1707) and background music (BGM). At this time, the electronic device (1000) can generate sound effects corresponding to each of the first photograph (1701), the second photograph (1702), the third photograph (1703), the fourth photograph (1704), the fifth photograph (1705), and the sixth photograph (1706). For example, the electronic device (1000) can generate the sound of an airplane taking off as a sound effect since the first photo (1701) is an airplane photo, generate the sound of wind and insects as a sound effect since the second photo (1702) is a park photo, generate the sound of water falling strongly as a sound effect since the third photo (1703) is a waterfall photo, generate the sound of waves crashing as a sound effect since the fourth photo (1704) and the fifth photo (1705) are seaside photos, and generate the sound of water flowing gently as a sound effect since the sixth photo (1706) is a river photo. However, the electronic device (1000) may not generate a sound effect for the seventh photo (1707) because an audio feature vector similar to the image feature vector of the seventh photo (1707) is not stored in the first audio feature dictionary (100). Meanwhile, since the audio feature vector corresponding to the fourth photo (1704) and the audio feature vector corresponding to the fifth photo (1705) belong to the same category, the electronic device (1000) can generate sound effects for the fourth photo (1704) and the fifth photo (1705) at once.

[0233] Referring to FIG. 17, when a user requests the playback of a video, the electronic device (1000) can sequentially play the first photo (1701), the second photo (1702), the third photo (1703), the fourth photo (1704), the fifth photo (1705), the sixth photo (1706), and the seventh photo (1707) along with background music (BGM). At this time, the electronic device (1000) can output the sound of an airplane taking off as a sound effect while the first photo (1701) is displayed (T0~T1), output the sound of wind and grasshoppers as a sound effect while the second photo (1702) is displayed (T1~T2), output the sound of water falling strongly as a sound effect while the third photo (1703) is displayed (T2~T3), output the sound of waves crashing as a sound effect while the fourth photo (1704) and the fifth photo (1705) are displayed (T3~T5), output the sound of water trickling as a sound effect while the sixth photo (1706) is displayed (T5~T6), and output only background music while the seventh photo (1707) is displayed (T6~T7).

[0234] According to one embodiment of the present disclosure, by generating and outputting natural sound effects suitable for each photo, a more vivid video can be provided.

[0235] FIG. 18 is a block diagram for explaining the function of an electronic device (1000) according to one embodiment of the present disclosure.

[0236] FIG. 19 is a detailed block diagram for explaining the function of an electronic device according to one embodiment of the present disclosure.

[0237] As illustrated in FIG. 18, an electronic device (1000) according to one embodiment of the present disclosure may include a processor (1300) and a memory (1700). However, not all components illustrated in FIG. 18 are essential components. The electronic device (1000) may be implemented with more components than those illustrated in FIG. 18, or with fewer components. For example, as illustrated in FIG. 19, an electronic device (1000) according to one embodiment of the present disclosure may include an output unit (1100), a sensor unit (1200), a processor (1300), a communication interface (1400), an A / V input unit (1500), a user input unit (1600), and a memory (1700).

[0238] The above components will be examined in turn below.

[0239] The output unit (1100) is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit (1111), an audio output unit (1112), a vibration motor (1113), etc.

[0240] When the display unit (1111) and the touchpad form a layered structure and are configured as a touchscreen, the display unit (1111) can be used as an input interface in addition to an output interface. The display unit (1111) may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. Additionally, depending on the implementation form of the electronic device (1000), two or more display units (1111) may be included.

[0241] The sound output unit (1112) outputs an audio signal received from the communication interface (1400) or stored in the memory (1700). Additionally, the sound output unit (1112) outputs an audio signal related to a function performed by the electronic device (1000) (e.g., a call signal reception sound, a message reception sound, a notification sound). The sound output unit (1112) may include a speaker, a buzzer, etc.

[0242] The vibration motor (1113) can output a vibration signal. For example, the vibration motor (1113) can output a vibration signal corresponding to the output of audio data or video data (e.g., a call signal reception sound, a message reception sound, etc.). Additionally, the vibration motor (1113) can output a vibration signal when a touch is input to the touchscreen.

[0243] The sensor unit (1200) may include at least one of a magnetic sensor (1211), an acceleration sensor (1212), a tilt sensor (1213), an infrared sensor (1214), a gyroscope sensor (1215), a position sensor (e.g., GPS) (1216), a temperature and humidity sensor (1217), a proximity sensor (1218), and a light sensor (1219), but is not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description is omitted.

[0244] The processor (1300) typically controls the overall operation of the electronic device (1000). For example, the processor (1300) can control the output unit (1100), sensor unit (1200), communication interface (1400), A / V input unit (1500), user input unit (1600), memory (1700), etc., by executing programs stored in memory (1700).

[0245] The processor (1300) may be composed of one or more processors. The one or more processors included in the processor (1300) may be circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). The one or more processors included in the processor (1300) may be general-purpose processors such as a CPU (Central Processing Unit), MPU (Micro Processor Unit), AP (Application Processor), or DSP (Digital Signal Processor); graphics-dedicated processors such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit); artificial intelligence-dedicated processors such as an NPU (Neural Processing Unit); or communication-dedicated processors such as a CP (Communication Processor). If the one or more processors included in the processor (1300) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The processor (1300) may be implemented as a single core processor or as a multicore processor.

[0246] The processor (1300) can write data to memory (1700) or read data stored in memory (1700), and in particular, can process data according to a predefined operation rule or artificial intelligence model by executing a program or at least one instruction stored in memory (1700).

[0247] The communication interface (1400) may include one or more components that enable communication between an electronic device (1000) and an IoT device (3000) or between an electronic device (1000) and a server (2000). For example, the communication interface (1400) may include a short-range communication unit (1411), a mobile communication unit (1412), and a broadcast receiving unit (1413).

[0248] The short-range wireless communication unit (221) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra-wideband (UWB) communication unit, an Ant+ communication unit, etc.

[0249] The mobile communication unit (1412) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various forms of data such as voice call signals, video call call signals, or text / multimedia message transmission and reception.

[0250] The broadcast receiver (1413) receives broadcast signals and / or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Depending on the implementation example, the electronic device (1000) may not include the broadcast receiver (1413).

[0251] The A / V (Audio / Video) input unit (1500) is for inputting audio signals or video signals and may include a camera (1511), a microphone (1512), etc. The camera (1511) can obtain image frames, such as still images or video, through an image sensor in a video call mode or a shooting mode. Images captured through the image sensor may be processed through a processor (1300) or a separate image processing unit (not shown). Image frames processed by the camera (1511) may be stored in a memory (1700) or transmitted externally through a communication interface (1400). According to one embodiment of the present disclosure, the camera (1511) may include at least one of a telephoto camera, a wide-angle camera, and a standard camera, but is not limited thereto.

[0252] The microphone (1512) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1512) can receive an acoustic signal from an external device or a speaker. The microphone (1512) can use various noise removal algorithms to remove noise generated during the process of receiving the external acoustic signal.

[0253] The user input unit (1600) refers to a means for a user to input data for controlling an electronic device (1000). For example, the user input unit (1600) may include a key pad, a dome switch, a touch pad (contact capacitive method, pressure resistive method, infrared detection method, surface ultrasonic conduction method, integral tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, etc., but is not limited thereto.

[0254] The memory (1700) may store a program for processing and controlling the processor (1300), and may also store input / output data (e.g., voice data, photo images, memo data, user's biometric information, etc.).

[0255] The memory (1700) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0256] The memory (1700) may not exist separately but may be configured to be included in the processor (1300). The memory (1700) may be composed of volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. The memory (1700) may store a program or at least one instruction for performing operations according to one embodiment of the present disclosure. The memory (1700) may provide stored data to the processor (1300) upon the request of the processor (1300).

[0257] According to one embodiment of the present disclosure, the memory (1700) may store a first audio feature dictionary (100) and a second audio feature dictionary (200). The first audio feature dictionary (100) may include audio feature vectors of a category for which sound effects are desired to be generated. The second audio feature dictionary (200) may include audio feature vectors of a category for which sound effects are not desired to be generated.

[0258] A method for generating a sound effect for an image using an electronic device (1000) according to one embodiment of the present disclosure may include: acquiring an input image; applying the input image to an image encoding model to acquire an image feature vector; selecting an audio feature vector from a first audio feature dictionary (100) in which audio feature vectors of a predetermined category are stored, wherein the similarity to the image feature vector is greater than or equal to a threshold value; and applying the selected audio feature vector to a generative AI model to generate a sound effect corresponding to the input image.

[0259] The step of selecting an audio feature vector having a similarity to an image feature vector greater than or equal to a threshold value according to one embodiment of the present disclosure may include: a step of obtaining similarities between each audio feature vector and an image feature vector stored in a first audio feature dictionary (100); and a step of selecting an audio feature vector having the maximum similarity among the obtained similarities when the maximum similarity is greater than or equal to a threshold value.

[0260] A generative AI model according to one embodiment of the present disclosure may include a latent diffusion-based audio generation model.

[0261] A method according to one embodiment of the present disclosure may include the steps of: obtaining reference sound effects included in a predetermined category; applying the reference sound effects to an audio encoding model to obtain audio feature vectors; and storing the audio feature vectors in a first audio feature dictionary (100).

[0262] An audio encoding model and an image encoding model according to one embodiment of the present disclosure can be trained together using audio and image pairs as training data.

[0263] A step of generating a sound effect according to one embodiment of the present disclosure may include: a step of generating a predetermined number of audio feature vectors by copying a selected audio feature vector; a step of generating a plurality of candidate sound effects corresponding to the audio feature vectors by applying the predetermined number of audio feature vectors to a generative AI model; a step of obtaining a plurality of audio feature vectors corresponding to the plurality of candidate sound effects by applying each of the plurality of candidate sound effects to an audio encoding model; and a step of determining a candidate sound effect corresponding to the audio feature vector with the greatest similarity to the image feature vector among the plurality of candidate sound effects as the sound effect corresponding to the input image based on the result of comparing the plurality of audio feature vectors and the image feature vectors.

[0264] A step of acquiring an image feature vector according to one embodiment of the present disclosure may include: a step of acquiring a global feature vector by performing a pooling operation on the entire feature map corresponding to an input image; and a step of acquiring a plurality of local feature vectors by dividing the feature map corresponding to the input image into window units of a predetermined size and performing a pooling operation.

[0265] The step of selecting an audio feature vector having a similarity to an image feature vector greater than or equal to a threshold value according to one embodiment of the present disclosure may include: a step of selecting a first audio feature vector having a first maximum similarity when the first maximum similarity between a global feature vector and audio feature vectors stored in a first audio feature dictionary (100) is greater than or equal to a threshold value; and a step of selecting a second audio feature vector having a second maximum similarity when the second maximum similarity between a plurality of local feature vectors and audio feature vectors stored in a first audio feature dictionary (100) is greater than or equal to a threshold value.

[0266] A step of generating a sound effect corresponding to an input image according to one embodiment of the present disclosure may include: a step of generating a first sound effect and a second sound effect by applying a first audio feature vector and a second audio feature vector to a generative AI model; and a step of generating a sound effect corresponding to an input image by synthesizing the first sound effect and the second sound effect.

[0267] The step of generating a sound effect corresponding to an input image according to one embodiment of the present disclosure may include: applying a first audio feature vector and a second audio feature vector to a feature fusion model to obtain a fused audio feature vector; and applying the fused audio feature vector to a generative AI model to generate a sound effect corresponding to the input image by synthesizing a first sound effect corresponding to the first audio feature vector and a second sound effect corresponding to the second audio feature vector.

[0268] A feature fusion model according to one embodiment of the present disclosure may be an artificial intelligence model trained to fuse a plurality of feature vectors into a single vector while preserving each feature of a plurality of feature vectors.

[0269] A method according to one embodiment of the present disclosure may include the step of generating a second audio feature dictionary (200) in which audio feature vectors of a category not desired for sound effect generation are stored.

[0270] A step of generating a sound effect according to one embodiment of the present disclosure may include: determining similarities between all audio feature vectors and image feature vectors stored in a first audio feature dictionary (100) and a second audio feature dictionary (200); and, if an audio feature vector having maximum similarity to an image feature vector is included in the first audio feature dictionary (100), applying the audio feature vector having maximum similarity to an image feature vector to a generative AI model to generate a sound effect corresponding to an input image.

[0271] A method according to one embodiment of the present disclosure may include: acquiring a plurality of input images; inputting a plurality of input images into an image encoding model to acquire a plurality of image feature vectors; detecting a plurality of audio feature vectors from a first audio feature dictionary (100) such that the similarity to each of the plurality of image feature vectors is greater than or equal to a threshold value; selecting a representative audio feature vector among the detected plurality of audio feature vectors when the categories of the detected plurality of audio feature vectors are the same; and applying the representative audio feature vector to a generative AI model to generate sound effects corresponding to all of the plurality of input images.

[0272] The step of generating a sound effect corresponding to a plurality of input images according to one embodiment of the present disclosure may include a step of adjusting the playback length of the sound effect based on the number of input images.

[0273] A method according to one embodiment of the present disclosure may include: receiving a playback request for a video including an input image; and outputting a sound effect at the time when the input image is played while playing the video in accordance with the playback request.

[0274] An electronic device (1000) according to one embodiment of the present disclosure may include: a memory (1700) for storing one or more instructions; and at least one processor (1300) operably coupled to the memory (1700) and comprising a processing circuit. The electronic device (1000) may acquire an input image by having at least one processor execute the one or more instructions alone or in cooperation. The electronic device (1000) may acquire an image feature vector by applying the input image to an image encoding model (620) by having at least one processor execute the one or more instructions alone or in cooperation. The electronic device (1000) may select an audio feature vector whose similarity to an image feature vector is greater than or equal to a threshold value from a first audio feature dictionary (100) in which audio feature vectors of a predetermined category are stored, by having at least one processor execute the one or more instructions alone or in cooperation. The electronic device (1000) can generate sound effects corresponding to an input image by applying a selected audio feature vector to a generative AI model by having at least one processor execute one or more instructions alone or in cooperation.

[0275] An electronic device (1000) according to one embodiment of the present disclosure can obtain similarities between each audio feature vector and an image feature vector stored in a first audio feature dictionary (100) by having at least one processor (1300) execute the one or more instructions alone or in cooperation. The electronic device (1000) can select an audio feature vector having the maximum similarity among the similarities obtained when the maximum similarity is greater than or equal to a threshold value by having at least one processor (1300) execute the one or more instructions alone or in cooperation.

[0276] A generative AI model according to one embodiment of the present disclosure may include a latent diffusion-based audio generation model.

[0277] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0278] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

1. A method in which an electronic device (1000) generates a sound effect for an image, Step of acquiring an input image; A step of obtaining an image feature vector by applying the above input image to an image encoding model; A step of selecting an audio feature vector whose similarity to the image feature vector is greater than or equal to a threshold value from a first audio feature dictionary (100) in which audio feature vectors of a predetermined category are stored; and A method comprising the step of applying the selected audio feature vector to a generative AI model to generate a sound effect corresponding to the input image.

2. In claim 1, the step of selecting an audio feature vector having a similarity to the image feature vector greater than or equal to a threshold value is: A step of obtaining similarities between each audio feature vector stored in the first audio feature dictionary (100) and the image feature vector; and A method comprising the step of selecting an audio feature vector having the maximum similarity among the similarities obtained above, wherein the maximum similarity is greater than or equal to the threshold value.

3. In paragraph 1 or 2, the generative AI model is, A method comprising a latent diffusion-based audio generation model.

4. In any one of paragraphs 1 to 3, the method is, A step of obtaining reference sound effects included in the above-mentioned predetermined category; A step of obtaining the audio feature vectors by applying the above reference sound effects to an audio encoding model; and The method further includes the step of storing the above audio feature vectors in the first audio feature dictionary (100), A method in which the above audio encoding model and the above image encoding model are trained together using audio and image pairs as training data.

5. In any one of paragraphs 1 to 4, the step of generating the sound effect is, A step of generating a predetermined number of audio feature vectors by copying the selected audio feature vectors above; A step of applying the above-mentioned predetermined number of audio feature vectors to the generative AI model to generate a plurality of candidate sound effects; A step of applying each of the above-mentioned plurality of candidate sound effects to an audio encoding model to obtain a plurality of audio feature vectors corresponding to the above-mentioned plurality of candidate sound effects; and A method comprising the step of determining, based on the result of comparing the plurality of audio feature vectors and the image feature vectors, the candidate sound effect corresponding to the audio feature vector having the greatest similarity to the image feature vector among the plurality of candidate sound effects as the sound effect corresponding to the input image.

6. In any one of claims 1 to 5, the step of acquiring the image feature vector is, A step of obtaining a global feature vector by performing a pooling operation on the entire feature map corresponding to the input image; and A method comprising the step of dividing a feature map corresponding to the input image into window units of a predetermined size and performing a pooling operation to obtain a plurality of local feature vectors.

7. In claim 6, the step of selecting an audio feature vector having a similarity to the image feature vector greater than or equal to a threshold value is: A step of selecting a first audio feature vector having the first maximum similarity when the first maximum similarity between the global feature vector and the audio feature vectors stored in the first audio feature dictionary (100) is greater than or equal to the threshold value; and The method includes the step of selecting a second audio feature vector having the second maximum similarity when the second maximum similarity between the plurality of local feature vectors and the audio feature vectors stored in the first audio feature dictionary (100) is greater than or equal to the threshold value. The step of generating a sound effect corresponding to the above input image is: A step of generating a first sound effect and a second sound effect by applying the first audio feature vector and the second audio feature vector to the generative AI model; and A method comprising the step of synthesizing the first sound effect and the second sound effect to generate a sound effect corresponding to the input image.

8. In claim 7, the step of generating a sound effect corresponding to the input image is, A step of applying the first audio feature vector and the second audio feature vector to a feature fusion model (1110) to obtain a fused audio feature vector; and The method includes the step of applying the fused audio feature vector to the generative AI model to generate a sound effect corresponding to the input image by synthesizing a first sound effect corresponding to the first audio feature vector and a second sound effect corresponding to the second audio feature vector, and the feature fusion model (1110) A method, which is an artificial intelligence model trained to fuse multiple feature vectors into a single vector while preserving each feature of the multiple feature vectors.

9. In any one of paragraphs 1 to 8, the method is, A method further comprising the step of generating a second audio feature dictionary (200) in which audio feature vectors of categories not desired for sound effect generation are stored.

10. In claim 9, the step of generating the sound effect is, A step of determining similarities between all audio feature vectors stored in the first audio feature dictionary (100) and the second audio feature dictionary (200) and the image feature vectors; and A method comprising the step of, when an audio feature vector having maximum similarity to the image feature vector is included in the first audio feature dictionary (100), applying the audio feature vector having maximum similarity to the image feature vector to the generative AI model to generate a sound effect corresponding to the input image.

11. In any one of claims 1 to 10, the method is, A step of acquiring multiple input images; A step of acquiring a plurality of image feature vectors by inputting the plurality of input images into the image encoding model; A step of detecting a plurality of audio feature vectors from the first audio feature dictionary (100) such that the similarity with each of the plurality of image feature vectors is greater than or equal to a threshold value; When the categories of the detected plurality of audio feature vectors are the same, a step of selecting a representative audio feature vector among the detected plurality of audio feature vectors; and A method comprising the step of applying the representative audio feature vector to the generative AI model to generate a sound effect corresponding to all of the plurality of input images.

12. In claim 11, the step of generating sound effects corresponding to the plurality of input images is, A method comprising the step of adjusting the playback length of the sound effect based on the number of input images above.

13. In any one of paragraphs 1 to 12, the method is, A step of receiving a playback request for a video including the above input image; and A method further comprising the step of outputting the sound effect at the time when the input image is played while the video is played in accordance with the above playback request.

14. Memory (1700) for storing one or more instructions; and It includes at least one processor (1300) that is operably coupled to the memory (1700) and includes a processing circuit, and By having at least one processor (1300) execute one or more instructions either alone or in cooperation, the electronic device (1000) Acquire the input image, and The above input image is applied to an image encoding model to obtain an image feature vector, and From a first audio feature dictionary (100) in which audio feature vectors of a predetermined category are stored, an audio feature vector having a similarity to the image feature vector greater than or equal to a threshold value is selected, and An electronic device that generates a sound effect corresponding to the input image by applying the selected audio feature vector to a generative AI model.

15. A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 1 through 13 on a computer.

Images