Method and apparatus for generating music video, device, and storage medium

Abstract

Embodiments of the present disclosure relate to a method and apparatus for generating a music video, a device, and a storage medium. The method proposed herein comprises: acquiring a media generation request, the media generation request indicating reference text content and reference image content; and generating a music video on the basis of the media generation request, audio content of the music video being generated on the basis of selected reference text content, and at least some images of the music video being generated on the basis of the reference image content. In this way, the embodiments of the present disclosure can improve the efficiency of generating a music video.

Description

Methods, apparatus, devices and storage media for generating music videos

[0001] This application claims priority to Chinese Patent Application No. 202411997188.1, filed on December 31, 2024, entitled "Method, Apparatus, Device and Storage Medium for Generating Music Videos", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for generating music videos. Background Technology

[0003] With the increasing maturity of internet technology, more and more users are utilizing online platforms to assist in generating video content. For example, online platforms can generate video content based on user-provided images. However, when users generate video content using existing online platforms, they need to select preset video templates for video production and cannot generate video content based on their own configurations. Furthermore, based on current technology, users can only select existing music content as background music for their videos and cannot create new music content during the video creation process. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for generating a music video is provided. The method includes: obtaining a media generation request, the media generation request indicating reference text content and reference image content; and generating a music video based on the media generation request, wherein the audio content of the music video is generated based on the selected reference text content, and at least a portion of the images in the music video are generated based on the reference image content.

[0005] In a second aspect of this disclosure, an apparatus for generating a music video is provided. The apparatus includes: an acquisition module configured to acquire a media generation request, the media generation request indicating reference text content and reference image content; and a generation module configured to generate a music video based on the media generation request, wherein the audio content of the music video is generated based on the selected reference text content, and at least a portion of the images in the music video are generated based on the reference image content.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure may be implemented;

[0012] Figure 2 shows a flowchart of an example process for generating a music video according to some embodiments of the present disclosure;

[0013] Figures 3A to 3G illustrate example interfaces according to some embodiments of the present disclosure;

[0014] Figure 4 illustrates a flowchart of an example process for generating a music video according to some embodiments of the present disclosure;

[0015] Figure 5 shows a schematic structural block diagram of an example apparatus for generating music videos according to some embodiments of the present disclosure; and

[0016] Figure 6 shows a block diagram of an electronic device capable of implementing several embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0021] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0022] As mentioned above, with the increasing maturity of internet technology, more and more users are using online platforms to assist in generating video content. For example, online platforms can generate video content based on images provided by users. However, when users generate video content using existing online platforms, they need to select preset video templates for video production and cannot generate video content based on their own configurations. In addition, based on current technology, users can only select existing music content as background music for video content and cannot create new music content during the video creation process.

[0023] Embodiments of this disclosure propose a scheme for generating music videos. According to this scheme, a media generation request can be obtained, the media generation request indicating reference text content and reference image content; and a music video can be generated based on the media generation request, wherein the audio content of the music video is generated based on the selected reference text content, and at least a portion of the images of the music video are generated based on the reference image content.

[0024] In this manner, embodiments of the present disclosure are able to obtain media generation requests. Furthermore, embodiments of the present disclosure are able to generate audio content for a music video based on the reference text content indicated by the media generation request, and generate at least a portion of the images for the music video based on the reference image content indicated by the media generation request. Thus, embodiments of the present disclosure can generate music videos based on user configurations, thereby meeting the user's needs for creating music and video content and improving the efficiency of user-generated media content.

[0025] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in Figure 1, the example environment 100 may include an electronic device 110.

[0028] In this example environment 100, electronic device 110 may run an application 120 that supports user interface interaction. Application 120 may be any suitable type of application for user interface interaction, and examples may include, but are not limited to, calling applications, conferencing applications, social applications, or other suitable applications. User 140 may interact with application 120 via electronic device 110 and / or its attached devices.

[0029] In environment 100 of Figure 1, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting interface interaction.

[0030] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0031] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 that support virtual scenarios in electronic devices 110.

[0032] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0035] Example of generating music video

[0036] Figure 2 shows a flowchart of an example process 200 for generating a music video according to some embodiments of the present disclosure. Process 200 can be implemented at electronic device 110 and / or server 130. Process 200 will now be described with reference to Figure 1.

[0037] In some embodiments, electronic device 110 and / or server 130 may receive a media generation request. The media generation request may specify reference text content and reference image content.

[0038] As shown in Figure 2, in box 205, the electronic device 110 can receive a text item selected by the user. The text item can, for example, indicate the lyrics and content style of the music video to be generated. As an example, the content style of the music video can include the style of the audio content (e.g., hip-hop, jazz, R&B, etc.). As an example, the content style of the music video can include the style of the image content (e.g., retro style, modern style, etc.). As an example, the style of the audio content can be associated with the style of the image content. For example, hip-hop style can correspond to retro style.

[0039] The process 200 is described below by way of example based on Figures 3A to 3G. It should be understood that Figures 3A to 3G and the corresponding embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0040] Figures 3A to 3G illustrate example interfaces 300A to 300G according to some embodiments of the present disclosure. Interfaces 300A to 300G may be provided, for example, by the electronic device 110 shown in Figure 1.

[0041] In some embodiments, as shown in FIG3A, the electronic device 110 may present a guide interface 300A. As an example, the electronic device 110 may present guide content on the guide interface 300A to guide the user in generating a music video. The guide content may include, for example, text content (e.g., “Generate your XXXX with one click”) and image content.

[0042] Additionally, the electronic device 110 may present a generation entry 305 in the guidance interface 300A. For example, the electronic device 110 may present text content (e.g., “Generate my XX single MV”) in the generation entry. Furthermore, the electronic device 110 may present a generation interface in response to a trigger on the generation entry 305 (e.g., a click operation).

[0043] In some embodiments, as shown in FIG3B, the electronic device 110 may present a generation interface 300B. The electronic device 110 may present multiple candidate text items in the generation interface 300B. For example, the multiple candidate text items may include text item 306-1 (e.g., “Relax all year round”), text item 306-2 (e.g., “Stay optimistic”), text item 306-3 (e.g., “Encounter romance”), and text item 306-4 (e.g., “XXX”), etc.

[0044] Additionally or alternatively, the electronic device 110 may, in response to receiving a sliding operation associated with multiple candidate text items, adjust the position of the multiple candidate text items in the generation interface 300B and / or present additional candidate text items in the generation interface 300B. Additionally, the electronic device 110 may, in response to receiving an update operation associated with multiple candidate text items (e.g., a click operation on an update control (not shown in the figure), update the multiple candidate text items.

[0045] In some embodiments, multiple candidate text items may be associated with a set of preset music themes. As an example, a set of preset music themes may include, for example, hip-hop themes, R&B themes, jazz themes, folk themes, etc.

[0046] Additionally, the electronic device 110 can receive a user's selection of at least one text item from a plurality of candidate text items.

[0047] As an example, as shown in Figure 3C, electronic device 110 can receive selections from a user for text items 306-1, 306-3, and 306-4. As an example, electronic device 110 can, in response to receiving a selection for at least one text item, distinguish and display at least one text item in the generation interface 300C. As an example, distinguishing and displaying may include, for example, highlighting, adjusting the background color, displaying indicator elements, etc. For example, electronic device 110 can distinguish and display (e.g., adjust the background color of the text items) text items 306-1, 306-3, and 306-4 in the generation interface 300C. Alternatively, electronic device 110 can, in response to a click operation on at least one text item, deselect at least one text item.

[0048] Additionally, the electronic device 110 may present at least one selected text item in a preset area (e.g., area 308) within the generation interface 300C. Further, the electronic device 110 may present a delete control associated with at least one text item in the preset area. The electronic device 110 may deselect at least one text item in response to a triggering (e.g., a click operation) of the delete control associated with at least one text item.

[0049] As an example, electronic device 110 or server 130 can determine reference text content based on at least one selected text item. For instance, electronic device 110 or server 130 can use at least one text item as reference text content.

[0050] Additionally or alternatively, the reference text content may include text content entered by the current user. For example, electronic device 110 may obtain the text content entered by the current user via an input control (not shown in the figure).

[0051] Additionally, the referenced text content may include text content determined based on reference information associated with the current user. For example, reference information may include publicly available content provided by the user. It should be noted that the user's reference information mentioned in this disclosure is obtained and used with the user's knowledge and authorization.

[0052] Referring again to Figure 2, in box 210, electronic device 110 or server 130 can generate audio content for a music video based on reference text content. As an example, electronic device 110 or server 130 can utilize a preset audio model to generate audio content based on reference text content. For example, the preset audio model can be implemented as a generative model capable of generating audio content and / or lyrics text based on text. This disclosure is not intended to limit the specific implementation and / or training process of such generative model.

[0053] Additionally or alternatively, electronic device 110 or server 130 may determine a prompt text (or prompt message) associated with the reference text content based on the reference text content. For example, electronic device 110 or server 130 may generate the prompt text based on the reference text content using a preset text model. For example, the prompt text may be an extension of the reference text content. As an example, the preset text model may be implemented as a generative model capable of generating music-related prompt text based on text, and this disclosure is not intended to limit the specific implementation and training process of such generative model.

[0054] For example, if the reference text includes phrases like "relax all year" and "encounter romance," the corresponding prompt text could include, for instance, "Write a song reflecting on the past year. The lyrics should be related to 'relax all year,' 'encounter romance,' and 'happiness,' in an R&B style, with relaxed and soothing lyrics. The musical structure should include an intro and two choruses." Furthermore, the electronic device 110 or server 130 can utilize a preset audio model to generate audio content and corresponding lyrics based on the prompt words.

[0055] In some embodiments, at box 245, the electronic device 110 or the server 130 can generate dynamic effects corresponding to the lyrics. As examples, dynamic effects may include changes in lyric color, changes in lyric font size, changes in lyric position, and / or the display or hiding of lyric.

[0056] Referring again to Figure 2, in box 215, electronic device 110 or server 130 can obtain user-uploaded videos and / or images.

[0057] Referring again to FIG3C, the electronic device 110 may present multiple candidate image contents associated with the user in a generation interface in response to confirmation of at least one selected text item. For example, the electronic device 110 may present multiple candidate image contents associated with the user in a generation interface in response to receiving a trigger (e.g., a click operation) on a confirmation control (e.g., "Next 310").

[0058] As shown in Figure 3D, the electronic device 110 can present multiple candidate image contents in the generation interface 300D. As an example, the multiple candidate image contents may include image content from the user's local media library (e.g., a local photo album). Additionally or alternatively, the multiple candidate image contents may include media content stored by the user in the cloud. For example, the multiple candidate media contents may include candidate image content 312-1, candidate image content 312-2, candidate image content 312-3, candidate image content 312-4, etc.

[0059] It should be noted that the image content mentioned in this disclosure, whether in the local media library or in the cloud storage, was obtained and used with the user's knowledge and authorization.

[0060] Additionally, the electronic device 110 can receive a user's selection of at least one candidate image content from a plurality of candidate image contents. For example, the electronic device 110 can receive a user's selection of candidate image content 312-3 and candidate image content 312-4. Alternatively, the electronic device 110 can distinguish and display (e.g., highlight, thicken edges, adjust colors, etc.) the selected candidate image content in the generation interface 300D.

[0061] Additionally, the electronic device 110 may present at least one selected candidate image content in a preset area (e.g., area 314). Alternatively, the electronic device 110 may present a delete control associated with the at least one selected candidate image content in the preset area. Further, the electronic device 110 may deselect at least one candidate image content in response to a triggering (e.g., a click operation) of the delete control associated with at least one candidate image content.

[0062] As an example, electronic device 110 or server 130 can determine reference image content based on at least one selected candidate image content. For instance, electronic device 110 or server 130 can use at least one candidate image content as reference image content.

[0063] In some embodiments, the music video includes a cover image. The cover image is generated based on at least a portion of the content of a reference image.

[0064] As an example, electronic device 110 or server 130 can determine an intermediate image based on the content of a reference image.

[0065] Referring again to Figure 2, in box 220, electronic device 110 or server 130 can determine whether the content of the reference image is video.

[0066] In box 230, electronic device 110 or server 130 can perform cloud editing and frame extraction on the reference image content in response to the reference image content being video. That is, electronic device 110 or server 130 can extract multiple image frames from the reference image content.

[0067] In box 235, electronic device 110 or server 130 can select highlight frames from multiple image frames for use in generating a music video. As an example, electronic device 110 or server 130 can use a preset model to select highlight frames from multiple image frames. As an example, the characteristics of a highlight frame may include: clear image, complete facial representation, etc. The preset model can be implemented as a machine learning model capable of selecting highlight frames from multiple image frames; this disclosure is not intended to limit the specific implementation or training process of such machine learning model.

[0068] In box 225, electronic device 110 or server 130 may randomly select an image from the reference image content in response to the reference image content being non-video.

[0069] As an example, electronic device 110 or server 130 can use a highlight frame or a randomly selected image as an intermediate image for generating a cover image.

[0070] Additionally, the electronic device 110 can apply a target effect to the intermediate image to generate a cover image. As an example, the target effect can be determined based on the type of the intermediate image. For example, if the intermediate image is a portrait or pet photo, the corresponding target effect could include, for example, an American portrait, a retro photo booth, etc. If the intermediate image is not a portrait (e.g., a landscape photo), the corresponding target effect could include, for example, a cartoon style, a traditional Chinese ink painting style, etc. As an example, the specific target effect can be randomly selected from multiple effects that match the type of the intermediate image or specified by the user.

[0071] In frame 250, electronic device 110 or server 130 can determine a splicing draft for generating a music video based on audio content, cover image, reference image content, music rhythm and lyric animation corresponding to the audio content, etc.

[0072] In frame 255, electronic device 110 or server 130 can perform editing operations on the splicing draft to generate a draft of the music video.

[0073] As an example, electronic device 110 or server 130 can determine the audio-video scene switching process based on the rhythm information of the audio content. As an example, multiple frames associated with the scene switching process can correspond to different frame portions generated based on multiple parts of reference image content. The scene switching process here can be understood as a transition process within the video content.

[0074] As an example, electronic device 110 or server 130 determines multiple time points based on the rhythm information of the audio content (e.g., the change in strong and weak beats corresponding to musical beats). Further, electronic device 110 or server 130 can determine multiple video segments corresponding to the multiple time points based on reference image content (e.g., each video segment can also be a set of image frames composed of multiple pictures). For example, four time points can correspond to five video segments. In other words, there are four time points for segment switching between the five video segments. For example, the last frame of each video segment can be an image frame suitable for transitions to ensure smooth transitions in the generated music video. As an example, multiple video segments can be associated with a preset duration. For example, if the preset duration is 10 seconds, then for a music video at 24 frames per second, the multiple video segments need to include 240 image frames.

[0075] Additionally, the electronic device 110 or server 130 can generate at least a portion of the images of the music video based on multiple video clips. For example, the music video may include image content corresponding to multiple video clips, and may also include image content generated based on multiple video clips.

[0076] As an example, electronic device 110 or server 130 can also determine the visual style based on the music style of the audio content. For example, hip-hop style can correspond to retro modeling style. For example, R&B can correspond to modern modeling style. Furthermore, electronic device 110 or server 130 can adjust multiple video clips based on the visual style to generate at least a portion of the image of the audio content. This at least a portion of the image is associated with the determined visual style.

[0077] In box 260, electronic device 110 or server 130 can determine whether the clipping was successful.

[0078] In box 270, electronic device 110 or server 130 may, in response to successful editing, use the image content generated based on the editing process as a draft of the music video.

[0079] In box 265, electronic device 110 or server 130 may, in response to editing failure, simply splice multiple parts of the splicing draft to obtain a draft of the music video. For example, electronic device 110 or server 130 no longer strictly adheres to the rhythm of the image portion of the music video with the audio content. For example, electronic device 110 or server 130 no longer adjusts the image based on the visual style of the audio content.

[0080] In box 275, server 130 can render the draft of the music video to obtain the music video. As an example, the music video may include audio content and image content. The image content may include image portions determined based on reference image content, lyrics text representations corresponding to the audio content, transition effects, and lyric effects, etc.

[0081] As shown in Figure 3E, the electronic device 110 can display a music video in the generation interface 300E in response to the generation of the music video. For example, the electronic device 110 may display the cover image of the music video in the generation interface 300E or play the music video in the generation interface 300E.

[0082] As an example, electronic device 110 can display reference text content in the generation interface 300E. For example, “#1 Relax all year round”, “#2 Encounter romance”, and “#3 XXX”.

[0083] Additionally, the electronic device 110 can present a creation entry point (e.g., "Make another 318") in the creation interface 300E. Furthermore, the electronic device 110 can present the creation interface 300B as shown in Figure 3B in response to the triggering of the creation entry point (e.g., a click operation).

[0084] Alternatively, the electronic device 110 may display history entries (e.g., history records 322) in the generation interface 300E. Further, the electronic device 110 may display multiple historical music videos in response to a triggering of a history entry (e.g., a click operation). These multiple historical music videos may include music videos created or saved by the current user.

[0085] Additionally, the electronic device 110 can present a sharing entry in the generated interface 300E (e.g., share my XX single 320). Furthermore, the electronic device 110 can trigger the sharing entry as a sharing request for music videos.

[0086] As shown in Figure 3F, the electronic device 110 can, in response to receiving a sharing request for a music video, display the cover image 324 of the music video in the generation interface 300F. Furthermore, the electronic device 110 can, in response to receiving an editing operation on the cover image, adjust the cover image. For example, the electronic device 110 can, in response to an update of the image content in the cover image, adjust the image content in the cover image to the image currently selected by the user. For example, the electronic device 110 can, in response to editing the text content in the cover image, adjust the text content in the cover image.

[0087] In some embodiments, the electronic device 110 may present a sharing window 326 in response to receiving a sharing request for a music video. The electronic device 110 may present multiple applications in the sharing window 326. These multiple applications may include, for example, application 1, application 2, and application 3. Further, the electronic device 110 may publish the music video in one of the multiple applications (e.g., application 2) in response to receiving a trigger on one of the applications.

[0088] Additionally, the electronic device 110 may present a save local option in the sharing window 326. Furthermore, in response to the triggering of the save local option, the electronic device 110 may save the music video to its local storage unit.

[0089] Alternatively, electronic device 110 may, in response to receiving a request to share a music video, create a web playback page associated with the music video. As an example, the web playback page may include a web playback page (e.g., an H5 playback page). The web playback page can be used to play the music video.

[0090] As an example, electronic device 110 may share a web playback page associated with a music video with at least one user based on the user's selection.

[0091] As shown in Figure 3G, electronic devices associated with other users can display a web playback page 300G corresponding to a music video. The web playback page 300G may include interactive controls 328. Interactive controls 328 can be used to receive interactive content from users. As an example, the interactive content can be text content (e.g., bullet comments). The interactive content can be configured to be overlaid on the music video. For example, interactive content 329-1 and interactive content 329-2, etc.

[0092] As an example, when the current user is playing music or video on the web playback page, they can also see interactive content sent by other users. Additionally, the current user can also delete interactive content on the web playback page. Alternatively, the current user can also set the display status of interactive content on the web playback page (e.g., show or hide).

[0093] As an example, the web playback page 300G may also include a creation portal 330. Furthermore, in response to other users triggering the creation portal 330, a generation interface for creating music videos is presented on the other users' electronic devices to guide them in completing the creation of the music videos.

[0094] In this manner, embodiments of the present disclosure are able to obtain media generation requests. Furthermore, embodiments of the present disclosure are able to generate audio content for a music video based on reference text content indicated by the media generation request, and generate at least a portion of the images for the music video based on reference image content indicated by the media generation request. Additionally, the image portion of the music video created based on embodiments of the present disclosure can be associated with rhythm information corresponding to the audio portion. Thus, embodiments of the present disclosure are able to generate music videos that match image and audio content based on user configurations, thereby meeting users' needs for creating music and video content and improving the efficiency of user-generated media content.

[0095] Example process

[0096] Figure 4 shows a flowchart of an example process 400 for generating a music video according to some embodiments of the present disclosure. Process 400 can be implemented at an electronic device 110. Process 400 will now be described with reference to Figure 1.

[0097] As shown in Figure 4, in box 410, electronic device 110 obtains a media generation request, which indicates reference text content and reference image content.

[0098] In box 420, electronic device 110 generates a music video based on a media generation request, wherein the audio content of the music video is generated based on selected reference text content, and at least a portion of the images in the music video are generated based on reference image content.

[0099] In some embodiments, process 400 further includes: presenting a plurality of candidate text items; and determining reference text content based on the selection of at least one text item among the plurality of candidate text items.

[0100] In some embodiments, multiple candidate text items are associated with a set of preset music themes.

[0101] In some embodiments, the reference text content includes text content entered by the user or text content determined based on reference information associated with the user.

[0102] In some embodiments, the scene transitions in a music video are determined based on the rhythm information of the audio content.

[0103] In some embodiments, audio content is generated based on the following process: determining a cue text corresponding to reference text content; and providing the cue text to an audio generation model to generate audio content.

[0104] In some embodiments, the music video includes a cover image, and the cover image is generated based on at least a portion of the content of a reference image.

[0105] In some embodiments, the cover image is generated based on the following process: determining an intermediate image based on the content of a reference image; and applying a target effect to the intermediate image to generate the cover image, wherein the target effect is determined based on the type of the intermediate image.

[0106] In some embodiments, process 400 further includes: presenting a cover image of a music video; and adjusting the cover image in response to receiving an editing operation on the cover image.

[0107] In some embodiments, the audio video includes dynamic effects generated based on lyrics corresponding to the audio content.

[0108] In some embodiments, at least a portion of the images in a music video are generated based on the following process: determining multiple time points based on rhythm information of the audio content; determining multiple video segments corresponding to the multiple time points based on reference image content; and generating at least a portion of the images based on the multiple video segments.

[0109] In some embodiments, generating at least a portion of the image further includes: determining the visual style based on the music style of the audio content; and adjusting multiple video clips based on the visual style to generate at least a portion of the image.

[0110] In some embodiments, process 400 further includes: in response to receiving a sharing request for a music video, creating a web playback page associated with the music video, the web playback page being used to play the music video; and sending the web playback page to the at least one user based on a selection of at least one user.

[0111] In some embodiments, the web playback page includes interactive controls for receiving user interaction content, which is configured to be overlaid on the music video.

[0112] Example devices and equipment

[0113] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 5 shows a schematic structural block diagram of an example apparatus 500 for generating music videos according to certain embodiments of this disclosure. Apparatus 500 may be implemented as or included in electronic device 110. The various modules / components in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0114] As shown in Figure 5, the device 500 includes: an acquisition module 510 configured to present a generation interface, the generation interface including at least one camera movement control; and a generation module 520 configured to generate a music video based on a media generation request, wherein the audio content of the music video is generated based on selected reference text content, and at least a portion of the images in the music video are generated based on reference image content.

[0115] In some embodiments, the apparatus 500 further includes a selection module configured to: present a plurality of candidate text items; and determine reference text content based on the selection of at least one of the plurality of candidate text items.

[0116] In some embodiments, multiple candidate text items are associated with a set of preset music themes.

[0117] In some embodiments, the reference text content includes text content entered by the user or text content determined based on reference information associated with the user.

[0118] In some embodiments, the scene transitions in a music video are determined based on the rhythm information of the audio content.

[0119] In some embodiments, audio content is generated based on the following process: determining a cue text corresponding to reference text content; and providing the cue text to an audio generation model to generate audio content.

[0120] In some embodiments, the music video includes a cover image, and the cover image is generated based on at least a portion of the content of a reference image.

[0121] In some embodiments, the cover image is generated based on the following process: determining an intermediate image based on the content of a reference image; and applying a target effect to the intermediate image to generate the cover image, wherein the target effect is determined based on the type of the intermediate image.

[0122] In some embodiments, the device 500 further includes an adjustment module configured to: present a cover image of a music video; and adjust the cover image in response to receiving an editing operation on the cover image.

[0123] In some embodiments, the audio video includes dynamic effects generated based on lyrics corresponding to the audio content.

[0124] In some embodiments, at least a portion of the images in a music video are generated based on the following process: determining multiple time points based on rhythm information of the audio content; determining multiple video segments corresponding to the multiple time points based on reference image content; and generating at least a portion of the images based on the multiple video segments.

[0125] In some embodiments, generating at least a portion of the image further includes: determining the visual style based on the music style of the audio content; and adjusting multiple video clips based on the visual style to generate at least a portion of the image.

[0126] In some embodiments, the apparatus 500 further includes a sharing module configured to: in response to receiving a sharing request for a music video, create a web playback page associated with the music video, the web playback page being used to play the music video; and send the web playback page to at least one user based on a selection of at least one user.

[0127] In some embodiments, the web playback page includes interactive controls for receiving user interaction content, which is configured to be overlaid on the music video.

[0128] The units included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0129] Figure 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in Figure 6 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 600 shown in Figure 6 can be used to implement the electronic device 110 of Figure 1.

[0130] As shown in Figure 6, the electronic device 600 is in the form of a general-purpose electronic device. Components of the electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage devices 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0131] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0132] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0133] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0134] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0135] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0136] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0137] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0138] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0140] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating a music video, comprising: Obtain a media generation request, wherein the media generation request indicates reference text content and reference image content; as well as Based on the media generation request, a music video is generated, wherein the audio content of the music video is generated based on the selected reference text content, and at least a portion of the images of the music video are generated based on the reference image content.

2. The method according to claim 1, further comprising: Present multiple candidate text items; as well as The reference text content is determined based on the selection of at least one text item from the plurality of candidate text items.

3. The method of claim 2, wherein the plurality of candidate text items are associated with a set of preset music themes.

4. The method according to any one of claims 1 to 3, wherein the reference text content includes text content input by the user or text content determined based on reference information associated with the user.

5. The method according to any one of claims 1 to 4, wherein the switching of the music video is determined based on the rhythm information of the audio content.

6. The method according to any one of claims 1 to 5, wherein the audio content is generated based on the following process: Determine the prompt text corresponding to the reference text content; and The prompt text is provided to the audio generation model to generate the audio content.

7. The method according to any one of claims 1 to 6, wherein the music video includes a cover image, and the cover image is generated based on at least a portion of the content of the reference image.

8. The method of claim 7, wherein the cover image is generated based on the following process: Based on the content of the reference image, an intermediate image is determined; and A target effect is applied to the intermediate image to generate the cover image, wherein the target effect is determined based on the type of the intermediate image.

9. The method according to claim 8, further comprising: Present the cover image of the music video; as well as In response to receiving an editing operation on the cover image, the cover image is adjusted.

10. The method according to any one of claims 1 to 9, wherein the audio-video includes dynamic effects generated based on lyrics corresponding to the audio content.

11. The method according to any one of claims 1 to 10, wherein the at least portion of the images of the music video are generated based on the following process: Based on the rhythm information of the audio content, multiple time points are determined; Based on the content of the reference image, multiple video segments corresponding to the multiple time points are determined; as well as The at least partial images are generated based on the multiple video clips.

12. The method of claim 11, wherein generating the at least partial image further comprises: Based on the music style of the audio content, determine the visual style; as well as The multiple video clips are adjusted based on the described visual style to generate at least some of the images.

13. The method according to any one of claims 1 to 12, further comprising: In response to receiving a sharing request for the music video, a web playback page associated with the music video is created, the web playback page being used to play the music video; as well as Based on the selection of at least one user, the online playback page is sent to the at least one user.

14. The method of claim 13, wherein the network playback page includes an interactive control for receiving user interaction content, and the interactive content is configured to be overlaid on the music video.

15. An apparatus for generating music videos, comprising: The acquisition module is configured to acquire a media generation request, wherein the media generation request indicates reference text content and reference image content; as well as The generation module is configured to generate a music video based on the media generation request, wherein the audio content of the music video is generated based on the selected reference text content, and at least a portion of the images of the music video are generated based on the reference image content.

16. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processor.

17. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 14.

18. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 14.

Images