Subtitle content display method and apparatus, device, medium, and product

Abstract

A subtitle content display method and apparatus, a device, a medium, and a product. The method comprises: playing a first video, and displaying a first subtitle corresponding to voice material of currently played first video content (210); and determining a first playing volume of the first video, and, when the first playing volume is lower than a first condition, displaying a second subtitle corresponding to voice material of currently played second video content, subtitle display styles of the first subtitle and the second subtitle being different (220).

Description

Methods, devices, equipment, media, and products for displaying subtitle content

[0001] This application claims priority to Chinese Patent Application No. 202510008833.0, filed on January 2, 2025, entitled "Method, Apparatus, Device, Medium and Product for Displaying Subtitle Content", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a method, apparatus, device, medium and product for displaying subtitle content. Background Technology

[0003] Video advertising is emerging as a new form of advertising, aiming to attract viewers' attention by combining visual and auditory elements, thereby promoting products, services, or brands.

[0004] In related technologies, displaying advertising videos is usually implemented as a splash screen display or an interstitial display. In order to reduce users' resistance to advertising videos, they are usually played in a default mute mode.

[0005] However, when the ad video is muted, users cannot perceive the auditory elements of the ad video, resulting in a lack of information in the audio content of the ad video. Summary of the Invention

[0006] This application provides a method, apparatus, device, medium, and product for displaying subtitle content. The technical solution is as follows:

[0007] On one hand, a method for displaying subtitle content is provided, the method being executed by a computer device, the method comprising:

[0008] Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content;

[0009] Determine the first playback volume of the first video. If the first playback volume is lower than a first condition, display a second subtitle corresponding to the audio material of the currently playing second video content. The first subtitle and the second subtitle have different subtitle display styles.

[0010] On the other hand, a method for displaying subtitle content is provided, the method being executed by a computer device, the method comprising:

[0011] Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content. The first video is used to recommend the first recommended content.

[0012] The first subtitle and the second subtitle are displayed in different styles.

[0013] Receive interactive operations on the first video;

[0014] In response to the interactive operation, the recommendation information corresponding to the first recommended content is displayed.

[0015] On the other hand, a device for displaying subtitle content is provided, the device comprising:

[0016] The first display module is used to play the first video and display the first subtitles corresponding to the audio material of the currently playing first video content;

[0017] The first determining module is used to determine the first playback volume of the first video;

[0018] The first display module is further configured to display a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition, wherein the first subtitle and the second subtitle have different subtitle display styles.

[0019] On the other hand, a device for displaying subtitle content is provided, the device comprising:

[0020] The second display module is used to play the first video and display the first subtitles corresponding to the audio material of the currently playing first video content. The first video is used to recommend the first recommended content.

[0021] The second display module is also used to display a second subtitle corresponding to the audio material of the currently playing second video content, wherein the first subtitle and the second subtitle have different subtitle display styles;

[0022] The second receiving module is used to receive interactive operations on the first video;

[0023] The second display module is also configured to respond to the interactive operation by displaying recommendation information corresponding to the first recommended content.

[0024] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the method for displaying subtitle content as described in any of the embodiments of this application above.

[0025] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method for displaying subtitle content as described in any of the embodiments of this application above.

[0026] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for displaying subtitle content as described in any of the above embodiments.

[0027] The technical solution provided in this application includes at least the following beneficial effects:

[0028] During the playback of the first video at the first playback volume, if the first playback volume of the first video is lower than the first condition, a second subtitle is displayed to compensate for the transmission of audio content-related information. The second subtitle is different from the first subtitle that is normally displayed in the video. That is, in scenarios where the first playback volume cannot meet the user's audio perception needs of the first video, the second subtitle is used to compensate for the transmission of audio information, thereby improving the information transmission efficiency of the first video and enhancing the user experience.

[0029] When recommending content to users through the first video, a second subtitle, distinct from the first subtitle, is used to assist in conveying information from the audio content in the first video. On the one hand, the second subtitle improves the information delivery efficiency of the first video in low-volume scenarios, enhancing the user experience; on the other hand, it can also increase user attention to the first video, thereby improving the recommendation conversion rate when recommending content through the first video; furthermore, by automatically detecting the playback volume of the first video and dynamically triggering the display of subtitles with different visual effects, the automation level of the first video playback is improved, enhancing the flexibility of content display in different volume scenarios. Attached Figure Description

[0030] Figure 1 is a structural block diagram of a computer system provided in an exemplary embodiment of this application;

[0031] Figure 2 is a flowchart of a method for displaying subtitle content provided in an exemplary embodiment of this application;

[0032] Figure 3 is a schematic diagram of the implementation of the subtitle function provided in an exemplary embodiment of this application;

[0033] Figure 4 is a schematic diagram of the implementation of the subtitle function provided in an exemplary embodiment of this application;

[0034] Figure 5 is a schematic diagram of the implementation of the subtitle function provided in an exemplary embodiment of this application;

[0035] Figure 6 is a schematic diagram of the implementation of the subtitle function provided in an exemplary embodiment of this application;

[0036] Figure 7 is a schematic diagram of a first interactive component provided in an exemplary embodiment of this application;

[0037] Figure 8 is a flowchart of a method for displaying subtitle content provided in an exemplary embodiment of this application;

[0038] Figure 9 is a schematic diagram illustrating the playback triggering of a first video provided in an exemplary embodiment of this application;

[0039] Figure 10 is a schematic diagram illustrating the playback triggering of a first video provided in an exemplary embodiment of this application;

[0040] Figure 11 is a flowchart of a method for generating subtitle data provided in an exemplary embodiment of this application;

[0041] Figure 12 is a schematic diagram of a subtitle display style provided in an exemplary embodiment of this application;

[0042] Figure 13 is a schematic diagram of the advertising video display process in an advertising scenario provided by an exemplary embodiment of this application;

[0043] Figure 14 is a structural block diagram of a caption content display device provided in an exemplary embodiment of this application;

[0044] Figure 15 is a structural block diagram of a caption content display device provided in an exemplary embodiment of this application;

[0045] Figure 16 is a structural block diagram of a caption content display device provided in an exemplary embodiment of this application;

[0046] Figure 17 is a structural block diagram of a caption content display device provided in an exemplary embodiment of this application;

[0047] Figure 18 is a structural block diagram of a terminal provided in an exemplary embodiment of this application. Detailed Implementation

[0048] Figure 1 shows a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system 100 includes: a first terminal 110, a second terminal 120, and a server 130.

[0049] The first terminal 110 may have a client application for a first application installed. This first application has video playback functionality and displays subtitles corresponding to the audio content of the first video during playback. Optionally, users can publish videos and watch videos published by other users within the first application. In some embodiments, the first application may possess the social functions of a social application, meaning users can watch videos published by social media users within the first application.

[0050] The second terminal 120 may have a client for a second application installed. This second application has the function of publishing video content and is used to broadcast video content on the application platform where the first application resides. Optionally, users can publish videos and watch videos published by other users in the second application. In some embodiments, the second application may have the social functions of a social application, meaning users can watch videos published by social media users in the second application.

[0051] Optionally, the first application and the second application can be implemented as the same application, or the first application and the second application can be implemented as different applications.

[0052] Optionally, the first application and / or the second application may be an application that requires downloading and installation, or it may be an application that can be used instantly; this application does not limit this. Regarding the type of the aforementioned first application and / or second application, this application does not limit it, and includes, but is not limited to, video applications, social applications, web applications, e-commerce applications, game applications, etc. In some embodiments, the first application may be implemented as a user-side application of a video application, and the second application may be implemented as a creator-side application of a video application; in other embodiments, the first application may be implemented as a video application, and the second application may be implemented as an advertising delivery application.

[0053] The device types of the first terminal 110 and the second terminal 120 include at least one of the following: game console, virtual reality (VR) device, desktop computer, smartphone, tablet computer, e-book reader, and laptop computer.

[0054] The first terminal 110 is connected to the server 130 via a wireless network or a wired network; the second terminal 120 is connected to the server 130 via a wireless network or a wired network.

[0055] Those skilled in the art will understand that the number of the aforementioned devices can be more or less. For example, there may be only one device, or there may be dozens or hundreds of devices, or even more. This application does not limit the number or type of devices.

[0056] Server 130 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 130 is used to provide background services for a first client of a first application installed and running on a first terminal 110 and a second client of a second application installed and running on a second terminal 120.

[0057] In some embodiments, the creator uploads a first video to the server 130 via a second terminal 120. The first video contains a first subtitle. The server 130 calls a subtitle generation service to add a second subtitle to the first video, and the second subtitle has a different display style than the first subtitle. When a user watches the first video via the first terminal 110, the second subtitle is displayed when the first playback volume of the first video is lower than a first condition.

[0058] In other embodiments, the advertiser uploads an advertising video (first video) to the server 130 via the second terminal 120. The advertising video contains a first subtitle. The server 130 calls a subtitle generation service to add a second subtitle to the advertising video, and the display style of the second subtitle is different from that of the first subtitle. When a user browses content via the first terminal 110, if the advertising push conditions are met, the first terminal 110 plays the advertising video and provides the second subtitle to the user.

[0059] It is worth noting that the server 130 described above can be implemented as a physical server or as a cloud server in the cloud. In some embodiments, the server 130 can also be implemented as a node in a blockchain system.

[0060] In one example, taking the aforementioned server 130 as an example of providing cloud services to users, server 130 includes a cloud server 131, an artificial intelligence (AI) processing server 132, and a database 133. The cloud server 131 is used to receive a first video uploaded by an advertiser or creator through a second terminal 120. If there is a need to generate subtitles for the first video, the cloud server 131 sends the first video to the AI ​​processing server 132, which generates subtitle data and stores the subtitle data and the first video in the database 133. When there is a need to play the first video on the first terminal 110, the cloud server 131 sends the first video and the corresponding subtitle data to the first terminal 110.

[0061] Example 1: In scenarios where there is a specified volume requirement, a subtitle function is provided to the user. The subtitle function enables the expression of the emotions of the audio material in the video through the subtitle content in scenarios with specified volume requirements.

[0062] Please refer to Figure 2, which shows a flowchart of a method for displaying subtitle content provided in an exemplary embodiment of this application. The method is executed by a computer device. For example, the method is executed by a terminal. The method includes the following steps 210 to 220.

[0063] Step 210: Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content.

[0064] The first video is a video published by a publisher account. The publisher account can be a user account logged in on a second terminal or a user account logged in on the current terminal (first terminal).

[0065] Optionally, the first video may be video content published on the application platform corresponding to the first application, or it may be advertising content delivered to the application platform corresponding to the first application through the second application.

[0066] The video content of the first video includes at least one of image content, subtitle content, and audio content. In some embodiments, the first video is a video containing only one of these contents; for example, the first video may be a video containing only image content, a video containing only subtitle content, or a video containing only audio content. In some embodiments, the first video is a video containing multiple of these contents; for example, the first video may be a video containing only image content and subtitle content, a video containing both image content and audio content, a video containing both subtitle content and audio content, or a video containing all three.

[0067] In this embodiment of the application, the first video includes audio material, wherein the audio material is a component of the audio content of the first video. In some embodiments, the audio content of the first video includes background music, dialogue, voice-over, background music, sound effects, ambient sounds, etc., and the audio material includes at least one of the above-mentioned audio contents.

[0068] In this embodiment of the application, the first video corresponds to a first subtitle. During the playback of the first video, the terminal displays the first subtitle corresponding to the audio material of the currently playing first video content.

[0069] In some embodiments, the first subtitle is a subtitle for at least one of the following: dialogue between characters, narration, and background music in the first video. In one example, taking a short drama video as an example, the first subtitle is the dialogue between characters in the short drama video.

[0070] Step 220: Determine the first playback volume of the first video. If the first playback volume is lower than the first condition, display the second subtitle corresponding to the audio material of the currently playing second video content. The first subtitle and the second subtitle have different subtitle display styles.

[0071] Schematic, the first playback volume of the first video is used to indicate the playback volume of the first video during playback. Optionally, the first playback volume of the first video is determined when playback begins; alternatively, the first playback volume of the first video is determined during playback.

[0072] In some embodiments, the first playback volume is determined by the playback scene information corresponding to the first video, that is, the determination of the first playback volume of the first video is implemented as follows: S1 and S2:

[0073] S1, obtain the playback scene information corresponding to the first video.

[0074] The playback scene information of the first video is used to determine the playback volume of the first video. For example, the playback scene information indicates relevant information affecting the playback volume within the playback scene of the first video. Optionally, the playback scene information of the first video includes information indicating the influence of the physical environment on the playback volume of the first video, information indicating the influence of the video type on the playback volume of the first video, information indicating the influence of playback control operations on the playback volume of the first video, etc.

[0075] Optionally, determining the playback scene information of the first video includes at least one of the following:

[0076] The first method involves collecting the physical environment information of the terminal playing the first video as the playback scene information.

[0077] Schematic, the physical environment information is used to indicate the environmental conditions of the surrounding environment of the terminal playing the first video. Optionally, the physical environment information includes ambient volume, the number of ambient sound sources, etc.

[0078] In one example, ambient audio is captured by enabling the terminal's microphone, and the ambient audio volume is detected to obtain the aforementioned ambient volume. In another example, after detecting ambient audio, ambient sound sources within the ambient audio are detected to determine the number of ambient sounds.

[0079] The second method is to obtain the playback scene information corresponding to the video type of the first video.

[0080] For illustrative purposes, the video type is used to indicate the category in which the first video is classified according to specified criteria and characteristics. For example, based on whether the video content of the first video is content recommendation-based, the resulting video types include content recommendation-based types (advertisements) and non-content recommendation-based types (uploaded by ordinary users); as another example, based on the video length of the first video, the resulting video types include short video types and long video types.

[0081] In some embodiments, playback scenario information corresponding to different video types indicates that the first video is played at different playback volumes. In one example, when the video type is a content recommendation type, the determined playback scenario information indicates that the first playback volume of the first video is a first volume value; when the video type is a non-content recommendation type, the determined playback scenario information indicates that the first playback volume of the first video is a second volume value, wherein the first volume value is greater than the second volume value.

[0082] The third method involves receiving playback control operations and determining the corresponding playback scene information based on these operations.

[0083] Indicatively, playback control operations are used to indicate operations that control the playback of the first video. Optionally, playback control operations include start playback, subtitle display, volume control, progress control, etc.

[0084] Optionally, a volume control operation is received, and playback scene information is determined based on the volume control operation. The volume control operation is used to control the first video to play at a first playback volume.

[0085] Optionally, a subtitle display operation is received, and playback scene information is determined based on the subtitle display operation. The subtitle display operation is used to trigger the display of subtitles for the first video, which are the subtitles used when the first video is played at a first playback volume.

[0086] Optionally, a progress control operation is received, and playback scene information is determined based on the progress control operation. The progress control operation is used to control the first video to fast-forward to the playback progress corresponding to the audio material of the first video, which is audio played at a first playback volume.

[0087] Optionally, the playback control operations can be implemented in ways including but not limited to single-click, double-click, swipe, selection, and click operations for the pause function.

[0088] S2, based on the playback scene information, determines the first playback volume of the first video.

[0089] In some embodiments, when the physical environment information of the terminal playing the first video is collected as the playback scene information of the first video, the first playback volume of the first video is determined based on the physical environment information, and the first video is played at the first playback volume. That is, the appropriate playback volume is automatically adapted to the playback scene of the first video to achieve playback of the first video, thereby dynamically realizing the switching between the first subtitle and the second subtitle. This allows the display of the first subtitle and the second subtitle to adapt to the specific playback scene, improving the diversity of the first video playback methods and the adaptability of the first video in different playback scenes, thereby improving the completion rate of the first video in diverse playback scenarios.

[0090] In one example, when the physical environment information includes ambient volume, the first playback volume of the first video is determined as the first volume value if the ambient volume is within the first volume range.

[0091] Optionally, the aforementioned first volume range can be a system preset, a user-defined setting, or a setting determined based on the device's current volume setting. For example, when the first volume range is determined based on the terminal's current volume setting, the process of determining the first volume range is as follows: obtain the terminal's corresponding volume setting, and combine the volume setting with a preset percentage to obtain the range threshold corresponding to the first volume range; for example, if the volume setting is 50 decibels and the preset percentage is 60%, then the range threshold for the first volume range is determined to be 30. When the ambient volume is less than 30 decibels, the first video is played at the first volume value; that is, when the ambient volume is low, the first video is played at a low / mute volume.

[0092] In some embodiments, when the playback scene information corresponding to the video type of the first video is obtained, the first playback volume of the first video is determined based on the video type of the first video, and the first video is played at the first playback volume. Illustratively, when the video type matches a specified video type, the first playback volume of the first video is determined as a first volume value.

[0093] In one example, when the video type includes both content recommendation and non-content recommendation types, if the first video's type is content recommendation, the first playback volume of the first video is determined as the first volume value. For example, if the first video is an advertisement video, it is played at a low / mute volume.

[0094] In some embodiments, when a playback control operation is received and corresponding playback scene information is determined based on the playback control operation, a first playback volume of the first video is determined based on the playback control operation, and the first video is played at the first playback volume. Illustratively, if the operation indicated by the playback control operation satisfies specified operation conditions, the first playback volume of the first video is determined as a first volume value.

[0095] In one example, when the playback control operation is implemented as a volume control operation, and when the volume control operation is used to control the first video to play at a first volume value, the first playback volume of the first video is determined to be the first volume value.

[0096] In another example, when the playback control operation is implemented as a subtitle display operation, and when the subtitle display operation is used to trigger the target subtitle function, the first playback volume of the first video is determined to be the first volume value. Here, the aforementioned target subtitle function is a subtitle function used to display subtitle content whose subtitle display style is related to the emotional tone of the audio material.

[0097] In this embodiment of the application, after determining the first playback volume of the first video, it is determined whether the first playback volume is lower than a first condition. If the first playback volume is lower than the first condition, a second subtitle corresponding to the audio material of the currently playing second video content is displayed. The first subtitle and the second subtitle have different subtitle display styles.

[0098] In some embodiments, the first condition can be implemented as a preset volume threshold, that is, when the first playback volume is less than the preset volume threshold, a second subtitle corresponding to the audio material of the currently playing second video content is displayed.

[0099] Optionally, the preset volume threshold can be a system-preset threshold or a user-defined threshold.

[0100] In some embodiments, a preset volume threshold is used to determine whether the first playback volume is a mute volume or a low volume. That is, the preset volume threshold is a low volume threshold, wherein a mute volume is a volume set to zero or the speaker is off.

[0101] In some embodiments, when the first playback volume is lower than a first condition, a second subtitle is displayed in a first subtitle display style. The first subtitle display style differs depending on the emotional tone of the audio material corresponding to different voice emotions in the second video content. That is, the display style of the second subtitle is associated with the emotional tone of the audio material. Therefore, in scenarios where the first playback volume cannot meet the user's perception needs for the audio information of the first video, the second subtitle compensates for the emotional transmission of the audio information, thereby improving the information transmission efficiency of the first video. This eliminates the need for the user to adjust the volume to obtain audio information, reducing user operational requirements while ensuring the user can obtain video information from the first video at low volumes, thus improving the user experience.

[0102] For example, the display style of the second subtitle is related to the emotional tone of the audio material of the second video content. That is, the display style of the first subtitle of the second subtitle is different when the audio material corresponds to different emotional tones.

[0103] Optionally, the aforementioned first subtitle display style includes subtitle font style, subtitle color style, subtitle animation effect, subtitle display duration, subtitle language type, subtitle size, subtitle transparency, subtitle display position, etc.

[0104] Optionally, the aforementioned vocal emotions include positive emotions (e.g., happiness, joy, excitement, etc.), neutral emotions (e.g., surprise, curiosity, relaxation, arrogance, sympathy, etc.), and negative emotions (e.g., sadness, anger, fear, disgust, frustration, etc.).

[0105] In some embodiments, the process of displaying the second subtitle of the first video is implemented as follows: acquiring the audio material corresponding to the content of the second video; converting the audio material into audio text; recognizing the semantics of the audio text to obtain the emotion recognition result corresponding to the audio text; and displaying the second subtitle in the first subtitle display style based on the emotion recognition result.

[0106] In one example, when the voice material corresponds to a negative emotion, the first caption display style includes using dark colors, bold fonts, and fast-paced dynamic effects. In another example, when the voice material corresponds to a positive emotion, the first caption display style includes using bright colors, large fonts, fast-paced dynamic effects, and decorative patterns. In yet another example, when the voice material corresponds to a neutral emotion, the first caption display style includes using small fonts, slow-paced dynamic effects, and high transparency.

[0107] Optionally, the second subtitle may be generated and rendered in real time during the playback of the first video; or, the second subtitle may be pre-generated by the server based on the first video.

[0108] Optionally, the second subtitle is displayed at a fixed position in the first video, for example, at the top or bottom of the first video; or, the display position of the second subtitle is associated with the first voice object corresponding to the voice material, wherein the voice emotion of the voice material matches the emotional state of the first voice object.

[0109] Optionally, the first voice object includes people, animals, plants, scene objects, etc. in the first video.

[0110] The matching of the emotional tone of the audio material with the emotional state of the first voice subject is used to indicate that the audio material is audio used to express the emotional state of the first voice subject. Optionally, the audio material may be the first voice subject's lines, background music when the camera focuses on the first voice subject, or narration directed at the first voice subject.

[0111] Schematic illustration: When the playback volume is lower than a first condition, a second subtitle is displayed at a first position associated with the first audio object. Optionally, the first position includes the object area or surrounding area corresponding to the first audio object. For example, the second subtitle is displayed above the head or face of the person expressing the emotional tone of the audio material. That is, displaying the second subtitle at the first position associated with the first audio object enhances the visual association between the first audio object and the second subtitle, thereby improving the display effect of the second subtitle. In other words, associating the second subtitle with the first audio object allows for a visual indication of which object the content and emotion expressed by the subtitle originates from, eliminating the need for auditory differentiation, increasing the amount of information conveyed by the second subtitle, and enhancing the information delivery capacity of the second subtitle.

[0112] In this embodiment of the application, the method for displaying the second subtitle can be implemented in at least one of the following scenarios:

[0113] 1. Provide users with subtitles for the first video in both quiet and noisy environments.

[0114] Optionally, the terminal detects the ambient volume. When the ambient volume is lower than a first volume threshold, it determines that the first video is played at a first playback volume and displays a second subtitle corresponding to the audio material of the currently playing second video content. The first volume threshold is used to determine whether the physical environment in which the terminal is located is a quiet environment.

[0115] Optionally, the terminal detects the ambient volume. When the ambient volume is higher than a second volume threshold, it determines that the first video is played at a first playback volume and displays a second subtitle corresponding to the audio material of the currently playing second video content. The second volume threshold is used to determine whether the physical environment in which the terminal is located is noisy.

[0116] Optionally, the first playback volume corresponding to the first volume threshold and the first playback volume corresponding to the second volume threshold can be the same volume or different volumes. For example, in both quiet and noisy environments, a low / mute volume is used as the first playback volume. In quiet environments, a low / mute volume is used to match the volume requirements of the quiet environment, and while meeting the volume requirements of the quiet environment, the displayed subtitles help the user perceive the emotional tone of the voice material. In noisy environments, using a low / mute volume reduces the power consumption of the terminal speaker, and the displayed subtitles help the user perceive the emotional tone of the voice material to compensate for the reduced information transmission of the voice material. Alternatively, the first playback volume used in a quiet environment is a low / mute volume, while the first playback volume used in a noisy environment is a high volume higher than the ambient volume. In noisy environments, the high volume and subtitles work together to assist in the information transmission of the voice material.

[0117] In one example, taking the function of providing subtitles for the first video to the user in a quiet environment using silent playback as an example, as shown in Figure 3, it illustrates a schematic diagram of the implementation of the subtitle function provided by an exemplary embodiment of this application. Before playing the first video, the terminal 300 detects the ambient volume of the surrounding environment. When the ambient volume is lower than a first volume threshold, it automatically adopts a silent mode to play the first video 310. When the first video 310 plays to the video content corresponding to the target audio material, it displays the second subtitle 311 corresponding to the target audio material. For example, if the audio material corresponding to the video content is a character 312 crying and calling for his mother, then the subtitle display style of the displayed second subtitle 311 is displayed in the manner of simulating tears, thereby expressing the sad emotion of the character 312 in this audio material.

[0118] 2. Provide the user with subtitles for the first video when the user controls the volume to play the first video at the first playback volume.

[0119] Indicatively, when a first video is in playback or about to be played, a volume control operation is received, instructing the first video to be played at a first playback volume. Based on the volume control operation, the first video is played at the first playback volume. When the first playback volume is less than a third volume threshold, a second subtitle is displayed during the playback of the first video.

[0120] Optionally, when the first playback volume is less than the third volume threshold, the first playback volume is a mute volume or a low volume.

[0121] Figure 4 shows a schematic diagram of the implementation of the subtitle function provided by an exemplary embodiment of this application. The video playback interface 400 displayed on the terminal includes a first video 410 in playback state or to be played and a volume control control 420. When a volume control operation is received through the volume control control 420, and the first playback volume indicated by the volume control operation is less than or equal to a third volume threshold (the figure takes the setting to mute through the volume control control 420 as an example), during the playback of the first video 410, when the first video 410 plays to the video content corresponding to the target audio material, a second subtitle 411 corresponding to the target audio material is displayed.

[0122] 3. Users can trigger the subtitle function independently through the subtitle display operation.

[0123] Indicatively, when the first video is in playback or about to be played, a subtitle display operation is received. The subtitle display operation instructs the display of a second subtitle of the first video. During the playback of the first video at the first playback volume, the second subtitle corresponding to the audio material is displayed. This second subtitle is triggered by the subtitle display operation.

[0124] Optionally, the first playback volume can be a mute volume or a low volume, or the first playback volume can be any volume set by the user.

[0125] Figure 5 shows a schematic diagram of the implementation of the subtitle function provided by an exemplary embodiment of this application. The video playback interface 500 displayed on the terminal includes a first video 510 in playback status or to be played and a subtitle display control 520. When a subtitle display operation is received through the subtitle display control 520, the subtitle function is enabled. During the playback of the first video 510, when the first video 510 plays to the video content corresponding to the target audio material, a second subtitle 511 corresponding to the target audio material is displayed.

[0126] 4. When the first video is an advertisement video, provide users with a subtitle function for the first video.

[0127] Indicatively, the video type of the first video is obtained. If the video type of the first video is an advertisement, the first video is played at the first playback volume, and during the playback of the first video, the second subtitle corresponding to the audio material is displayed.

[0128] Optionally, the first playback volume can be a mute volume or a low volume, or the first playback volume can be any volume set by the user.

[0129] For example, when the first playback volume is set to a mute or low volume, the ad video is played at a low or mute volume, which reduces the user's resistance to the ad video. The second subtitle, which uses voice to express emotions, compensates for the information delivery problems caused by the low volume / mute, improves the user experience, thereby increasing the completion rate of the ad video and ensuring the conversion rate of the ad video.

[0130] Figure 6 shows a schematic diagram of the implementation of the subtitle function provided in an exemplary embodiment of this application. The video playback interface 600 displayed on the terminal includes a first video 610 in a playback state or to be played. When the first video 610 is an advertising video, the first video 610 is played at a first playback volume. When the first video 610 plays to the video content corresponding to the target audio material, the second subtitle 611 corresponding to the target audio material is displayed.

[0131] In some optional embodiments, when the first playback volume is lower than a first condition, a first interactive component is displayed, wherein the component display style of the first interactive component is associated with the emotional tone of the audio material of the second video content; in response to receiving a trigger operation on the first interactive component, a second subtitle corresponding to the audio material is displayed.

[0132] Optionally, the above component display styles include component color style, component dynamic effects, component display duration, component size, component transparency, component display position, etc.

[0133] In one example, when the voice material corresponds to a negative emotion, the component display style includes using dark colors, a large size, and fast-paced dynamic effects. In another example, when the voice material corresponds to a positive emotion, the component display style includes using bright colors, a large size, fast-paced dynamic effects, and decorative patterns. In yet another example, when the voice material corresponds to a neutral emotion, the component display style includes using a small size, slow-paced dynamic effects, and high transparency.

[0134] In some embodiments, the first interactive component corresponds to the audio material of the second audio content. Illustratively, the display duration of the first interactive component is associated with the audio duration corresponding to the audio material; for example, the display duration of the first interactive component is the same as the audio duration corresponding to the audio material, or, for another example, the display duration of the first interactive component is half the audio duration corresponding to the audio material.

[0135] In one example, as shown in Figure 7, a schematic diagram of a first interactive component provided in an exemplary embodiment of this application is illustrated. The video playback interface 700 displayed on the terminal shows a first video 710 playing at a first playback volume. When the first video 710 plays to the video content corresponding to the target audio material, the first interactive component 711 corresponding to the target audio material is displayed. For example, the audio material corresponding to the video content is a character 712 crying and calling for his mother. The first interactive component 711 is displayed around the character 712 (taking an interactive component with a button pattern as an example). When the first interactive component 711 receives a trigger operation, a second subtitle 713 is displayed. The subtitle display style of the displayed second subtitle 713 is displayed in the manner of simulating tears, thereby expressing the sad emotion of the character 712 in this audio material.

[0136] That is, by displaying the first interactive component to guide the user's interaction with the first video, the diversity of the second subtitle display is enhanced. At the same time, the first interactive component is displayed using a component display style that is related to the voice emotion, so that the user can understand the emotional information of the video scene corresponding to the current voice material through the component display style, thereby enhancing the diversity of information transmission.

[0137] In summary, during the playback of the first video at the first playback volume, when the first playback volume of the first video is lower than the first condition, the transmission of audio-related information is compensated by displaying a second subtitle. The second subtitle is different from the first subtitle that is normally displayed in the video. That is, in scenarios where the first playback volume cannot meet the user's perception needs of the audio of the first video, the second subtitle compensates for the transmission of audio information, thereby improving the information transmission efficiency of the first video and enhancing the user experience.

[0138] When recommending content to users through the first video, a second subtitle, distinct from the first subtitle, is used to assist in conveying information from the audio content in the first video. On the one hand, the second subtitle improves the information delivery efficiency of the first video in low-volume scenarios, enhancing the user experience; on the other hand, it can also increase user attention to the first video, thereby improving the recommendation conversion rate when recommending content through the first video; furthermore, by automatically detecting the playback volume of the first video and dynamically triggering the display of subtitles with different visual effects, the automation level of the first video playback is improved, enhancing the flexibility of content display in different volume scenarios.

[0139] Example 2: In a content recommendation scenario, a subtitle function is provided to the user during the playback of the first video used for content recommendation. The subtitle function enables the subtitle content to express the emotions of the audio material in the video.

[0140] Please refer to Figure 8, which shows a flowchart of a method for displaying subtitle content provided in an exemplary embodiment of this application. The method is performed by a computer device. For example, the method is performed by a terminal. The method includes the following steps 810 to 840.

[0141] Step 810: Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content.

[0142] The video content of the first video includes at least one of image content, subtitle content, and audio content. In some embodiments, the first video is a video containing only one of these contents; for example, the first video may be a video containing only image content, a video containing only subtitle content, or a video containing only audio content. In some embodiments, the first video is a video containing multiple of these contents; for example, the first video may be a video containing only image content and subtitle content, a video containing both image content and audio content, a video containing both subtitle content and audio content, or a video containing all three.

[0143] In this embodiment of the application, the first video includes audio material, wherein the audio material is a component of the audio content of the first video. In some embodiments, the audio content of the first video includes background music, dialogue, voice-over, background music, sound effects, ambient sounds, etc., and the audio material includes at least one of the above-mentioned audio contents.

[0144] In this embodiment of the application, the first video corresponds to a first subtitle. During the playback of the first video, the terminal displays the first subtitle corresponding to the audio material of the currently playing first video content.

[0145] In some embodiments, the first subtitle is a subtitle for at least one of the following: dialogue between characters, narration, and background music in the first video. In one example, taking a short drama video as an example, the first subtitle is the dialogue between characters in the short drama video.

[0146] Step 820: Display the second subtitle corresponding to the audio material of the currently playing second video content. The display styles of the first subtitle and the second subtitle are different.

[0147] In this embodiment of the application, the first video is used to recommend first recommended content. In one example, the first video can be implemented as an advertising video.

[0148] Optionally, the number of first recommended content items can be one or more.

[0149] Optionally, when the first video is used to recommend the first recommended content, the playback method of the first video can be implemented as at least one of the following:

[0150] The first method involves playing the first video when the running state of the first application meets the first content recommendation criteria.

[0151] To illustrate, the first application is run; if the running state of the first application meets the first content recommendation criteria, the first video is played.

[0152] Optionally, the first application includes social applications, audio and video applications, game applications, browsers, web applications, etc., without limitation.

[0153] The running status of the first application is used to indicate the stage the first application is in during its operation. Optionally, the running status of the first application includes startup status, running status, background running status, crash status, suspended status, closed status, update status, idle status, etc.

[0154] Optionally, in response to receiving an application launch operation for the first application, the first application is launched, and while the first application is in the launched state, the first video is played. That is, the first video is a content recommendation video provided by the first application during the launch phase; for example, the first video is implemented as the splash screen advertisement video of the first application.

[0155] Optionally, the first video is played when the first application is running and its runtime reaches a first duration threshold. That is, the first video is a recommended video provided by the first application during its runtime.

[0156] Optionally, when the first application is running and an operation to open the target application content within the first application is received, the first video is played. That is, the first video is a recommended video inserted when the first application starts the target application content. Optionally, the target application content includes at least one of the following: text content, video content, audio content, and application functions within the first application.

[0157] Optionally, in response to receiving a running state switching operation for the first application, a first video is played. That is, the first video is a content recommendation video provided by the first application during the application state switching phase. The aforementioned running state switching operation includes at least one operation for switching from a closed state to a running state, from a background running state to a running state, and from a suspended state to a running state.

[0158] In one example, as shown in FIG9, a schematic diagram of the playback trigger of the first video provided by an exemplary embodiment of the present application is illustrated. The desktop interface 900 displayed by the terminal includes an application icon 910 corresponding to the first application. When the application icon 910 corresponding to the first application receives a trigger operation, the terminal launches the first application and displays the application launch interface 920 corresponding to the first application. The first video 930 is displayed as a splash screen advertisement on the application launch interface 920.

[0159] In other words, by enabling the automatic playback of the first video through the running status of the first application, the dissemination efficiency of the first video is improved, and the intelligence level of the first video recommendation is also improved.

[0160] The second method involves playing the first video if the playback status of the second video meets the second content recommendation criteria.

[0161] As an example, play the second video; in response to the second video's playback status meeting the second content recommendation criteria, play the first video.

[0162] In some embodiments, the second video is video content provided by the first application currently running on the terminal. For example, the second video is a video published by a publisher account. The publisher account can be a user account logged in on the terminal device or a user account logged in on another terminal device.

[0163] Optionally, the playback status of the second video includes a waiting state, a playing state, a paused state, a playback ended state, a buffering state, a picture-in-picture playback state, a full-screen playback state, etc.

[0164] Optionally, in response to receiving a pause operation for the second video, the first video is played while the second video is paused. That is, the first video is played during the pause of the second video for content recommendation.

[0165] Optionally, the first video is played in response to the second video's playback duration reaching a second duration threshold. That is, when the second video's playback duration reaches the second duration threshold, the first video is played for content recommendation. For example, taking a TV series as the second video, the first video is played after the opening theme of the current episode (which has a fixed duration threshold) has finished playing.

[0166] Optionally, in response to a playback state switch in the second video, the first video is played. That is, the first video is a content recommendation video provided during the playback state switch phase of the first video. The aforementioned playback state switch includes at least one of the following: switching from a playing state to a paused state, switching from a playing state to a buffering state, switching from a playing state to an ended state, and switching from a picture-in-picture playback state to a full-screen playback state.

[0167] In one example, as shown in FIG10, it illustrates a schematic diagram of the playback triggering of a first video provided by an exemplary embodiment of the present application. During the playback of the second video 1010, the terminal receives a pause operation for the second video 1010, the second video 1010 enters a paused state, and the first video 1020 is displayed on the display interface of the second video 1010 through a pop-up window.

[0168] In some embodiments, when the first video is used to recommend first recommended content, the second subtitle is displayed in a first subtitle display style. The first subtitle display style is different when the audio material of the second video content corresponds to different audio emotions. That is, the first subtitle display style of the second subtitle is associated with the audio emotion of the audio material.

[0169] Optionally, the aforementioned first subtitle display style includes subtitle font style, subtitle color style, subtitle animation effect, subtitle display duration, subtitle language type, subtitle size, subtitle transparency, subtitle display position, etc.

[0170] Optionally, the aforementioned vocal emotions include positive emotions (e.g., happiness, joy, excitement, etc.), neutral emotions (e.g., surprise, curiosity, relaxation, arrogance, sympathy, etc.), and negative emotions (e.g., sadness, anger, fear, disgust, frustration, etc.).

[0171] In some embodiments, the process of displaying the second subtitle of the first video is implemented as follows: acquiring the audio material corresponding to the content of the second video; converting the audio material into audio text; recognizing the semantics of the audio text to obtain the emotion recognition result corresponding to the audio text; and displaying the second subtitle in the first subtitle display style based on the emotion recognition result.

[0172] In one example, when the voice material corresponds to a negative emotion, the first caption display style includes using dark colors, bold fonts, and fast-paced dynamic effects. In another example, when the voice material corresponds to a positive emotion, the first caption display style includes using bright colors, large fonts, fast-paced dynamic effects, and decorative patterns. In yet another example, when the voice material corresponds to a neutral emotion, the first caption display style includes using small fonts, slow-paced dynamic effects, and high transparency.

[0173] Optionally, the second subtitle may be generated and rendered in real time during the playback of the first video; or, the second subtitle may be pre-generated by the server based on the first video.

[0174] Optionally, the second subtitle is displayed at a fixed position in the first video, for example, at the top or bottom of the first video; or, the display position of the second subtitle is associated with the first voice object corresponding to the voice material, wherein the voice emotion of the voice material matches the emotional state of the first voice object.

[0175] Optionally, the first voice object includes people, animals, plants, scene objects, etc. in the first video.

[0176] Schematic illustration: A second subtitle is displayed at a first position associated with the first audio object. Optionally, the first position includes the object area or surrounding area corresponding to the first audio object; for example, the second subtitle is displayed above the head or face of the person expressing the emotional tone of the audio material. That is, displaying the second subtitle at the first position associated with the first audio object enhances the visual association between the first audio object and the second subtitle, thereby improving the display effect of the second subtitle and increasing the amount of information conveyed by the second subtitle.

[0177] In some optional embodiments, a first interactive component associated with the second subtitle is displayed, wherein the component display style of the first interactive component is associated with the emotional tone of the voice material; in response to receiving a trigger operation on the first interactive component, the second subtitle corresponding to the voice material is displayed.

[0178] Optionally, the above component display styles include component color style, component dynamic effects, component display duration, component size, component transparency, component display position, etc.

[0179] In one example, when the voice material corresponds to a negative emotion, the component display style includes using dark colors, a large size, and fast-paced dynamic effects. In another example, when the voice material corresponds to a positive emotion, the component display style includes using bright colors, a large size, fast-paced dynamic effects, and decorative patterns. In yet another example, when the voice material corresponds to a neutral emotion, the component display style includes using a small size, slow-paced dynamic effects, and high transparency.

[0180] In some embodiments, the display duration of the first interactive component is associated with the audio duration corresponding to the audio material. For example, the display duration of the first interactive component is the same as the audio duration corresponding to the audio material, or the display duration of the first interactive component is half of the audio duration corresponding to the audio material.

[0181] That is, by displaying the first interactive component to guide the user's interaction with the first video, the diversity of the second subtitle display is enhanced. At the same time, the first interactive component is displayed using a component display style that is related to the voice emotion, so that the user can understand the emotional information of the video scene corresponding to the current voice material through the component display style, thereby enhancing the diversity of information transmission.

[0182] Step 830: Receive interactive operations on the first video.

[0183] Optionally, the interactive operations on the first video may include at least one of the following:

[0184] The first method involves receiving control-triggered operations performed by the user on the screen displaying the first video.

[0185] As an illustration, when the terminal is a device with a touch screen, the aforementioned interactive operation on the first video can be determined by the user touching the interactive control corresponding to the displayed first video on the touch screen. Optionally, the aforementioned control triggering operation can be implemented as at least one of the following: single-click operation, double-click operation, continuous press operation, heavy press operation, drag operation, check operation, etc.

[0186] The second method is to receive shortcut key operation signals. When the shortcut key corresponding to the shortcut key operation signal is the target shortcut key bound to the interactive operation of the first video, it is determined that the interactive operation on the first video has been received.

[0187] As an illustration, when the terminal is a desktop computer, laptop, or game console, the shortcut key operation signal can be triggered by input operations from an external input device, such as by clicking the right mouse button; when the receiving end is a mobile terminal such as a mobile phone or tablet, the shortcut key operation signal can be triggered by triggering a physical button on the mobile terminal, or by triggering an auxiliary input device connected to the mobile terminal.

[0188] The third method is to receive user voice. When the voice recognition result corresponding to the user's voice indicates that an interactive action on the first video has been performed, it is determined that an interactive action on the first video has been received.

[0189] Schematic illustration: When the terminal includes a voice acquisition component, or when the terminal is connected to a voice acquisition device, the aforementioned interactive operation on the first video can be triggered by user voice input through the voice acquisition component or the voice acquisition device. In some embodiments, the terminal performs voice recognition on the received user voice, and when the recognition result indicates that the user voice contains a voice keyword matching the interactive action on the first video, it determines that an interactive operation on the first video has been received.

[0190] In one example, if the terminal receives the user's voice message "jump", it is determined that an interactive operation on the first video has been received.

[0191] In some embodiments, the interactive operation is triggered by a second interactive component associated with the audio material. Illustratively, during the playback of the second video content, the second interactive component associated with the second subtitle is displayed, wherein the component display style of the second interactive component is associated with the emotional expression of the audio material; and a trigger operation on the second interactive component is received as an interactive operation. That is, by dynamically triggering the second subtitle through the second interactive component associated with the audio material, the user's sense of participation during the playback of the first video is enhanced, and the interactive functions provided by the second interactive component enhance the emotional expression of the audio content represented by the first video, thereby increasing the user's immersion.

[0192] Optionally, the above component display styles include component color style, component dynamic effects, component display duration, component size, component transparency, component display position, etc.

[0193] In some embodiments, the display duration of the second interactive component is associated with the audio duration corresponding to the audio material. For example, the display duration of the second interactive component is the same as the audio duration corresponding to the audio material, or the display duration of the second interactive component is half of the audio duration corresponding to the audio material.

[0194] In some embodiments, when the first video is used to recommend at least two first recommended content items, different second interactive components are provided for different first recommended content items. Illustratively, in response to the first video playing to the second video content associated with the i-th first recommended content, the second interactive component corresponding to the i-th first recommended content is displayed, and an interaction operation is received on the second interactive component corresponding to the i-th first recommended content, where i is a positive integer. That is, when the first video is used to recommend multiple recommended content items, by providing relevant second interactive components when playing video content related to the recommended content, targeted second subtitles are displayed based on user interests. This eliminates the need to generate second subtitles for all audio material in the first video, reducing the cost of generating second subtitles while increasing user interest in the recommended content.

[0195] Step 840: In response to the interactive operation, display the recommendation information corresponding to the first recommended content.

[0196] Optionally, the recommendation information corresponding to the first recommended content includes, but is not limited to, images related to the first recommended content, videos related to the first recommended content, text describing the first recommended content, links related to the first recommended content, and interfaces related to the first recommended content.

[0197] Optionally, the link related to the first recommended content can be a response link of the first recommended content within the current application, or a response link of the first recommended content in other applications; alternatively, the interface related to the first recommended content can be a response interface of the first recommended content within the current application, or a response interface of the first recommended content in other applications.

[0198] In some embodiments, when the first recommended content is implemented as a first product, the recommended information corresponding to the first recommended content includes product introduction information, purchase link information, merchant introduction information, product review information, historical sales information, and product store information. Optionally, the first product can be a commodity, such as clothing, furniture, appliances, food, virtual items, etc.; the first product can also be a service, such as housekeeping services, massage services, errand services, etc.; the first product can also be a store address, such as a restaurant address, supermarket address, clothing store address, etc. For example, if the first video is a video used to recommend clothing of brand XX, then in response to the interactive operation, the user is redirected to the purchase page corresponding to clothing of brand XX.

[0199] In other embodiments, when the first recommended content is a third video, the recommendation information corresponding to the first recommended content includes the publisher information of the third video, video content information, video length information, the video set to which the video belongs (e.g., episodes of a TV series), video comment information, etc. Optionally, the third video can be a film or television work, such as a TV series, movie, short drama, music video (MV), variety show, etc.; the third video can be a live broadcast, such as a sports live broadcast, game live broadcast, chat live broadcast, entertainment live broadcast, news live broadcast, etc.; the third video can be user-generated content (UGC) or professional user-generated content (PUGC), such as social media videos, short videos, etc.

[0200] In some embodiments, when the first video is used to recommend at least two first recommended contents, different second interactive components are provided for different first recommended contents. When the second interactive component corresponding to different first recommended contents receives a trigger operation, it jumps to the recommendation information corresponding to the corresponding first recommended content. Illustratively, in response to the first video playing to the video content corresponding to the audio material associated with the i-th first recommended content, the second interactive component corresponding to the i-th first recommended content is displayed, where i is a positive integer; a trigger operation on the second interactive component corresponding to the i-th first recommended content is received as an interactive operation; in response to the interactive operation, the recommendation information corresponding to the i-th first recommended content is displayed.

[0201] In one example, the first video is used to recommend multiple products. The first video includes multiple audio materials corresponding to each product. When the audio material corresponding to the i-th product is played, the second interactive component corresponding to the i-th product is displayed. When the second interactive component corresponding to the i-th product receives a trigger operation, it jumps to the product details page corresponding to the i-th product.

[0202] That is, for multiple first recommended content items within the first video, a corresponding second interactive component is displayed based on the voice emotion expressed by the corresponding voice material of the first recommended content. In this way, the second interactive component provides a jump function for the recommendation information of the first recommended content, thereby improving the diversity of recommendation information reach.

[0203] In summary, when recommending content to users through the first video, using a second subtitle, distinct from the first subtitle, to assist in conveying information from the audio content of the first video can improve the information delivery efficiency of the first video, enhance the user experience, and increase user attention to the first video, thereby improving the recommendation conversion rate when recommending content through the first video.

[0204] When recommending content to users through the first video, a second subtitle, distinct from the first subtitle, is used to assist in conveying information from the audio content in the first video. On the one hand, the second subtitle improves the information delivery efficiency of the first video in low-volume scenarios, enhancing the user experience; on the other hand, it can also increase user attention to the first video, thereby improving the recommendation conversion rate when recommending content through the first video; furthermore, by automatically detecting the playback volume of the first video and dynamically triggering the display of subtitles with different visual effects, the automation level of the first video playback is improved, enhancing the flexibility of content display in different volume scenarios.

[0205] In some optional embodiments, the subtitle content corresponding to the audio material is displayed using generated subtitle data. For illustration, please refer to Figure 11, which shows a flowchart of a subtitle data generation method provided in an exemplary embodiment of this application. It is worth noting that this method can be executed by a terminal or a server. In this embodiment, the method is illustrated using a server as an example. This method can be applied to the subtitle content display methods provided in Embodiments 1 and 2. The method includes steps 1110 to 1130.

[0206] Step 1110: Obtain the audio text corresponding to the audio material of the second video content.

[0207] In this embodiment of the application, the first video includes audio material, wherein the audio material is a component of the audio content of the first video. In some embodiments, the audio content of the first video includes background music, dialogue, voice-over, background music, sound effects, ambient sounds, etc., and the audio material includes at least one of the above-mentioned audio contents.

[0208] In some embodiments, the method for obtaining the voice text corresponding to the voice material is as follows: obtaining video data corresponding to the first video; separating audio data from the video data; and converting the audio data into voice text through speech recognition.

[0209] Optionally, the video data of the first video can be uploaded to the server by the terminal, or it can be obtained by the server from the database.

[0210] In one example, separating audio data from video data can be achieved by using a video decoding library (e.g., Fast Forward Moving Picture Experts Group, FFmpeg) to decode the video data into frame image data and audio data.

[0211] In one example, converting audio data into speech text via speech recognition can be achieved by using speech recognition technology to convert audio data into speech text. Optionally, the aforementioned speech recognition technology includes Automatic Speech Recognition (ASR), Transformer-based ASR models, Attention-based Encoder-Decoder (AED), Wav2vec, Hidden Markov Models (HMMs), etc.

[0212] In some embodiments, the audio data processing operations further include: segmenting the audio data based on the start of dialogue content in the audio data to obtain multiple audio segments, and recording the start timestamps and end timestamps corresponding to each of the multiple audio segments. The start timestamps and end timestamps are used to control the start and end times of the display of the corresponding second subtitle in the first video.

[0213] In some embodiments, after acquiring the speech text, content cleaning is performed on the speech text to ensure the accuracy of the text. Optionally, content cleaning includes operations such as removing noise, repetitions, and errors.

[0214] In some embodiments, to reduce data processing volume, the audio material to be used for subtitle generation is determined by the importance of candidate audio objects in the first video. Illustratively, the audio of each of at least two candidate audio objects in the second video content is obtained, and the object importance of each of the at least two candidate audio objects in the first video is obtained; based on the object importance of the candidate audio objects, at least one first audio object is determined from the at least two candidate audio objects; and the audio of the object corresponding to the at least one first audio object is obtained as the audio material of the second video content.

[0215] In this embodiment, audio materials are filtered by the object importance of candidate audio objects in the first video, thereby prioritizing the selection of audio materials of key objects in the plot to create second subtitles. This highlights the key information expressed by the audio while reducing the generation cost of the second subtitles and improving their display efficiency.

[0216] Optionally, the importance of the aforementioned candidate speech object in the first video can be determined based on at least one of the following information: the duration of appearance of the candidate speech object in the first video, the number of lines, the number of times the shot is focused, the object type, and historical feedback.

[0217] Optionally, the importance of a candidate speech object is determined based on its appearance duration in the first video, where the appearance duration is positively correlated with the object importance. In one example, the appearance duration of a candidate speech object in the first video is used as its object importance, and candidate speech objects with an appearance duration greater than a third duration threshold are identified as the first speech object. Alternatively, at least two candidate speech objects are sorted according to their appearance duration in the first video to obtain an object list, where the order of the candidate speech objects in the object list is positively correlated with their appearance duration, and the top N candidate speech objects in the object list are identified as the first speech object, where N is a positive integer.

[0218] Optionally, the importance of a candidate audio object is determined based on the number of lines it receives in the first video, where the number of lines received by a candidate audio object is positively correlated with its importance. In one example, the number of lines received by a candidate audio object in the first video is used as its importance, and candidate audio objects with a number of lines greater than a first threshold are identified as the first audio object. Alternatively, at least two candidate audio objects are sorted according to the number of lines received in the first video to obtain an object list, where the order of the candidate audio objects in the object list is positively correlated with the number of lines received, and the top N candidate audio objects in the object list are identified as the first audio object.

[0219] Optionally, the importance of a candidate speech object is determined based on the number of times it is focused on in the first video, where the number of times it is focused on is positively correlated with the object importance. In one example, the number of times a candidate speech object is focused on in the first video is used as the object importance, and candidate speech objects with a number of times they are focused on more than a second threshold are identified as the first speech object; or, at least two candidate speech objects are sorted according to the number of times they are focused on in the first video to obtain an object list, where the order of the candidate speech objects in the object list is positively correlated with the number of times they are focused on, and the top N candidate speech objects in the object list are identified as the first speech object.

[0220] Optionally, the importance of a candidate audio object is determined based on its object type in the first video. Optionally, the object types include protagonist type, supporting character type, narrator type, host type, guest type, etc. In one example, when the object type of a candidate audio object is protagonist type, the candidate audio object is determined as the first audio object; or, when the object type of a candidate audio object is guest type, the candidate audio object is determined as the first audio object.

[0221] Optionally, the importance of candidate audio objects is determined based on information such as historical feedback of the first video, wherein the historical feedback data indicates the feedback received by the first video within a historical time period. Optionally, the aforementioned historical feedback information includes comments, bullet comments, explanations, evaluations, etc., corresponding to the first video. Illustratively, historical feedback data of the first video is obtained; based on the correlation between at least two candidate audio objects and the historical feedback data, the importance of at least two candidate audio objects is determined.

[0222] Optionally, the correlation between the aforementioned candidate speech objects and historical feedback data can be determined by information such as the frequency of occurrence of the candidate speech objects in historical feedback data, positive feedback rate, and confidence level.

[0223] In one example, the term frequency (TF-IDF) and inverse document frequency (TRF) of candidate speech objects in historical feedback data are determined. Candidate speech objects with TF-IDF values ​​higher than a first frequency threshold are identified as the first speech object. Alternatively, at least two candidate speech objects are sorted according to their corresponding TF-IDF values ​​to obtain a list of objects. The order of candidate speech objects in the list is positively correlated with the magnitude of their TF-IDF values. The top N candidate speech objects in the list are identified as the first speech object.

[0224] In another example, the object information and historical feedback data of candidate speech objects are input into a correlation evaluation model to obtain the correlation degree between the candidate speech objects and the historical feedback data. This correlation degree is used as the object importance of the candidate speech object. The candidate speech object whose object importance reaches a specified importance threshold among at least two candidate speech objects is determined as the first speech object. Alternatively, at least two candidate speech objects are sorted according to their object importance to obtain an object list. The order of candidate speech objects in the object list is positively correlated with word frequency and object importance. The top N candidate speech objects in the object list are determined as the first speech object.

[0225] The aforementioned correlation evaluation model is a pre-trained machine learning model used to analyze the object importance of candidate speech objects in the first video based on their object information and historical feedback data. Optionally, the object information of the candidate speech objects includes object name, object type, object dialogue, object background information, etc.

[0226] Alternatively, the correlation evaluation model can be implemented using neural network models such as Convolutional Neural Networks (CNN), Feedforward Neural Networks (FNN), Residual Networks (ResNet), and Transformer, without any specific limitations.

[0227] In some embodiments, the training process of the correlation evaluation model is implemented as follows: obtaining sample object information corresponding to sample speech objects in sample videos, and obtaining sample feedback data corresponding to sample videos. Each sample speech object has a corresponding sample label, which is used to indicate the importance of the sample speech object in the sample video. The sample object information and sample feedback data are input into the correlation evaluation model to be trained to obtain the predicted importance. A first loss value between the predicted importance and the sample label is determined through a first loss function. The model parameters of the correlation evaluation model to be trained are iteratively adjusted according to the first loss value until convergence is achieved, thus obtaining the correlation evaluation model.

[0228] Optionally, the first loss function mentioned above can be implemented as at least one of the following: cross-entropy loss function, mean squared error loss (MSE) function, log loss function, least absolute deviations loss (L1 loss) function, etc., without limitation.

[0229] In this embodiment, the importance of the candidate audio object is determined by the historical feedback data generated after the first video is played. This allows for the selection of audio materials for generating the second subtitle by combining the audience's experience, thereby improving the accuracy of the selected audio materials and thus enhancing the display effect of the generated second subtitle.

[0230] Step 1120: Identify the semantics of the speech text to obtain the emotion recognition result corresponding to the speech text.

[0231] Optionally, the emotion recognition result includes the emotion category corresponding to the speech text, or the emotion score corresponding to the speech text under multiple candidate emotion categories.

[0232] Optionally, the aforementioned emotion categories include positive categories (e.g., happiness, joy, excitement, etc.), neutral categories (e.g., surprise, curiosity, relaxation, arrogance, sympathy, etc.), and negative categories (e.g., sadness, anger, fear, disgust, frustration, etc.). Alternatively, emotion categories may also include more granular categories such as pleasure, sadness, anger, curiosity, surprise, and calmness.

[0233] In some embodiments, emotion recognition of speech text is achieved through a pre-trained emotion recognition model. Illustratively, speech text is input into the emotion recognition model, which identifies the corresponding emotion category of the speech text to obtain the emotion recognition result. Here, the emotion recognition model is a pre-trained machine learning model used to analyze the emotions expressed in the speech text.

[0234] Alternatively, the emotion recognition model can be implemented using neural network models such as CNN, FNN, ResNet, and Transformer, without specific limitations.

[0235] In some embodiments, the training process of the emotion recognition model is implemented as follows: obtaining sample text and sample labels corresponding to the sample text, the sample labels being used to indicate the emotion recognition result corresponding to the sample text; inputting the sample text into the emotion recognition model to be trained to obtain the predicted recognition result; determining the second loss value between the predicted recognition result and the sample label through a second loss function; iteratively adjusting the model parameters of the emotion recognition model to be trained according to the second loss value until convergence, thereby obtaining the emotion recognition model.

[0236] Optionally, the second loss function mentioned above can be implemented as at least one of the following: cross-entropy loss function, mean squared error loss function, log loss function, minimum absolute deviation loss function, etc., without limitation.

[0237] In other embodiments, the emotion recognition model can also be implemented as a multimodal model, wherein the input of the emotion recognition model includes the speech text and speech audio corresponding to the speech material, so that the emotion recognition model can combine the semantic features corresponding to the speech text and the intonation features corresponding to the speech audio to output the emotion recognition result, thereby improving the recognition accuracy of the emotion recognition result.

[0238] In this embodiment, the emotion recognition model obtained through pre-training is used to realize emotion recognition for speech text, which can improve the subtitle display efficiency when displaying the second subtitle during the first video playback. At the same time, it enables the system for displaying subtitle content to be reused in diverse video playback scenarios, thereby improving the reusability of the system.

[0239] Step 1130: Generate subtitle data based on emotion recognition results.

[0240] In some embodiments, a matching subtitle display style is obtained based on the emotion recognition result, and subtitle data is generated based on the subtitle display style and the voice text.

[0241] Optionally, the above-mentioned subtitle display styles include subtitle font style, subtitle color style, subtitle animation effects, subtitle display duration, subtitle language type, subtitle size, subtitle transparency, and subtitle display position.

[0242] In one example, when the emotion recognition result indicates that the corresponding voice material corresponds to a negative emotion, the caption display style includes using dark colors, bold fonts, and fast-paced dynamic effects. In another example, when the emotion recognition result indicates that the corresponding voice material corresponds to a positive emotion, the caption display style includes using bright colors, large fonts, fast-paced dynamic effects, and decorative patterns. In yet another example, when the emotion recognition result indicates that the corresponding voice material corresponds to a neutral emotion, the caption display style includes using small fonts, slow-paced dynamic effects, and high transparency.

[0243] For example, as shown in FIG12, which illustrates a schematic diagram of a subtitle display style provided in an exemplary embodiment of this application, in the video frame 1210 corresponding to voice material A, character 1211 is crying because he cannot find his mother, and the crying is visualized by the second subtitle 1212; in the video frame 1220 corresponding to voice material B, character 1221 has an impatient emotion, and the corresponding voice material B is used to order other characters, so the tone, intonation and semantics of voice material B are visualized by the second subtitle 1222; in the video frame 1230 corresponding to voice material C, character 1231 has a feeling of longing because he cannot find anyone, so the emotional expression of voice material C is visualized by the second subtitle 1232.

[0244] In some embodiments, a first mapping table between emotion recognition results and subtitle display styles is pre-configured. Schematic, the first mapping table is obtained, the corresponding subtitle display style is retrieved from the first mapping table based on the emotion recognition results, and corresponding subtitle data is generated based on the obtained subtitle display style and the voice text.

[0245] In some embodiments, an artificial intelligence model is used to generate subtitle data. Illustratively, emotion recognition results and spoken text are input into the subtitle generation model to obtain subtitle data. The subtitle generation model is a pre-trained machine learning model used to combine emotion recognition results and spoken text to determine the subtitle display style and generate corresponding subtitle data.

[0246] Optionally, the subtitle generation model can be implemented using neural network models such as CNN, FNN, ResNet, and Transformer, without specific limitations.

[0247] In some embodiments, at least one keyword is extracted from the speech text; based on the emotion recognition result and at least one keyword, subtitle data corresponding to the speech material of the second video content is generated; and the second subtitle is displayed in a first subtitle display style based on the subtitle data.

[0248] In this embodiment, by extracting keywords from the speech text for use in generating the second subtitle, the second subtitle can combine the emotion recognition results to highlight the core content that the speech material needs to express, thereby improving the information transmission efficiency of the displayed second subtitle. At the same time, it can also highlight the highlights of the video content through concise keywords, attract the user's attention, and thus enhance the attractiveness of the first video.

[0249] In some embodiments, when generating subtitle data, only subtitle data corresponding to keywords is generated, that is, subtitle data corresponding to at least one keyword is generated based on the emotion recognition result. In other words, keyword extraction is used to reduce the amount of subtitle data generated, thereby improving the display efficiency of the second subtitle.

[0250] In other embodiments, keywords are used to guide the subtitle generation model in the process of generating subtitle data. That is, the input of the subtitle generation model also includes keywords extracted from the speech text, thereby further guiding the generation of subtitle data through keywords. Illustratively, the emotion recognition result, at least one keyword, and speech text are input into the subtitle generation model to obtain subtitle data. The subtitle generation model is a pre-trained machine learning model used to combine the emotion recognition result and keywords to determine the subtitle display style and generate the corresponding subtitle data.

[0251] In some embodiments, the training process of the subtitle generation model is as follows: obtain the emotion recognition result, sample speech text, and sample keywords extracted from the sample speech text corresponding to the sample video; input the emotion recognition result, sample speech text, and sample keywords into the subtitle generation model to be trained to obtain predicted subtitle data; determine the third loss value corresponding to the predicted subtitle data through the third loss function; iteratively adjust the model parameters of the subtitle generation model to be trained according to the third loss value to achieve self-supervised learning until convergence, and obtain the subtitle generation model.

[0252] Optionally, the third loss function mentioned above can be implemented as at least one of the following: cross-entropy loss function, mean squared error loss function, log loss function, minimum absolute deviation loss function, etc., without limitation.

[0253] In this embodiment of the application, the subtitle generation model obtained through pre-training is used to realize the automatic generation of subtitle data, which can improve the subtitle display efficiency of displaying the second subtitle when the first video is played and reduce the manual cost of subtitle generation.

[0254] In some embodiments, the keyword extraction process is implemented as follows: segmenting the speech text to obtain at least two candidate words; determining the word frequency and inverse document frequency corresponding to the at least two candidate words respectively; determining the word importance corresponding to the candidate words based on the word frequency and the inverse document frequency; and determining at least one keyword from the at least two candidate words based on the word importance corresponding to the at least two candidate words respectively.

[0255] In some embodiments, the word frequency and inverse document frequency corresponding to the candidate words are used as the word importance of the candidate words. Illustratively, at least two candidate words are sorted according to their word importance to obtain a word list, and the top M candidate words in the word list are determined as keywords, where M is a positive integer, and the order of the candidate words in the word list is positively correlated with their word importance; alternatively, candidate words with word importance higher than a specified importance threshold are determined as keywords.

[0256] In this embodiment of the application, the importance of candidate words in the speech text is characterized by the word frequency and inverse document frequency of the candidate words. This can evaluate the importance of candidate words from the dual perspectives of word frequency and prevalence, thereby ensuring the accuracy of keyword selection.

[0257] In some embodiments, caption data includes caption display style, voice text (or keywords), and start and end times displayed in the first video.

[0258] In some embodiments, the server sends the generated subtitle data to a terminal that needs to play the first video. The terminal plays the first video using the video data of the first video, and during the playback of the first video, when playing the video content corresponding to the audio material, it displays the second subtitle based on the subtitle data and in the first subtitle display style.

[0259] Specifically, the implementation of displaying the second subtitle in the first subtitle display style based on subtitle data is as follows: After obtaining the subtitle data, the subtitle data is parsed to extract subtitle information such as the first subtitle display style, voice text, and the start and end times displayed in the first video. The subtitle display logic is set according to the indicated start and end times and the first subtitle display style. The second subtitle displayed in the first subtitle display style is rendered and displayed in the playback window of the first video. The subtitle content of the second subtitle is updated in real time according to the playback progress of the first video.

[0260] In other embodiments, the server integrates the generated subtitle data with the video data of the first video to obtain subtitled video data, and sends the subtitled video data to the terminal that needs to play the first video. The terminal plays the first video based on the subtitled video data, and displays the second subtitle corresponding to the audio material when playing the video content corresponding to the audio material.

[0261] In some embodiments, first component data corresponding to the first interactive component corresponding to the voice material is generated based on the emotion recognition result. The component display style of the first interactive component is related to the voice emotion of the voice material, and the first interactive component is used to trigger the display of the bullet screen content corresponding to the voice material.

[0262] In some embodiments, a second mapping table between the first interactive component and the component display style is pre-configured. Schematic, the second mapping table is obtained, the corresponding component display style is retrieved from the second mapping table based on the emotion recognition result, and the corresponding first component data is generated based on the obtained component display style.

[0263] In some embodiments, the first component data includes the component type corresponding to the first interactive component, the component display style, and the start and end times displayed in the first video.

[0264] In one example, taking the process of generating this subtitle data as an application in an advertising scenario, as shown in Figure 13, the process of displaying the advertising video in the advertising scenario includes: 1301, the advertiser uploads the advertising video to the server; 1302, the server extracts text from the advertising video to obtain speech text; 1303, the server performs semantic analysis on the speech text to obtain sentiment recognition results; 1304, the server generates subtitle data based on the sentiment recognition results; 1305, the server sends the subtitle data and the advertising video to the terminal, and the terminal plays the advertising video in combination with the subtitle data.

[0265] In summary, by analyzing the emotional content of the audio material in the first video, corresponding subtitle data is generated based on the obtained emotion recognition results. This enables the terminal to display the second subtitle in a subtitle display style associated with the emotional content of the audio material. In other words, the second subtitle can visually convey the emotions expressed by the audio material in the first video, thereby improving the information transmission efficiency of the first video. Users do not need to adjust the volume to perceive the emotions expressed by the audio material, reducing user operations and improving user experience.

[0266] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0267] Please refer to Figure 14, which shows a structural block diagram of a caption content display device provided in an exemplary embodiment of this application. The device includes the following modules:

[0268] The first display module 1410 is used to play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content;

[0269] The first determining module 1420 is used to determine the first playback volume of the first video;

[0270] The first display module 1410 is further configured to display a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition, wherein the first subtitle and the second subtitle have different subtitle display styles.

[0271] In some optional embodiments, the first display module 1410 is further configured to display the second subtitle in a first subtitle display style when the first playback volume is lower than the first condition, wherein the first subtitle display style is different when the voice material of the second video content corresponds to different voice emotions.

[0272] In some optional embodiments, as shown in FIG15, the apparatus further includes a first generation module 1430, the first generation module 1430 comprising:

[0273] The first acquisition unit 1431 is used to acquire the voice text corresponding to the voice material of the second video content;

[0274] The first recognition unit 1432 is used to recognize the semantics of the speech text and obtain the emotion recognition result corresponding to the speech text;

[0275] The first display module 1410 is also used to display the second subtitle in the first subtitle display style based on the emotion recognition result.

[0276] In some optional embodiments, the first recognition unit 1432 is further configured to input the speech text into an emotion recognition model, identify the emotion category corresponding to the speech text through the emotion recognition model, and obtain the emotion recognition result. The emotion recognition model is a pre-trained machine learning model, and the emotion recognition model is used to analyze the emotion expressed by the speech text.

[0277] In some optional embodiments, the first recognition unit 1432 is further configured to extract at least one keyword from the spoken text;

[0278] The first generation module 1430 further includes:

[0279] The first generation unit 1433 is used to generate subtitle data corresponding to the speech material based on the emotion recognition result and the at least one keyword;

[0280] The first display module 1410 is also configured to display the second subtitle in the first subtitle display style based on the subtitle data.

[0281] In some optional embodiments, the first recognition unit 1432 is further configured to segment the speech text to obtain at least two candidate words;

[0282] The first identification unit 1432 is further configured to determine the word frequency and inverse document frequency corresponding to the at least two candidate words, respectively;

[0283] The first identification unit 1432 is further configured to determine the word importance corresponding to the candidate word based on the word frequency and the inverse document frequency corresponding to the candidate word;

[0284] The first identification unit 1432 is further configured to determine at least one keyword from the at least two candidate words based on the word importance corresponding to the at least two candidate words respectively.

[0285] In some optional embodiments, the first generation unit 1433 is further configured to input the emotion recognition result, the at least one keyword, and the speech text into the subtitle generation model to obtain the subtitle data. The subtitle generation model is a pre-trained machine learning model. The subtitle generation model is used to combine the emotion recognition result and the keyword to determine the subtitle display style and generate the corresponding subtitle data.

[0286] In some optional embodiments, the first acquisition unit 1431 is further configured to acquire the object audio corresponding to at least two candidate speech objects in the second video content, and to acquire the object importance of the at least two candidate speech objects in the first video.

[0287] The first recognition unit 1432 is further configured to determine at least one first speech object from the at least two candidate speech objects based on the object importance corresponding to the candidate speech object;

[0288] The first acquisition unit 1431 is further configured to acquire the object audio corresponding to the at least one first voice object as the voice material of the second video content.

[0289] In some optional embodiments, the first acquisition unit 1431 is further configured to acquire historical feedback data of the first video, the historical feedback data being used to indicate the feedback received by the first video within a historical time period;

[0290] The first recognition unit 1432 is further configured to determine the object importance corresponding to the at least two candidate speech objects based on the correlation degree between the at least two candidate speech objects and the historical feedback data.

[0291] In some optional embodiments, the first display module 1410 is further configured to display the second subtitle at a first position associated with the first voice object when the first playback volume is lower than the first condition, wherein the voice emotion of the voice material of the second video content matches the emotional state of the first voice object.

[0292] In some optional embodiments, the first determining module 1420 is further configured to obtain playback scene information corresponding to the first video, the playback scene information being used to determine the playback volume of the first video; and based on the playback scene information, to determine the first playback volume of the first video.

[0293] In some optional embodiments, the first determining module 1420 is further configured to obtain the playback scene information corresponding to the video type of the first video;

[0294] The device further includes:

[0295] The first receiving module 1440 is used to receive a volume control operation, the volume control operation being used to control the first video to play at the first playback volume;

[0296] The first determining module 1420 is further configured to determine the playback scene information based on the volume control operation;

[0297] The first receiving module 1440 is further configured to receive a subtitle display operation, wherein the subtitle display operation is used to trigger the display of subtitles of the first video;

[0298] The first determining module 1420 is further configured to determine the playback scene information based on the subtitle display operation; or,

[0299] The first determining module 1420 is further configured to collect physical environment information of the terminal playing the first video as playback scene information of the first video.

[0300] In some optional embodiments, the first display module 1410 is further configured to display a first interactive component when the first playback volume is lower than the first condition, wherein the component display style of the first interactive component is associated with the voice emotion of the voice material of the second video content.

[0301] The first display module 1410 is further configured to display the second subtitle in response to the first interactive component receiving a trigger operation.

[0302] Please refer to Figure 16, which shows a structural block diagram of a caption content display device provided in an exemplary embodiment of this application. The device includes the following modules:

[0303] The second display module 1610 is used to play the first video and display the first subtitles corresponding to the audio material of the currently playing first video content. The first video is used to recommend the first recommended content.

[0304] The second display module 1610 is also used to display a second subtitle corresponding to the audio material of the currently playing second video content, wherein the first subtitle and the second subtitle have different subtitle display styles;

[0305] The second receiving module 1620 is used to receive interactive operations on the first video;

[0306] The second display module 1610 is also configured to respond to the interactive operation by displaying recommendation information corresponding to the first recommended content.

[0307] In some optional embodiments, as shown in FIG17, the apparatus further includes a second generation module 1630, the second generation module 1630 comprising:

[0308] The second acquisition unit 1631 is used to acquire the voice text corresponding to the voice material of the second voice content;

[0309] The second recognition unit 1632 is used to recognize the semantics of the speech text and obtain the emotion recognition result corresponding to the speech text;

[0310] The second display module 1610 is also used to display the second subtitle in the first subtitle display style based on the emotion recognition result.

[0311] In some optional embodiments, the second recognition unit 1632 is further configured to input the speech text into an emotion recognition model, identify the emotion category corresponding to the speech text through the emotion recognition model, and obtain the emotion recognition result. The emotion recognition model is a pre-trained machine learning model, and the emotion recognition model is used to analyze the emotion expressed by the speech text.

[0312] In some optional embodiments, the second recognition unit 1632 is further configured to extract at least one keyword from the spoken text;

[0313] The second generation module 1630 further includes:

[0314] The second generation unit 1633 is used to generate subtitle data corresponding to the audio material of the second video content based on the emotion recognition result and the at least one keyword;

[0315] The second display module 1610 is also used to display the second subtitle in the first subtitle display style based on the subtitle data.

[0316] In some optional embodiments, the second recognition unit 1632 is further configured to segment the speech text to obtain at least two candidate words;

[0317] The second identification unit 1632 is further configured to determine the word frequency and inverse document frequency corresponding to the at least two candidate words, respectively;

[0318] The second identification unit 1632 is further configured to determine the word importance corresponding to the candidate word based on the word frequency and the inverse document frequency corresponding to the candidate word;

[0319] The second identification unit 1632 is further configured to determine at least one keyword from the at least two candidate words based on the word importance corresponding to the at least two candidate words respectively.

[0320] In some optional embodiments, the second generation unit 1633 is further configured to input the emotion recognition result, the at least one keyword, and the speech text into the subtitle generation model to obtain the subtitle data. The subtitle generation model is a pre-trained machine learning model, which is used to combine the emotion recognition result and the keyword to determine the subtitle display style and generate the corresponding subtitle data.

[0321] In some optional embodiments, the second acquisition unit 1631 is further configured to acquire the object audio corresponding to at least two candidate speech objects in the second video content, and to acquire the object importance of the at least two candidate speech objects in the first video.

[0322] The second recognition unit 1632 is further configured to determine at least one first speech object from the at least two candidate speech objects based on the object importance corresponding to the candidate speech object;

[0323] The second acquisition unit 1631 is further configured to acquire the object audio corresponding to the at least one first voice object as the voice material of the second video content.

[0324] In some optional embodiments, the second acquisition unit 1631 is further configured to acquire historical feedback data of the first video, the historical feedback data being used to indicate the feedback received by the first video within a historical time period;

[0325] The second recognition unit 1632 is further configured to determine the object importance corresponding to the at least two candidate speech objects based on the correlation degree between the at least two candidate speech objects and the historical feedback data.

[0326] In some optional embodiments, the second display module 1610 is further configured to display the second subtitle at a first position associated with the first voice object, wherein the voice emotion of the voice material of the second voice content matches the emotional state of the first voice object.

[0327] In some optional embodiments, the second display module 1610 is further configured to run a first application; and play the first video when the running state of the first application meets the first content recommendation conditions.

[0328] The second display module 1610 is also used to play a second video; in response to the second video's playback status meeting the second content recommendation conditions, the first video is played.

[0329] In some optional embodiments, the second display module 1610 is further configured to display a first interactive component, wherein the component display style of the first interactive component is associated with the voice emotion of the voice material of the second video content;

[0330] The second display module 1610 is further configured to display the second subtitle in response to receiving a trigger operation on the first interactive component.

[0331] In some optional embodiments, the second display module 1610 is further configured to display a second interactive component during the playback of the second video content, wherein the component display style of the second interactive component is associated with the voice emotion of the voice material of the second video content;

[0332] The second receiving module 1620 is further configured to receive a trigger operation on the second interactive component as the interactive operation.

[0333] In some optional embodiments, the first video is used to recommend at least two first recommended items;

[0334] The second display module 1610 is further configured to, in response to the first video playing to the video content corresponding to the second audio material associated with the i-th first recommended content, display the second interactive component corresponding to the i-th first recommended content, where i is a positive integer;

[0335] The second receiving module 1620 is further configured to receive a trigger operation on the second interactive component corresponding to the i-th first recommended content as the interactive operation;

[0336] The second display module 1610 is also configured to respond to the interactive operation by displaying recommendation information corresponding to the i-th first recommended content.

[0337] It should be noted that the subtitle display device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the subtitle display device and the subtitle display method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0338] Figure 18 shows a structural block diagram of a terminal 1800 provided in an exemplary embodiment of this application. The terminal 1800 may be a smartphone, tablet computer, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop computer, or desktop computer. The terminal 1800 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0339] Typically, terminal 1800 includes a processor 1801 and a memory 1802.

[0340] Processor 1801 may include one or more processing cores. Processor 1801 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). In some embodiments, processor 1801 may integrate a Graphics Processing Unit (GPU) responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1801 may also include an Artificial Intelligence (AI) processor for handling computational operations related to machine learning.

[0341] The memory 1802 may include one or more computer-readable storage media, which may be non-transitory. In some embodiments, the non-transitory computer-readable storage media in the memory 1802 are used to store at least one instruction, which is executed by the processor 1801 to implement the method for displaying subtitle content provided in the method embodiments of this application.

[0342] Indicatively, terminal 1800 also includes other components 1803. Those skilled in the art will understand that the structure shown in FIG18 does not constitute a limitation on terminal 1800 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0343] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into the terminal. The computer-readable storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set. The at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by the processor to implement any of the subtitle content display methods described in the above embodiments.

Claims

1. A method for displaying subtitle content, the method being executed by a computer device, the method comprising: Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content; Determine the first playback volume of the first video. If the first playback volume is lower than a first condition, display a second subtitle corresponding to the audio material of the currently playing second video content. The first subtitle and the second subtitle have different subtitle display styles.

2. The method according to claim 1, wherein, The step of displaying a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition includes: When the first playback volume is lower than the first condition, the second subtitle is displayed in the first subtitle display style, wherein the first subtitle display style is different when the voice material of the second video content corresponds to different voice emotions.

3. The method according to claim 1 or 2, wherein, The display of the second subtitle in the first subtitle display style includes: Obtain the audio text corresponding to the audio material of the second video content; The semantics of the spoken text are identified to obtain the emotion recognition result corresponding to the spoken text; Based on the emotion recognition result, the second subtitle is displayed in the first subtitle display style.

4. The method according to any one of claims 1 to 3, wherein, The process of recognizing the semantics of the speech text and obtaining the emotion recognition result corresponding to the speech text includes: The speech text is input into the emotion recognition model, and the emotion recognition model identifies the emotion category corresponding to the speech text to obtain the emotion recognition result. The emotion recognition model is a pre-trained machine learning model, which is used to analyze the emotion expressed by the speech text.

5. The method according to any one of claims 1 to 4, wherein, The step of displaying the second subtitle in the first subtitle display style based on the emotion recognition result includes: Extract at least one keyword from the spoken text; Based on the emotion recognition results and the at least one keyword, subtitle data corresponding to the audio material of the second video content is generated; The second subtitle is displayed in the first subtitle display style based on the subtitle data.

6. The method according to any one of claims 1 to 5, wherein, Extracting at least one keyword from the spoken text includes: The speech text is segmented to obtain at least two candidate words; Determine the term frequency and inverse document frequency corresponding to the at least two candidate words, respectively; The importance of the candidate words is determined based on the word frequency and inverse document frequency corresponding to the candidate words. Based on the word importance corresponding to the at least two candidate words, at least one keyword is determined from the at least two candidate words.

7. The method according to any one of claims 1 to 6, wherein, The step of generating subtitle data corresponding to the audio material of the second video content based on the emotion recognition result and the at least one keyword includes: The emotion recognition result, the at least one keyword, and the speech text are input into the subtitle generation model to obtain the subtitle data. The subtitle generation model is a pre-trained machine learning model. The subtitle generation model is used to combine the emotion recognition result and the keyword to determine the subtitle display style and generate the corresponding subtitle data.

8. The method according to any one of claims 1 to 7, wherein, Before obtaining the audio text corresponding to the audio material of the second video content, the method further includes: Obtain the object audio corresponding to at least two candidate speech objects in the second video content, and obtain the object importance of the at least two candidate speech objects in the first video; Based on the object importance corresponding to the candidate speech object, at least one first speech object is determined from the at least two candidate speech objects; Obtain the audio of the object corresponding to at least one first voice object as the audio material of the second video content.

9. The method according to any one of claims 1 to 8, wherein, Obtaining the object importance of the at least two candidate speech objects in the first video includes: Obtain historical feedback data of the first video, wherein the historical feedback data is used to indicate the feedback received by the first video within a historical time period; Based on the correlation between the at least two candidate speech objects and the historical feedback data, the object importance corresponding to the at least two candidate speech objects is determined.

10. The method according to any one of claims 1 to 9, wherein, The step of displaying a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition includes: When the first playback volume is lower than the first condition, the second subtitle is displayed at the first position associated with the first voice object, wherein the voice emotion of the voice material of the second video content matches the emotional state of the first voice object.

11. The method according to any one of claims 1 to 10, wherein, Determining the first playback volume of the first video includes: Obtain the playback scene information corresponding to the first video, and the playback scene information is used to determine the playback volume of the first video; Based on the playback scenario information, the first playback volume of the first video is determined.

12. The method according to any one of claims 1 to 11, wherein, The step of obtaining the playback scene information corresponding to the first video includes: Obtain the playback scene information corresponding to the video type of the first video; or, Receive a volume control operation, the volume control operation being used to control the first video to play at a first playback volume; determine the playback scene information based on the volume control operation; or... The physical environment information of the terminal playing the first video is collected as the playback scene information.

13. The method according to any one of claims 1 to 12, wherein, The step of displaying a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition includes: When the first playback volume is lower than the first condition, the first interactive component is displayed, and the component display style of the first interactive component is related to the voice emotion of the voice material of the second video content; In response to the first interactive component receiving a trigger operation, the second subtitle is displayed.

14. A method for displaying subtitle content, the method being executed by a computer device, the method comprising: Play the first video and display the first subtitle corresponding to the audio material of the currently playing first video content. The first video is used to recommend the first recommended content. The first subtitle and the second subtitle are displayed in different styles. Receive interactive operations on the first video; In response to the interactive operation, the recommendation information corresponding to the first recommended content is displayed.

15. The method according to claim 14, wherein, Playing the first video includes: Run the first application; if the running state of the first application meets the first content recommendation criteria, play the first video; or, Play the second video; in response to the second video's playback status satisfying the second content recommendation condition, play the first video.

16. The method according to claim 14 or 15, wherein, The second subtitle, which displays the audio material corresponding to the currently playing second video content, includes: The first interactive component is displayed, and the component display style of the first interactive component is related to the emotional tone of the audio material of the second video content. In response to receiving a trigger operation on the first interactive component, the second subtitle is displayed.

17. The method according to any one of claims 14 to 16, wherein, The receiving of interactive operations on the first video includes: During the playback of the second video content, a second interactive component is displayed, and the component display style of the second interactive component is related to the emotional tone of the audio material of the second video content; The trigger operation on the second interactive component is received as the interactive operation.

18. The method according to any one of claims 14 to 17, wherein, The first video is used to recommend at least two primary recommended content items; The process of displaying the second interactive component during the playback of the second video content includes: In response to the first video playing to the second video content associated with the i-th first recommended content, the second interactive component corresponding to the i-th first recommended content is displayed, where i is a positive integer; The receiving of a trigger operation on the second interactive component as the interactive operation includes: The trigger operation of the second interactive component corresponding to the i-th first recommended content is received as the interactive operation; The step of displaying recommendation information corresponding to the first recommended content in response to the interactive operation includes: In response to the interactive operation, the recommendation information corresponding to the i-th first recommended content is displayed.

19. A device for displaying subtitle content, the device comprising: The first display module is used to play the first video and display the first subtitles corresponding to the audio material of the currently playing first video content; The first determining module is used to determine the first playback volume of the first video; The first display module is further configured to display a second subtitle corresponding to the audio material of the currently playing second video content when the first playback volume is lower than the first condition, wherein the first subtitle and the second subtitle have different subtitle display styles.

20. A device for displaying subtitle content, the device comprising: The second display module is used to play the first video and display the first subtitles corresponding to the audio material of the currently playing first video content. The first video is used to recommend the first recommended content. The second display module is also used to display a second subtitle corresponding to the audio material of the currently playing second video content, wherein the first subtitle and the second subtitle have different subtitle display styles; The second receiving module is used to receive interactive operations on the first video; The second display module is also configured to respond to the interactive operation by displaying recommendation information corresponding to the first recommended content.

21. A computer device comprising a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the method for displaying subtitle content as described in any one of claims 1 to 18.

22. A computer-readable storage medium storing at least one piece of program code, the program code being loaded and executed by a processor to implement the method for displaying subtitle content as described in any one of claims 1 to 18.

23. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the method for displaying subtitle content as described in any one of claims 1 to 18.

Images