Live-photo generation method and apparatus, and terminal and computer program product

Abstract

The embodiments of the present application belong to the technical field of image processing. Disclosed are a live-photo generation method and apparatus, and a terminal and a computer program product. The method comprises: acquiring an event screenshot of a target event in an application (101), wherein the event screenshot includes a screen picture when the target event occurs; acquiring an event video of the target event (102), wherein the event video is a screen-recording video within an event time period corresponding to the target event; and on the basis of the event screenshot and the event video, generating a live photo of the target event (103). By using the solution provided in the embodiments of the present application, a live photo of a target event occurring in an application can be automatically generated.

Description

Live photo generation methods, devices, terminals, and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202510124919.X, filed on January 26, 2025, entitled “Live Photo Generation Method, Apparatus, Terminal and Computer Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a method, apparatus, terminal, and computer program product for generating live photos. Background Technology

[0003] Live photos, as a novel form of photo display, allow users to view both static images and a dynamic video related to those static images.

[0004] In related technologies, live photos are usually taken by a camera from a terminal that supports live streaming functionality. Summary of the Invention

[0005] This application provides a method, apparatus, terminal, and computer program product for generating live photos. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for generating live photos, the method being used on a terminal, the method comprising:

[0007] Obtain an event screenshot of a target event within the application, the event screenshot including the screen view when the target event occurs;

[0008] Obtain the event video of the target event, wherein the event video is a screen recording video of the event time period corresponding to the target event;

[0009] A live photo of the target event is generated based on the event screenshots and the event video.

[0010] On the other hand, embodiments of this application provide a live photo generation apparatus, the apparatus comprising:

[0011] The screenshot acquisition module is used to acquire an event screenshot of a target event within the application, wherein the event screenshot includes the screen image when the target event occurs;

[0012] The video acquisition module is used to acquire the event video of the target event, wherein the event video is a screen recording video within the event time period corresponding to the target event;

[0013] The generation module is used to generate a live photo of the target event based on the event screenshot and the event video.

[0014] On the other hand, embodiments of this application provide a terminal, the terminal including a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the live photo generation method as described above.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the live photo generation method as described above.

[0016] On the other hand, embodiments of this application provide a computer program product, which includes computer instructions, and a processor executes the computer instructions to implement the live photo generation method as described above. Attached Figure Description

[0017] Figure 1 shows a flowchart of a live photo generation method provided in an exemplary embodiment of this application;

[0018] Figure 2 is an exemplary embodiment of this application illustrating the process of generating live photos based on event screenshots and videos;

[0019] Figure 3 is an exemplary embodiment of this application illustrating the process of generating a live photo corresponding to a highlight moment in a game scene;

[0020] Figure 4 is a schematic diagram illustrating an exemplary embodiment of the application running video generation process of this application;

[0021] Figure 5 is a flowchart illustrating an exemplary embodiment of the event video generation process based on a cache queue in this application;

[0022] Figure 6 is a schematic diagram illustrating an exemplary embodiment of the event video generation process of this application;

[0023] Figure 7 is a schematic diagram illustrating an implementation of the event audio and video data extraction process in an exemplary embodiment of this application;

[0024] Figure 8 is a schematic diagram illustrating an exemplary embodiment of the target event audio and video data extraction process of this application;

[0025] Figure 9 is a schematic diagram illustrating the correspondence between event information, live photo duration, and target time period duration in an exemplary embodiment of this application;

[0026] Figure 10 is a schematic diagram illustrating an implementation of the target event setting process in an exemplary embodiment of this application;

[0027] Figure 11 shows a structural block diagram of a live photo generation apparatus provided in another exemplary embodiment of this application;

[0028] Figure 12 shows a structural block diagram of a terminal provided in an exemplary embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] During application use, when certain in-app events are triggered, users typically use various methods to record these events. For example, in game applications, when a highlight action is performed during a game (i.e., a highlight action event is triggered), users can manually take a screenshot of the highlight moment.

[0032] However, because some in-app events are fleeting, users may easily miss the opportunity to manually take screenshots. Furthermore, since only static images are available, subsequent event review based on these static images is less effective.

[0033] To improve the efficiency and effectiveness of recording in-application events, the solution provided in this application embodiment includes a function in which the terminal automatically generates live photos of target events occurring within the application. These live photos include a screenshot of the screen when the target event occurs, and a screen recording video of the corresponding event time period. This allows users to quickly review the target event through the live photos, improving the effectiveness of event review. Furthermore, since the live photos are automatically generated by the terminal, there is no need for users to manually record or take screenshots, simplifying the live photo generation process and improving the efficiency of live photo generation.

[0034] The method provided in this application embodiment can be executed by a terminal, which may be a smartphone, tablet computer, personal computer, wearable device, etc. The terminal has application running and screen display functions, capable of displaying the application running screen and generated live photos. Furthermore, the function of generating live photos for in-application events can be provided by the terminal's operating system, i.e., live photo generation is implemented at the system level.

[0035] In the following embodiments, for ease of description, the method of generating live photos is described as being executed by the terminal, but this does not constitute a limitation.

[0036] Please refer to Figure 1, which shows a flowchart of a live photo generation method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a terminal as an example for illustration, and the method may include the following steps:

[0037] Step 101: Obtain an event screenshot of the target event within the application. The event screenshot includes the screen image when the target event occurs.

[0038] Here, "application" can be any application running on the terminal, or a specific application running on the terminal. For example, the specific application is an application on the application whitelist, and the terminal system supports generating live photos of in-application events for applications on the application whitelist.

[0039] The target event is a specified event that occurs during the operation of the application, typically an event with logistic significance. Furthermore, the target event may be the same or different for different applications. For example, a highlight event in a game application, a designated person speaking in an online meeting application, a successful ticket purchase event in a ticketing application, etc. This application embodiment does not limit the specific event type of the target event.

[0040] In some embodiments, the target event can be an event pre-defined by the application, an event pre-defined by the terminal system, or a user-defined event. For example, each application may have its own set of target events that need to be recorded in the form of live photos, or the terminal system may have a set of target events that support the generation of live photos, or the user may select the target event from several candidate events to generate a live photo.

[0041] In some embodiments, the event screenshot is a screenshot automatically taken by the terminal when the target event occurs, and includes the screen display at the time the target event occurs. Specifically, the event screenshot includes the screen display area corresponding to the application's display window. For example, when the application is displayed in full-screen mode, the event screenshot includes the entire screen display area; when the application is displayed in split-screen mode, the event screenshot includes the screen display area corresponding to the split-screen area where the application is located; when the application is displayed in a floating window, the event screenshot includes the screen display area corresponding to the floating window where the application is located.

[0042] Step 102: Obtain the event video of the target event. The event video is a screen recording video of the time period corresponding to the target event.

[0043] The event time period corresponding to the target event is determined based on the occurrence time of the target event, and the event time period includes the occurrence time of the target event.

[0044] Furthermore, since the event video is subsequently used to generate live photos, the event segment is usually a short duration to meet the requirements for generating live photos. For example, the event video is a 3-second, 5-second, or 10-second clip corresponding to the target event.

[0045] In this embodiment, the event video is automatically recorded by the terminal and includes continuous screen footage within the event period. Specifically, the event video includes continuous screen footage within the display area corresponding to the application's window during the event period. For example, when the application is displayed in full-screen mode, the event video includes continuous screen footage of the entire display area during the event period; when the application is displayed in split-screen mode, the event video includes continuous screen footage of the display area corresponding to the split-screen area where the application is located during the event period; and when the application is displayed in a floating window, the event screenshot includes continuous screen footage of the display area corresponding to the floating window where the application is located during the event period.

[0046] It should be noted that there is no strict sequential execution order for steps 101 and 102 above. This embodiment only illustrates the example of obtaining the event screenshot first and then the event video, but it does not limit the execution order of the two.

[0047] Step 103: Generate a live photo of the target event based on the event screenshot and event video.

[0048] The terminal can generate a live photo in real time after acquiring event screenshots and videos, or it can delay generating the live photo when the conditions for generating the live photo are met, so as to avoid the generation process affecting the operation of the application due to the consumption of computing resources.

[0049] Regarding the specific method of generating live photos based on event screenshots and event videos, in one possible implementation, as shown in Figure 2, during the encoding of the live photo, the terminal inserts the event video 22 at the end of the event screenshot 21 and writes information such as the offset and video length of the event video into the description file (Extensible Metadata Platform, XMP) 23. When parsing the live photo subsequently, the video position of the event video can be determined based on the offset and video length information, and then the parsing and playback of the event video can begin from that position.

[0050] As illustrated in Figure 3, during a game using a game application, for any highlight operation event that occurs, the terminal automatically acquires an event screenshot 31 of the highlight operation and an event video 32 corresponding to the execution period of the highlight operation, and then generates a live photo 33 corresponding to the highlight operation event based on the event screenshot 31 and the event video 32.

[0051] For the generated live photos, the terminal can save them to its photo album; and / or, the terminal can display the live photo and provide different function entry points for further processing of the live photo. These function entry points include, but are not limited to, editing, saving, deleting, sharing, etc.

[0052] In summary, in this embodiment, the terminal can automatically generate live photos of target events occurring within the application. These live photos include a screenshot of the screen when the target event occurs, and a screen recording video of the corresponding event period, allowing users to quickly review the target event. For example, the terminal can generate a live photo corresponding to a highlight event in a game application based on a screenshot of the highlight event and a screen recording video of the highlight event's occurrence, facilitating quick review of the highlight moment. Since the live photos are automatically generated by the terminal, eliminating the need for manual recording or shooting by the user, the generation process is simplified, and the efficiency of live photo generation is improved.

[0053] In some embodiments, obtaining an event screenshot of a target event within the application includes at least one of the following:

[0054] Upon receiving an event occurrence command sent by the application, a screenshot is taken of the screen to obtain the event screenshot. The event occurrence command is used to notify that the target event has occurred.

[0055] Upon recognizing the target event within the application, a screenshot is taken of the screen to obtain the event screenshot;

[0056] From the application's running video, video frames are extracted as the event screenshot of the target event. The application running video is a video obtained by screen recording during the application's operation.

[0057] Upon receiving an external command that meets the screenshot criteria, a screenshot is taken of the screen to obtain the event screenshot. The external command includes at least one of voice commands, manual operation commands, physical button commands, and physiological signal commands.

[0058] In some embodiments, when the event screenshot is taken after the target event within the application has been identified, the method includes at least one of the following:

[0059] During application operation, screen content recognition is performed; if the screen content recognition result indicates the presence of an event visual element corresponding to the target event, the target event within the application is identified.

[0060] During application operation, audio content recognition is performed on the application's audio; if the audio content recognition result indicates the presence of an auditory element corresponding to the target event, the target event within the application is identified.

[0061] In some embodiments, extracting video frames from the application's running video as the event screenshot of the target event includes at least one of the following:

[0062] Based on the time of occurrence of the target event, the video frame located at the time of occurrence is extracted from the application's running video as the event screenshot of the target event;

[0063] Perform content recognition on the application running video; if the target event is recognized, extract the video frame containing the target event as the event screenshot of the target event.

[0064] In some embodiments, acquiring the event video of the target event includes at least one of the following:

[0065] The event audio and video data of the target event are obtained from the cache queue, which is used to cache audio and video data of the target duration. The audio and video data includes video data corresponding to the screen display during application operation and audio data of the application audio. The event audio and video data are encoded to obtain the event video of the target event.

[0066] The event video of the target event is extracted from the application's running video, which is a video recorded during the application's operation.

[0067] In some embodiments, retrieving the event audio and video data of the target event from a cache queue includes:

[0068] Using the time of occurrence of the target event as a reference point, the audio and video data of the event within the target time period are obtained from the cache queue, wherein the time of occurrence of the event is within the target time period.

[0069] In some embodiments, taking the time of occurrence of the target event as a reference point, retrieving the audio and video data of the event within the target time period from the cache queue includes:

[0070] Based on the time of the event and the first duration, the starting point of the target time period is determined, and the starting point of the target time period is located before the time of the event.

[0071] Based on the time of the event and the second duration, the end point of the target time period is determined, and the end point of the target time period is located after the time of the event.

[0072] Based on the start and end points of the target time period, the event audio and video data within the target time period are retrieved from the cache queue.

[0073] In some embodiments, the method further includes:

[0074] Create the cache queue, which is a circular cache queue;

[0075] During application operation, the audio and video data are cached in the cache queue.

[0076] In some embodiments, the duration of the target time period is greater than the duration of the live photo;

[0077] Encoding the event audio and video data to obtain the event video of the target event includes:

[0078] Based on the duration of the live photo, target event audio and video data is extracted from the event audio and video data, and the duration of the target event audio and video data matches the duration of the live photo.

[0079] The target event audio and video data are encoded to obtain the event video of the target event.

[0080] In some embodiments, extracting target event audio and video data from the event audio and video data based on the duration of the live photo includes:

[0081] Identify the audio and video data of the event to obtain the start and end points of the target event;

[0082] Based on the event start point, the event end point, and the duration of the live photo, the target event audio and video data is extracted from the event audio and video data, which includes audio and video data between the event start point and the event end point.

[0083] In some embodiments, the target time period and the duration of the live photo are determined based on the event information of the target event, or the target time period and the duration of the live photo are preset values.

[0084] In some embodiments, extracting the event video of the target event from the application's running video includes at least one of the following:

[0085] Using the time of occurrence of the target event as a reference point, the event video of the target event is extracted from the application's running video;

[0086] Perform content recognition on the application running video; based on the content recognition results, extract the event video of the target event from the application running video.

[0087] In some embodiments, the event screenshots and the event videos are obtained under at least one of the following conditions:

[0088] The application belongs to the target application list;

[0089] The application belongs to the target application list, and the application is running in the target application scenario;

[0090] The target application list includes applications that support live photo generation, and the target application scenario is an application scenario with a need for live photo generation.

[0091] In some embodiments, the method further includes:

[0092] The configuration interface is displayed, which includes at least one candidate event, which is an event that supports the generation of live photos;

[0093] In response to the event configuration operation in the configuration interface, the candidate event indicated by the event configuration operation is determined as the target event.

[0094] The process of obtaining event screenshots

[0095] Regarding the timing of obtaining event screenshots, it is optional that event screenshots can be obtained in real time when the target event occurs, or after the target event has occurred.

[0096] Regarding the method of obtaining event screenshots, optional methods include obtaining event screenshots through screen capture or extracting them from screen recording videos.

[0097] Regarding the method for determining the occurrence time of the target event corresponding to the event screenshot, optionally, the occurrence time of the target event can be determined by the terminal system based on the notification sent by the application, or it can be automatically identified by the terminal system, or it can be determined by the terminal system based on specific external instructions.

[0098] Based on the above-mentioned timing, method, and determination of the occurrence time of event screenshots, in some embodiments, obtaining event screenshots of target events within the application can be achieved using at least one of the following methods:

[0099] Method 1: Upon receiving an event occurrence command from the application, take a screenshot of the screen to obtain the event screenshot. The event occurrence command is used to notify that the target event has occurred.

[0100] In one possible implementation, the application has the function of identifying target events within the application. When a target event is identified, the application sends an event occurrence command to the terminal system through the interface provided by the terminal system to notify the terminal system that the target event has occurred.

[0101] Optionally, the event occurrence command may include event information for the target event, such as the event name, event type, event importance, and event timestamp.

[0102] Upon receiving an event notification, the terminal system takes a real-time screenshot of the screen to obtain an event screenshot of the target event. The terminal system can take screenshots using MediaProjection, ScreenCap, or adb commands; this embodiment does not limit the specific method used.

[0103] In some embodiments, when the application is displayed in full screen, the terminal system determines the captured image as the event screenshot of the target event; when the application is not displayed in full screen, the terminal system can further extract the event screenshot of the target event from an image containing the full screen based on the display position information of the application's display window.

[0104] In an illustrative example, during the operation of a game application, when a kill event command is received from the game application, the terminal system takes a screenshot to obtain the event screenshot corresponding to the kill event; during the operation of a video conferencing application, when a command is received from the video conferencing application to allow an important person to speak, the terminal system takes a screenshot to obtain the event screenshot of the important person speaking.

[0105] Method 2: If the target event within the application is detected, take a screenshot of the screen to obtain the event screenshot.

[0106] In addition to relying on application notifications, the terminal system can also automatically identify whether a target event has occurred within an application, and automatically take a screenshot of the screen when the target event is detected. Compared to application notification solutions, even if the application is not functionally adapted to the system (for example, the application does not support sending event occurrence commands), the terminal system's automatic identification solution can still achieve automatic generation of live photos, expanding the applicability of live photo generation functionality.

[0107] In one possible implementation, the terminal system can identify target events within the application from at least one dimension, either visual or auditory.

[0108] Identify target events within the application from a visual perspective.

[0109] During application operation, the terminal performs screen content recognition; if the screen content recognition results indicate the presence of visual elements corresponding to the target event, the target event within the application is identified.

[0110] The event visual elements may include text, symbols, patterns, and other visual elements specific to the occurrence of the target event. For example, for gaming applications, the event visual elements may include text such as "kill," "pentakill," or "extreme maneuver"; for video conferencing applications, the event visual elements may include "speech patterns corresponding to important characters." The presentation of the event visual elements can vary depending on the type of application and event, and this embodiment does not limit this.

[0111] In some embodiments, the terminal performs screen content recognition on the screen within the application's display window to obtain the screen content recognition result.

[0112] Optionally, the terminal may use technologies such as OCR and image matching to recognize screen content in order to identify different types of event visual elements. This embodiment does not limit this.

[0113] Furthermore, the visual elements corresponding to the target event can be selected from all event visual elements based on the application information of the current application, thereby reducing the computational load of the screen content recognition process.

[0114] In some embodiments, since the screen content typically changes little over a short period, the terminal can perform screen content recognition at preset frame intervals to reduce the computational load of the screen content recognition process. These preset frame intervals can be determined based on the application's display frame rate. For example, when the application's display frame rate is 60 frames per second, the terminal performs screen content recognition every 10 frames; when the application's display frame rate is 120 frames per second, the terminal performs screen content recognition every 20 frames.

[0115] Identifying target events within the application from an auditory perspective

[0116] During application operation, the terminal performs audio content recognition on the application's audio; if the audio content recognition results indicate the presence of an auditory element corresponding to the target event, the target event within the application is identified.

[0117] Since some content in the application's audio cannot be displayed on the application screen, identifying target events based on application audio can further improve the comprehensiveness and accuracy of target event identification.

[0118] The auditory elements of this event can include event keywords, event audio melodies, event music, and other auditory elements specific to the occurrence of the target event. For example, in a game application, the auditory elements of this event can include event keywords such as "victory" and / or victory music. The form of the auditory elements can vary depending on the type of application and event, and this embodiment does not limit this.

[0119] In some embodiments, the terminal performs audio content recognition on the audio content of the audio path corresponding to the application to obtain the audio content recognition result.

[0120] In one possible implementation, the terminal performs real-time text conversion on the application audio to obtain application audio text, and performs event keyword recognition on the application audio text to obtain keyword recognition results.

[0121] The event keywords can be determined from an event keyword database based on the application's information. For example, for a gaming application, the event keywords would include gaming terms from the database; for a video conferencing application, the event keywords would include business terms from the database.

[0122] In another possible implementation, the terminal performs melody recognition on the application audio to obtain a melody recognition result. This melody recognition process can involve matching the application audio with preset audio to determine whether the preset audio is included.

[0123] Method 3: Extract video frames from the application's running video as event screenshots of the target event. The application running video is a video recorded during the application's operation.

[0124] Because real-time screenshots or screenshots taken after real-time recognition may have a certain lag and may suffer from missing events or inaccurate recognition, in one possible implementation, to improve the accuracy and comprehensiveness of target event recognition and the quality of event screenshots, the terminal records the screen during application operation to obtain an application running video. Subsequently, the target event is identified from this application running video, and event screenshots are extracted from the application running video through video frame extraction.

[0125] Optionally, the terminal can continuously record the screen while the application is running in the foreground to obtain a video of the application running.

[0126] Optionally, the terminal can start screen recording when the application enters a specific application scenario and stop recording when the application leaves that scenario, thus obtaining a video of the application running, thereby reducing the amount of data in the video. For example, for a game application, the terminal can start screen recording when the application enters a game and stop recording when the game ends, thus obtaining a video of the application running that includes the complete game process; for a video conferencing application, the terminal can start screen recording when the video conference begins and stop recording when the video conference ends, thus obtaining a video of the application running that includes the complete video conference.

[0127] Regarding the method of determining the target event contained in the application's running video and extracting the corresponding video frame, in one possible implementation, the terminal extracts video frames based on the time of event occurrence recorded during the application's operation, and / or, the terminal determines the target event and extracts video frames by recognizing the video content.

[0128] Extraction Method 1: Based on the time of occurrence of the target event, extract the video frame located at the time of the event from the application's running video as the event screenshot of the target event.

[0129] Optionally, the time of the event can be extracted by the terminal from the event occurrence command sent by the application, and / or obtained by the terminal in real time during the application's operation.

[0130] In some embodiments, the video frame at the time of the event is a video frame in the application running video with the same timestamp as the time of the event.

[0131] In other embodiments, in order to improve the quality of the event screenshot, the video frame at the time of the event is a video frame selected from a number of candidate video frames, wherein the candidate video frames are determined based on the time of the event.

[0132] Optionally, the terminal selects several video frames before and after the event occurrence time as candidate video frames, using the event occurrence time as the midpoint. The selection method may include choosing the video frame with the highest event match to the target event (i.e., selecting the video frame that best represents the target event).

[0133] Extraction Method 2: Perform content recognition on the application running video; if the target event is recognized, extract the video frame containing the target event as the event screenshot of the target event.

[0134] In one possible implementation, the terminal performs content recognition on the application's running video to obtain a content recognition result indicating whether it contains the target event. The method for identifying whether the target event is included can refer to the acquisition method 2 described above, which identifies the target event from both visual and auditory dimensions; this embodiment will not elaborate further.

[0135] Optionally, the terminal can determine the candidate target events that may exist in the application based on the application information, and perform content recognition (identify whether the candidate target event is included) based on the candidate target events, thereby narrowing the recognition scope and improving the recognition accuracy.

[0136] When a target event is detected, the terminal extracts one frame from several video frames containing the target event as an event screenshot of the target event. Optionally, the method of selecting the video frame may include: selecting the video frame with the highest event matching degree to the target event (i.e., selecting the video frame that best represents the target event), or selecting the video frame with the highest image quality.

[0137] Method 4: Upon receiving an external command that meets the screenshot conditions, take a screenshot of the screen to obtain an event screenshot. The external command includes at least one of voice commands, manual operation commands, or physiological signal commands.

[0138] In addition to application notification events and automatic target event identification by the terminal, the terminal can also determine whether the external command meets the screenshot conditions when it receives an external command, and take a screenshot if the screenshot conditions are met to obtain an event screenshot.

[0139] The external command can be triggered actively by the user or automatically based on the user's state. Correspondingly, external commands that meet the screenshot criteria represent the user's intention to participate in an in-application event at the current moment. The event screenshot obtained in this way represents the target event that the user wishes to record.

[0140] In addition, the terminal can also determine the time when the external command is received as the time when the target event occurs.

[0141] In some embodiments, when the external command is a voice command, the terminal determines that the external command meets the screenshot conditions when a keyword is detected in the voice command. The keyword can be a keyword that directly triggers a screenshot, such as "Quickly take a screenshot for me," or it can be an application-related event keyword, such as "Give me a pentakill."

[0142] In some embodiments, when the external instruction is a manual operation instruction, and when the manual operation instruction is used to trigger a manual screenshot, the terminal determines that the external instruction meets the screenshot conditions. The manual operation instruction that triggers a manual screenshot can be an instruction triggered by a screenshot gesture or an instruction triggered by a combination of clicking a screenshot button.

[0143] In some embodiments, when the external instruction is a physiological signal instruction, the terminal determines that the physiological signal instruction meets the screenshot conditions when the signal value representing the physiological signal reaches a numerical threshold. The physiological signal can be collected by the terminal or by an external device with which it has a communication connection. For example, when the physiological signal is a heart rate signal collected by a smartwatch, the terminal takes a screenshot of the screen when the heart rate represented by the heart rate signal reaches a heart rate threshold, thus obtaining an event screenshot.

[0144] It should be noted that the terminal can use any of the above acquisition methods to obtain event screenshots, or it can use a combination of at least two acquisition methods to obtain event screenshots. This application embodiment does not limit the specific combination of acquisition methods.

[0145] In this embodiment, by setting multiple target event recognition methods and event screenshot acquisition methods, it is helpful to improve the accuracy and comprehensiveness of target event recognition, thereby helping to improve the comprehensiveness and quality of the acquired event screenshots.

[0146] The process of acquiring the event video

[0147] In one possible implementation, the terminal records the screen while the application is running, obtaining a video of the application's operation. The terminal then uses video clipping to extract event video of the target event from the application's running video. Since the application running video contains the complete picture of the application's operation, this method can improve the completeness and accuracy of the target event contained in the event video.

[0148] Optionally, the terminal can continuously record the screen while the application is running in the foreground to obtain a video of the application running.

[0149] Optionally, the terminal can start screen recording when the application enters a specific application scenario and stop recording when the application leaves that scenario, thus obtaining a video of the application running, thereby reducing the amount of data in the video. For example, for a game application, the terminal can start screen recording when the application enters a game and stop recording when the game ends, thus obtaining a video of the application running that includes the complete game process; for a video conferencing application, the terminal can start screen recording when the video conference begins and stop recording when the video conference ends, thus obtaining a video of the application running that includes the complete video conference.

[0150] Regarding the method of recording application runtime video, in one possible implementation, the terminal creates a virtual display and maps the real-time screen image onto this virtual display. Additionally, the terminal creates a surface to receive virtual display data; this surface can be placed within the virtual display to receive virtual display data. The virtual display data received through this surface is then sent to an encoder for encoding to obtain the application runtime video.

[0151] As illustrated in Figure 4, the terminal creates a MediaProjection object 42 through MediaProjectionManager 41 and sets a corresponding VirtualDisplay 43 for this MediaProjection object 42. MediaRecorder 44 or MediaCodec 45 creates a Surface 46 and sets it on the VirtualDisplay 43 to receive virtual screen data. The virtual screen data received by Surface 46 is sent to MediaCodec 45 for encoding to obtain the application running video 47.

[0152] In addition, the terminal can also use other methods to record application running videos, and this embodiment does not constitute a limitation.

[0153] In some embodiments, the terminal can capture event video using at least one of the following methods:

[0154] Method 1: Using the time when the target event occurs as a reference point, extract the event video of the target event from the application's running video.

[0155] Optionally, the time of the event can be extracted by the terminal from the event occurrence command sent by the application, and / or obtained by the terminal in real time during the application's operation.

[0156] In one possible implementation, the terminal extracts video footage from the application's running video, including footage from a period before and after the event, as the event video of the target event. The duration of the extracted event video is related to the video duration that the live photo can contain.

[0157] In another possible implementation, the terminal uses the time of the target event as a reference point to perform content recognition on video frames before and after the event, determines the relevance between the video frames and the target event, and extracts event video related to the target event based on this relevance. Specifically, the relevance between the video frames included in the event video and the target event is greater than a relevance threshold.

[0158] Method 2: Perform content recognition on the application running video; based on the content recognition results, extract the event video of the target event from the application running video.

[0159] To improve the comprehensiveness of target event identification, in one possible implementation, the terminal performs content recognition on the application's running video. The obtained content recognition result includes all target events contained in the application's running video (including, but not limited to, target events notified by the application through event notification commands). The method by which the terminal identifies target events can refer to the acquisition method 2 described above, and will not be elaborated upon in this embodiment.

[0160] Furthermore, based on the identified target event, the terminal extracts the corresponding event video from the application's running video. Optionally, the terminal can extract the event video from the application's running video based on the time of occurrence of the identified target event.

[0161] When using the above-mentioned video recording application, continuous audio and video encoding is required during application operation, resulting in high computational load and significant storage space consumption. To reduce the computational load when generating live photos and decrease storage space usage, in another possible implementation, the terminal can use a cache queue to cache audio and video data from a recent period. When a target event occurs, the terminal retrieves and encodes the audio and video data related to the target event from the cache queue. As shown in Figure 5, this process may include the following steps:

[0162] Step 501: Obtain the event audio and video data of the target event from the cache queue. The cache queue is used to cache audio and video data of the target duration. The audio and video data includes video data corresponding to the screen display during application operation and audio data of the application's application audio.

[0163] In one possible implementation, the terminal creates a cache queue during application runtime and stores the application's real-time audio and video data into this cache queue. This cache queue supports circular writes; that is, when the cache queue is full, new audio and video data will overwrite the historical audio and video data in the queue, thus utilizing the cache queue to store audio and video data from a recent period. In some embodiments, this cache queue is a circular cache queue.

[0164] Optionally, the cache queue supports cached audio and video data with a target duration greater than or equal to the duration of the video in the live photo. For example, if the duration of the video in the generated live photo is 3 seconds and the application's frame rate is 60 frames / second, the cache queue supports caching 200 frames (>180 frames) of audio and video data.

[0165] In one possible implementation, during application operation, the terminal creates a cache queue of a target length, where the audio and video duration corresponding to the target length is greater than or equal to the duration of a target time period, and the cache queue is a circular cache queue. During application operation, the terminal caches audio and video data into the cache queue.

[0166] As illustrated in Figure 7, audio and video data are written to a circular buffer queue in a counter-clockwise direction. This circular buffer queue is used to store audio and video data for the most recent 16 unit durations (each unit duration can correspond to multiple frames).

[0167] Regarding the source of audio and video data in the cache queue, in one possible implementation, the terminal creates a virtual display and maps the real-time screen image onto this virtual display. Furthermore, the terminal creates a surface for receiving virtual display data; this surface can be placed within the virtual display to receive virtual display data. Virtual display data received through this surface is then written to the cache queue.

[0168] As illustrated in Figure 6, the terminal creates a MediaProjection object 62 through MediaProjectionManager 61 and sets a corresponding VirtualDisplay 63 for this MediaProjection object 62. MediaRecorder 64 or MediaCodec 65 creates a Surface 66 and sets it on the VirtualDisplay 63 to receive virtual screen data. The virtual screen data received by Surface 66 is sent to the buffer queue 67 for caching.

[0169] Regarding the timing of retrieving the event audio and video data of the target event from the cache queue, in some embodiments, the terminal retrieves the event audio and video data of the target event from the cache queue upon receiving an event occurrence instruction sent by the application, and / or upon identifying a target event within the application.

[0170] To improve the correlation between the extracted event audio and video data and the target event, in one possible implementation, the terminal uses the time of occurrence of the target event as a reference point to obtain the event audio and video data within the target time period from the cache queue, wherein the time of occurrence of the event is within the target time period.

[0171] Optionally, the duration of the target time period can be greater than or equal to the duration of the live photo, so that the event audio and video data can be further cropped later. For example, if the duration of the live photo is 3 seconds, the duration of the target time period can be 3 seconds, 4 seconds, or 5 seconds.

[0172] Optionally, the target time period can start from the time of the event, or end at the time of the event, or the time of the event can be between the start and end of the target time period.

[0173] In some embodiments, when the target time period starts from the time of the event, the terminal obtains the audio and video data cached after the audio and video data corresponding to the time of the event in the cache queue, and uses it as the event audio and video data.

[0174] When the target time period ends at the time of the event, the terminal retrieves the audio and video data cached before the time of the event in the cache queue, and uses it as the event audio and video data.

[0175] When the event occurs between the start and end of the target time period, the terminal retrieves the audio and video data cached before and after the corresponding audio and video data from the cache queue as the event audio and video data.

[0176] Because there may be a lag in the application sending the event occurrence command and the terminal system recognizing the target event, and because the target event may last for a period of time, in order to ensure that the event audio and video data more completely reflects the target event, the terminal defines the audio and video data within a certain period before and after the event occurrence as the event audio and video data. The process of acquiring event audio and video data may include the following sub-steps:

[0177] Step 501A: Based on the time of the event and the first duration, determine the starting point of the target time period, which is located before the time of the event.

[0178] The first duration refers to the length of audio and video data prior to the event's occurrence. The starting point of the target time period corresponds to the event's occurrence minus the first duration.

[0179] Optionally, the first duration can be shorter than the duration of the target time period. The first duration can be a fixed value or dynamically adjusted based on the event type of the target event. For example, the first duration can be 1.5 seconds for all target events; or, for exciting kill events within a game application, the first duration can be 1 second, and for extreme escape events within a game application, the first duration can be 2 seconds.

[0180] As illustrated in Figure 7, when the application's event occurrence instruction is received at the 21st unit of time, the terminal determines the starting point of the target time period as the 14th unit of time based on 7 units of time.

[0181] Step 501B: Based on the time of the event and the second duration, determine the end point of the target time period, which is located after the time of the event.

[0182] The second duration refers to the duration of the audio and video data following the event occurrence. The starting point of the target time period corresponds to the event occurrence time plus the second duration.

[0183] Optionally, the second duration can be shorter than the duration of the target time period. The second duration can be a fixed value or dynamically adjusted based on the event type of the target event. For example, the second duration could be 1.5 seconds for all target events; or 2 seconds for a highlight kill event within a game application, and 1 second for a near-miss escape event within a game application.

[0184] As illustrated in Figure 7, when the application's event occurrence instruction is received at the 21st unit of time, the terminal determines the end point of the target time period as the 28th unit of time based on 7 units of time.

[0185] Step 501C: Based on the start and end points of the target time period, retrieve the event audio and video data within the target time period from the cache queue.

[0186] To prevent historical audio and video data within the first duration from being overwritten by newly written audio and video data (when the cache queue is small), in one possible implementation, the terminal retrieves the event audio and video data within the first duration from the cache queue after determining the start point of the target event time period.

[0187] Since the audio and video data within the second duration after the occurrence of the target event has not yet been generated when the target event occurs, in one possible implementation, the terminal retrieves the event audio and video data within the second duration from the cache queue when the duration of the target event reaches the second duration.

[0188] Of course, in other possible implementations, when the target length of the cache queue is greater than or equal to the duration of the target time period, the terminal can retrieve the audio and video data of the event within the first and second durations from the cache queue when the duration of the target event reaches the second duration (the audio and video data within the first duration will not be overwritten).

[0189] As illustrated in Figure 7, at the moment an event occurs, the terminal retrieves the event audio and video data for the 14th to 20th time units from the cache queue; when the duration of the target event reaches the second duration, the terminal retrieves the event audio and video data for the 22nd to 28th time units from the cache queue. Together with the event audio and video data for the 21st time unit, the terminal retrieves a total of 15 time units of event audio and video data.

[0190] Step 502: Encode the event audio and video data to obtain the event video of the target event.

[0191] In one possible implementation, if the duration of the event audio / video data matches the duration of the live photo, the terminal encodes the event audio / video data to obtain the event video.

[0192] As illustrated in Figure 6, the terminal retrieves the event video data 68 of the target event from the cache queue 71 and encodes it using MediaCodec 65 to obtain the event video 69.

[0193] In another possible implementation, the duration of the event audio / video data corresponding to the target time period retrieved by the terminal from the cache queue is longer than the duration of the live photo. Before generating the event video, the terminal can further filter the event audio / video data to obtain target event audio data that meets the duration requirements of the live photo and can more accurately reflect the target event. Optionally, this step may include the following sub-steps.

[0194] Step 502A: Based on the duration of the live photo, extract the target event audio and video data from the event audio and video data, and match the duration of the target event audio and video data with the duration of the live photo.

[0195] The target event audio and video data is a part of the event audio and video data. For example, if the acquired event audio and video data is 6 seconds long and the duration of the live photo is 3 seconds, the terminal will extract 3 seconds of target event audio and video data from the event audio and video data.

[0196] When acquiring event audio and video data for a fixed duration, incomplete pre-event or post-event sequences may occur for different events, resulting in the generated live photo failing to reflect the full picture of the target event. In this embodiment, the terminal acquires event audio and video data for a longer duration and extracts target event audio and video data that best reflects the full picture of the target event, based on the duration of the live photo, for use in generating the subsequent live photo.

[0197] Regarding the method for extracting audio and video data of the target event, one possible implementation may include the following sub-steps:

[0198] Sub-step 1: Identify the audio and video data of the event to obtain the start and end points of the target event.

[0199] In one possible implementation, the terminal performs content recognition on the event audio and video data to identify the start and end points of the target event contained therein. Specifically, the terminal can identify the start and end points of the target event using a pre-trained event start and end point recognition model (input including event information and event audio and video data).

[0200] As illustrated in Figure 8, when the duration of the live photo is 3 seconds, the terminal does not acquire the audio and video data 1.5 seconds before and 1.5 seconds after the event occurrence time. Instead, it acquires the audio and video data 3 seconds before and 3 seconds after the event occurrence time (T+3) as the event audio and video data for the target event. Before generating the live photo, the terminal identifies this 6-second event audio and video data, determining that the start point of the target event is T+1 and the end point is T+4.

[0201] Sub-step 2: Based on the event start point, event end point, and duration of the live photos, extract the target event audio and video data from the event audio and video data. The target event audio and video data includes the audio and video data between the event start point and the event end point.

[0202] Optionally, when the time interval between the start and end of the event is less than or equal to the duration of the live photo, the target event audio and video data extracted by the terminal includes all audio and video data between the start and end of the event. Furthermore, this target event audio and video data includes audio and video data prior to the start of the event and / or, after the end of the event.

[0203] Optionally, when the time interval between the start and end of the event is greater than the duration of the live photo, the target event audio and video data extracted by the terminal includes part of the audio and video data between the start and end of the event.

[0204] In some embodiments, the terminal can extract the target event audio and video data from the middle between the start and end points of the event based on the duration of the live photo.

[0205] In other embodiments, when the importance of the event starting point is high, the terminal can extract the target event audio and video data between the event starting point and the event ending point based on the duration of the live photo; when the importance of the event ending point is high, the terminal can extract the target event audio and video data between the event starting point and the event ending point based on the duration of the live photo.

[0206] It should be noted that, regardless of the extraction method used, it is necessary to ensure that the extracted target event audio and video data includes the audio and video data at the time the event occurred.

[0207] As illustrated in Figure 7, since the time interval between the start and end of the event is equal to the duration of the live photo (3 seconds), the terminal extracts the audio and video data of the event from T+1 to T+4 seconds as the target event audio and video data. Compared to only acquiring the audio and video data 1.5 seconds before and 1.5 seconds after the event (T+1.5 to T+4.5 seconds), this scheme can acquire the audio and video data for the entire event duration.

[0208] Step 502B: Encode the audio and video data of the target event to obtain the event video of the target event.

[0209] Furthermore, the terminal encodes the extracted target event audio and video data to obtain event videos that meet the duration requirements of live photos.

[0210] In the above embodiments, the duration of the live photos on the terminal can be a preset value, and correspondingly, the duration of the target time period is also a preset value, and the duration of the target time period is longer than the duration of the live photos.

[0211] Because different target events have different durations—for example, in gaming applications, consecutive kill events typically last longer, while extreme escape events typically last shorter—in order to improve the completeness of the target events reflected in the generated live photos, in one possible implementation, the terminal can set different live photo durations for different target events. That is, the duration of the live photos generated for different target events may be different. Correspondingly, the duration of the target time period is set to correspond with the duration of the live photo, thereby ensuring that the duration of the target time period is longer than the duration of the live photo.

[0212] In some embodiments, the target time period and the duration of the live photos are determined based on event information of the target event. This event information may include event type, event name, event importance, etc., which are not limited in this embodiment.

[0213] Optionally, the terminal is configured with a correspondence between event information, the duration of the live photo, and the duration of the target time period. When a target event occurs, the terminal determines the corresponding live photo duration and the target time period duration from the target event. This correspondence is illustrated in Figure 9.

[0214] Of course, in addition to dynamically adjusting the duration of the target time period based on event information, the terminal can further dynamically set the first duration and the second duration of the target time period based on event information, which will not be elaborated here.

[0215] In this embodiment, the terminal caches the audio and video data of the application in a recent period through a cache queue. When there is a need to generate a live photo, the terminal extracts the event audio and video data of the target event from the cache queue and generates the event video based on the event audio and video data. This eliminates the need to record and generate the video throughout the entire application process, which reduces the amount of computation required to implement the live photo generation function and also helps to reduce the storage space occupied.

[0216] In addition, the terminal acquires event audio and video data with a duration longer than the live photo, and before generating the live photo, it identifies the start and end points of the event in the event audio and video data. It then extracts target event audio and video data from the event audio and video data that has the same duration as the live photo and can completely reflect the target event. This data is then used to generate the live photo, which helps to improve the completeness of the event reflected in the generated live photo and improves the quality of the generated live photo.

[0217] Conditions for generating live photos

[0218] Since there is no need to generate live photos in all scenarios, in order to avoid wasting resources due to necessary computation, the terminal performs the steps of obtaining event screenshots and event videos and generating live photos when at least one of the following conditions is met.

[0219] Condition 1: The application belongs to the application list.

[0220] The target application list includes applications that support live photo generation. These applications can be preset or customized by the user.

[0221] For example, the target application list includes game A and game B by default. Users can add ticketing applications to the target application list according to their own needs, so as to trigger the terminal to automatically generate a real-time photo when the ticket is successfully purchased.

[0222] Optionally, the terminal triggers the live photo automatic generation function when the application running in the background is in the target application list; otherwise, it does not trigger the live photo automatic generation function. When the live photo automatic generation function is not triggered, the terminal will not perform the aforementioned screenshot, caching, and application running video recording operations.

[0223] For example, when the application running in the background is game A, the terminal triggers the live photo automatic generation function because game A is in the target application list; however, when the application running in the background is an e-book application, the terminal does not trigger the live photo automatic generation function because the e-book application is not in the target application list.

[0224] Condition 2: The application belongs to the application list and the application is running in the target application scenario.

[0225] The target application scenario is an application scenario that requires the generation of live photos. This target application scenario can be a system default scenario or a scenario that can be customized by the user according to their needs.

[0226] For example, for gaming applications, the target application scenario may include game matches, ranked matches, etc.; for ticketing applications, the target application scenario may include ticket-grabbing scenarios; for video conferencing applications, the target application scenario may include online meeting scenarios. This application does not limit the specific target application scenario.

[0227] Optionally, the application scenario to which the application is currently running can be determined by the application notifying the terminal system, or by the terminal system automatically recognizing the application screen. For example, when the terminal system recognizes that it has entered a specific application interface within the application, it determines that it is running in the target application scenario.

[0228] Optionally, when the application running in the background is in the target application list, the terminal checks whether it is running in the target application scenario. If it is running in the target application scenario, the terminal triggers the live photo automatic generation function; if it is not running in the target application scenario, the terminal does not trigger the live photo automatic generation function. When the live photo automatic generation function is not triggered, the terminal will not perform the aforementioned screenshot, caching, and application running video recording operations.

[0229] In this embodiment, the terminal sets conditions to trigger the automatic generation of live photos, thereby avoiding the waste of processing resources caused by performing screenshot, caching, and application video recording operations in any scenario, and improving the effectiveness of the live photos generated by the terminal.

[0230] Configuration of target events

[0231] In some embodiments, the terminal system provides a target event configuration function, which allows users to set the target events for which live photos need to be generated according to their own needs.

[0232] In one possible implementation, the terminal system provides a configuration entry point for the target event. When a trigger operation is received on the configuration entry point, the terminal displays a configuration interface containing at least one candidate event, which is an event that supports the generation of a live photo.

[0233] Optionally, the candidate event can be the default event, or the candidate event can be determined by the terminal system based on the application information of the installed applications. That is, the candidate event is an event that may occur in the installed applications and supports the generation of live photos.

[0234] Optionally, the candidate event can be displayed as a unit organization, that is, the terminal displays the candidate events corresponding to different applications respectively.

[0235] As illustrated in Figure 10, the live photo configuration interface 1001 provided by the terminal displays several installed applications that support live photo generation (Game A, Game B, Ticketing C, and Meeting D). When the terminal receives a trigger on the setting control 1002 corresponding to Game A, it further displays several candidate events in Game A that support live photo generation (including pentakill, quadrakill, extreme escape, and victory).

[0236] Furthermore, in response to the event configuration operation in the configuration interface, the terminal determines the candidate event indicated by the event configuration operation as the target event.

[0237] As illustrated in Figure 10, users can set the corresponding candidate event as the target event by using the enable control 1003 corresponding to each candidate event.

[0238] Of course, in addition to selecting the target event from the provided candidate events, the terminal can also provide an event customization function in other possible implementations. Through this event customization function, users can describe the event information of a custom target event and / or provide a video of the target event. The terminal then extracts the event features of the custom target event based on the event information and / or the event video, thereby enabling event recognition during application operation based on these event features. This embodiment will not elaborate further.

[0239] In this embodiment, the terminal provides a setting entry for target events, allowing users to set target events themselves, thus improving the flexibility of target event setting.

[0240] Please refer to Figure 11, which shows a structural block diagram of a live photo generation apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0241] The screenshot acquisition module 1101 is used to acquire an event screenshot of a target event within the application, wherein the event screenshot includes the screen image when the target event occurs;

[0242] The video acquisition module 1102 is used to acquire the event video of the target event, wherein the event video is a screen recording video within the event time period corresponding to the target event;

[0243] The generation module 1103 is used to generate a live photo of the target event based on the event screenshot and the event video.

[0244] Optionally, the screenshot acquisition module 1101 is used for:

[0245] Upon receiving an event occurrence command sent by the application, a screenshot is taken of the screen to obtain the event screenshot. The event occurrence command is used to notify that the target event has occurred.

[0246] Upon recognizing the target event within the application, a screenshot is taken of the screen to obtain the event screenshot;

[0247] From the application's running video, video frames are extracted as the event screenshot of the target event. The application running video is a video obtained by screen recording during the application's operation.

[0248] Upon receiving an external command that meets the screenshot criteria, a screenshot is taken of the screen to obtain the event screenshot. The external command includes at least one of voice commands, manual operation commands, physical button commands, and physiological signal commands.

[0249] Optionally, the screenshot acquisition module 1101 is used for:

[0250] During application operation, screen content recognition is performed; if the screen content recognition result indicates the presence of an event visual element corresponding to the target event, the target event within the application is identified.

[0251] During application operation, audio content recognition is performed on the application's audio; if the audio content recognition result indicates the presence of an auditory element corresponding to the target event, the target event within the application is identified.

[0252] Optionally, the screenshot acquisition module 1101 is used for:

[0253] Based on the time of occurrence of the target event, the video frame located at the time of occurrence is extracted from the application's running video as the event screenshot of the target event;

[0254] Perform content recognition on the application running video; if the target event is recognized, extract the video frame containing the target event as the event screenshot of the target event.

[0255] Optionally, the video acquisition module 1102 is used for:

[0256] The event audio and video data of the target event are obtained from the cache queue, which is used to cache audio and video data of the target duration. The audio and video data includes video data corresponding to the screen display during application operation and audio data of the application audio. The event audio and video data are encoded to obtain the event video of the target event.

[0257] The event video of the target event is extracted from the application's running video, which is a video recorded during the application's operation.

[0258] Optionally, the video acquisition module 1102 is used for:

[0259] Using the time of occurrence of the target event as a reference point, the audio and video data of the event within the target time period are obtained from the cache queue, wherein the time of occurrence of the event is within the target time period.

[0260] Optionally, the video acquisition module 1102 is used for:

[0261] Based on the time of the event and the first duration, the starting point of the target time period is determined, and the starting point of the target time period is located before the time of the event.

[0262] Based on the time of the event and the second duration, the end point of the target time period is determined, and the end point of the target time period is located after the time of the event.

[0263] Based on the start and end points of the target time period, the event audio and video data within the target time period are retrieved from the cache queue.

[0264] Optionally, the video acquisition module 1102 is also used for:

[0265] Create the cache queue, which is a circular cache queue;

[0266] During application operation, the audio and video data are cached in the cache queue.

[0267] Optionally, the duration of the target time period is greater than the duration of the live photo;

[0268] Video acquisition module 1102 is used for:

[0269] Based on the duration of the live photo, target event audio and video data is extracted from the event audio and video data, and the duration of the target event audio and video data matches the duration of the live photo.

[0270] The target event audio and video data are encoded to obtain the event video of the target event.

[0271] Optionally, the video acquisition module 1102 is used for:

[0272] Identify the audio and video data of the event to obtain the start and end points of the target event;

[0273] Based on the event start point, the event end point, and the duration of the live photo, the target event audio and video data is extracted from the event audio and video data, which includes audio and video data between the event start point and the event end point.

[0274] Optionally, the target time period and the duration of the live photo are determined based on the event information of the target event, or the target time period and the duration of the live photo are preset values.

[0275] Optionally, the video acquisition module 1102 is used for:

[0276] Using the time of occurrence of the target event as a reference point, the event video of the target event is extracted from the application's running video;

[0277] Perform content recognition on the application running video; based on the content recognition results, extract the event video of the target event from the application running video.

[0278] Optionally, the event screenshots and the event videos are obtained under at least one of the following conditions:

[0279] The application belongs to the target application list;

[0280] The application belongs to the target application list, and the application is running in the target application scenario;

[0281] The target application list includes applications that support live photo generation, and the target application scenario is an application scenario with a need for live photo generation.

[0282] Optionally, the device further includes a configuration module for:

[0283] The configuration interface is displayed, which includes at least one candidate event, which is an event that supports the generation of live photos;

[0284] In response to the event configuration operation in the configuration interface, the candidate event indicated by the event configuration operation is determined as the target event.

[0285] In this embodiment, the terminal can automatically generate live photos of target events occurring within an application. These live photos include a screenshot of the screen when the target event occurs, and a screen recording video of the corresponding event period, allowing users to quickly review the target event. For example, the terminal can generate a live photo corresponding to a highlight event in a game application based on a screenshot of the highlight event and a screen recording video of the highlight event's occurrence, facilitating quick review of the highlight moment. Since the live photos are automatically generated by the terminal, eliminating the need for manual recording or shooting by the user, the generation process is simplified and efficiency is improved.

[0286] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0287] Referring to Figure 12, which is a schematic diagram of the structure of a terminal provided in an exemplary embodiment of this application, the terminal may further include one or more of the following components: a processor 1210 and a memory 1220.

[0288] Optionally, the processor 1210 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1220, and by calling data stored in the memory 1220. Optionally, the processor 1210 can be implemented in at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0289] The processor 1210 can integrate one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), and a baseband chip. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content displayed on the touchscreen; the NPU implements artificial intelligence (AI) functions; and the baseband chip handles wireless communication. It is understood that the baseband chip can also be implemented as a separate chip without being integrated into the processor 1210.

[0290] The memory 1220 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1220 may include a non-transitory computer-readable storage medium. The memory 1220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the electronic device, etc.

[0291] In addition, those skilled in the art will understand that the structure of the terminal shown in the above figures does not constitute a limitation on the terminal. The terminal may include more (e.g., power supply components, display components, sensor components) or fewer components than shown, or combine certain components, or have different component arrangements.

[0292] This application provides a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the live photo generation method as described in the above embodiments.

[0293] On the other hand, this application provides a computer program product, which includes computer instructions. A processor executes the computer instructions to implement the live photo generation method as described in the above embodiments.

[0294] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0295] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating live photos, the method being used on a terminal, the method comprising: Obtain an event screenshot of a target event within the application, the event screenshot including the screen view when the target event occurs; Obtain the event video of the target event, wherein the event video is a screen recording video of the event time period corresponding to the target event; A live photo of the target event is generated based on the event screenshots and the event video.

2. The method according to claim 1, wherein, The acquisition of an event screenshot of a target event within the application includes at least one of the following: Upon receiving an event occurrence command sent by the application, a screenshot is taken of the screen to obtain the event screenshot. The event occurrence command is used to notify that the target event has occurred. Upon recognizing the target event within the application, a screenshot is taken of the screen to obtain the event screenshot; From the application's running video, video frames are extracted as the event screenshot of the target event. The application running video is a video obtained by screen recording during the application's operation. Upon receiving an external command that meets the screenshot criteria, a screenshot is taken of the screen to obtain the event screenshot. The external command includes at least one of voice commands, manual operation commands, physical button commands, and physiological signal commands.

3. The method according to claim 2, wherein, When the event screenshot is taken after identifying the target event within the application, the method includes at least one of the following: During application operation, screen content recognition is performed; if the screen content recognition result indicates the presence of an event visual element corresponding to the target event, the target event within the application is identified. During application operation, audio content recognition is performed on the application's audio. If the audio content recognition result indicates the presence of an auditory element corresponding to the target event, then the target event within the application is identified.

4. The method according to claim 2, wherein, The step of extracting video frames from the application's running video as the event screenshot of the target event includes at least one of the following: Based on the time of occurrence of the target event, the video frame located at the time of occurrence is extracted from the application's running video as the event screenshot of the target event; Perform content recognition on the video of the application running; Upon identifying the target event, the video frame containing the target event is extracted as the event screenshot of the target event.

5. The method according to any one of claims 1 to 4, wherein, The acquisition of the event video of the target event includes at least one of the following: The event audio and video data of the target event are obtained from the cache queue, which is used to cache audio and video data of the target duration. The audio and video data includes video data corresponding to the screen display during application operation and audio data of the application audio. The event audio and video data are encoded to obtain the event video of the target event; The event video of the target event is extracted from the application's running video, which is a video recorded during the application's operation.

6. The method according to claim 5, wherein, The step of retrieving the event audio and video data of the target event from the cache queue includes: Using the time of occurrence of the target event as a reference point, the audio and video data of the event within the target time period are obtained from the cache queue, wherein the time of occurrence of the event is within the target time period.

7. The method according to claim 6, wherein, The step of retrieving the audio and video data of the event within the target time period from the cache queue, using the time of occurrence of the target event as a reference point, includes: Based on the time of the event and the first duration, the starting point of the target time period is determined, and the starting point of the target time period is located before the time of the event. Based on the time of the event and the second duration, the end point of the target time period is determined, and the end point of the target time period is located after the time of the event. Based on the start and end points of the target time period, the event audio and video data within the target time period are retrieved from the cache queue.

8. The method according to claim 7, wherein, The method further includes: Create the cache queue, which is a circular cache queue; During application operation, the audio and video data are cached in the cache queue.

9. The method according to claim 6, wherein, The duration of the target time period is greater than the duration of the live photo; The process of encoding the event audio and video data to obtain the event video of the target event includes: Based on the duration of the live photo, target event audio and video data is extracted from the event audio and video data, and the duration of the target event audio and video data matches the duration of the live photo. The target event audio and video data are encoded to obtain the event video of the target event.

10. The method according to claim 9, wherein, The step of extracting target event audio and video data from the event audio and video data based on the duration of the live photos includes: Identify the audio and video data of the event to obtain the start and end points of the target event; Based on the event start point, the event end point, and the duration of the live photo, the target event audio and video data is extracted from the event audio and video data, which includes audio and video data between the event start point and the event end point.

11. The method according to claim 9, wherein, The target time period and the duration of the live photo are determined based on the event information of the target event, or the target time period and the duration of the live photo are preset values.

12. The method according to claim 5, wherein, The step of extracting the event video of the target event from the application's running video includes at least one of the following: Using the time of occurrence of the target event as a reference point, the event video of the target event is extracted from the application's running video; Perform content recognition on the video of the application running; Based on the content recognition results, the event video of the target event is extracted from the application running video.

13. The method according to any one of claims 1 to 12, wherein, The event screenshots and the event videos are obtained under at least one of the following conditions: The application belongs to the target application list; The application belongs to the target application list, and the application is running in the target application scenario; The target application list includes applications that support live photo generation, and the target application scenario is an application scenario with a need for live photo generation.

14. The method according to any one of claims 1 to 13, wherein, The method further includes: The configuration interface is displayed, which includes at least one candidate event, which is an event that supports the generation of live photos; In response to the event configuration operation in the configuration interface, the candidate event indicated by the event configuration operation is determined as the target event.

15. A live photo generation apparatus, the apparatus comprising: The screenshot acquisition module is used to acquire an event screenshot of a target event within the application, wherein the event screenshot includes the screen image when the target event occurs; The video acquisition module is used to acquire the event video of the target event, wherein the event video is a screen recording video within the event time period corresponding to the target event; The generation module is used to generate a live photo of the target event based on the event screenshot and the event video.

16. A terminal comprising a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the live photo generation method as described in any one of claims 1 to 14.

17. A computer-readable storage medium storing at least one computer instruction, the at least one computer instruction being executed by a processor to implement the live photo generation method as claimed in any one of claims 1 to 14.

18. A computer program product comprising computer instructions, wherein a processor executes the computer instructions to implement the live photo generation method as described in any one of claims 1 to 14.

Images