Panoramic video camera movement method and electronic device

Abstract

Disclosed in the present invention are a panoramic video camera movement method and an electronic device. The panoramic video camera movement method comprises: acquiring a panoramic video to be processed, wherein the panoramic video corresponds to at least one initial playback viewing angle sequence, and each initial playback viewing angle sequence comprises an initial playback viewing angle corresponding to each panoramic video frame; determining a camera movement trajectory matching the initial playback viewing angle sequence; and performing camera movement processing on the panoramic video on the basis of the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein the panoramic camera movement video comprises at least part of the initial playback viewing angle sequence.

Description

A panoramic video camera movement method and electronic device Technical Field

[0001] This application relates to the field of computer technology, and in particular to a panoramic video camera movement method and electronic device. Background Technology

[0002] In related technologies, panoramic video camera movements are typically edited manually by the user, which mainly involves adding keyframes to the panoramic video's timeline and setting the playback angle for each keyframe. This process is complex and time-consuming. Furthermore, the camera movement effects generated by these technologies are relatively simple and lack quality. Summary of the Invention

[0003] This application provides a panoramic video camera movement method and an electronic device.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides a panoramic video camera movement method, the method comprising:

[0006] A panoramic video to be processed is obtained, wherein the panoramic video corresponds to at least one initial playback view sequence, and each initial playback view sequence includes an initial playback view corresponding to each panoramic video frame.

[0007] Determine the camera movement trajectory that matches the initial playback perspective sequence;

[0008] The panoramic video is processed based on the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein the panoramic camera movement video includes at least a portion of the initial playback view sequence.

[0009] At least one embodiment of this application provides an electronic device, including:

[0010] The system includes a processor and a memory interconnected. The memory stores a computer program. When the computer program is executed by the processor, the processor is configured to acquire a panoramic video to be processed, the panoramic video containing an initial playback view sequence composed of the initial playback view of each panoramic video frame; determine a camera movement template matching the initial playback view sequence; the camera movement template indicates the relative positional relationship between the lens and the initial playback view; and perform camera movement processing on the panoramic video based on the camera movement template to obtain a corresponding panoramic camera movement video.

[0011] This invention provides a computer-readable storage medium, comprising: the computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the panoramic video camera movement method provided in the first aspect of this invention.

[0012] This application embodiment acquires a panoramic video to be processed, which corresponds to at least one initial playback viewpoint sequence. Each initial playback viewpoint sequence includes an initial playback viewpoint corresponding to each panoramic video frame. A camera movement trajectory matching the initial playback viewpoint sequence is determined, and the panoramic video is processed based on the camera movement trajectory to obtain a panoramic camera movement video associated with the initial playback viewpoint sequence. The obtained panoramic camera movement video includes at least a portion of the initial playback viewpoint sequence. In this scheme, different camera movement effects can be obtained for the same panoramic video based on different initial playback viewpoints, increasing the richness of camera movement. This not only lowers the barrier for users to achieve complex camera movement effects, but also allows for better expression of the user's editing intentions by matching camera movement trajectories based on the initial playback viewpoint. Furthermore, different camera movement trajectories also correspond to different camera movement effects. Through the application of camera movement trajectories, users can achieve richer camera movements. Attached Figure Description

[0013] Figure 1 is a schematic diagram of the implementation process of a panoramic video camera movement method provided in an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of a panoramic video preview interface provided in an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a camera movement effect provided by an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of another camera movement effect provided by an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of a camera movement editing point provided in an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Figure 1 is a schematic flowchart illustrating the implementation of a panoramic video camera movement method according to an embodiment of the present invention. The executing entity of the panoramic video camera movement method is an electronic device. Referring to Figure 1, the panoramic video camera movement method includes:

[0021] S101, Obtain the panoramic video to be processed, wherein the panoramic video corresponds to at least one initial playback view sequence, and each initial playback view sequence includes an initial playback view corresponding to each panoramic video frame.

[0022] The panoramic video data to be processed can be obtained by a panoramic camera, which consists of two or more fisheye lenses, and the resulting panoramic video is the original spherical video.

[0023] The panoramic video to be processed can be a segment of the original panoramic video, and the specific duration can be defined by the user.

[0024] The playback perspective corresponds to a shooting shot in the panoramic video. Different shots correspond to different shooting perspectives, as well as different camera movement trajectories, speeds, durations, and shooting subjects.

[0025] A panoramic video frame can include multiple shooting perspectives. One of these shooting perspectives can be used as the initial playback perspective, and then the initial playback perspectives of each panoramic video frame can form an initial playback perspective sequence.

[0026] Panoramic videos can include multiple initial playback view sequences. For example, a panoramic video frame may include multiple targets (including people, main characters, buildings, sculptures, landscapes, vehicles, etc.). An initial playback view sequence can be formed based on the shooting perspective of each target in each panoramic video frame. Multiple targets correspond to multiple initial playback view sequences.

[0027] S102, determine the camera movement trajectory that matches the initial playback viewpoint sequence.

[0028] The camera movement trajectory can represent the movement path of a virtual camera lens in three-dimensional space. The camera movement trajectory can be composed of multiple virtual camera poses (camera pose is the position of the camera in space and the orientation of the camera). The camera movement trajectory of a video can be understood as the trajectory of the lens movement during the shooting process of the virtual camera, that is, the camera movement trajectory of the virtual camera in the panoramic video. The camera movement trajectory is composed of a series of trajectory points, and each trajectory point corresponds to a target playback viewpoint.

[0029] In some embodiments, multiple camera movement trajectories can be preset, and a camera movement trajectory that matches the initial playback viewpoint sequence can be determined from among the preset multiple camera movement trajectories.

[0030] In some embodiments, a matching camera movement trajectory can be generated based on the initial playback perspective sequence.

[0031] In some embodiments, multiple camera movement templates can be preset. By determining the camera movement template that matches the initial playback viewpoint sequence, the camera movement trajectory that matches the initial playback viewpoint sequence can be determined based on the camera movement template.

[0032] Among them, a camera movement trajectory can be matched for each initial playback perspective sequence.

[0033] Users can also select one of at least one initial playback view sequence and then determine the camera movement trajectory that matches the initial playback view sequence selected by the user.

[0034] S103, perform camera movement processing on the panoramic video based on the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein the panoramic camera movement video includes at least a portion of the initial playback view sequence.

[0035] A panoramic video frame includes footage from multiple shooting angles. The playback angle of each panoramic video frame can be determined based on the camera movement trajectory. By playing the panoramic video based on the camera movement trajectory, the corresponding panoramic camera movement video can be obtained.

[0036] By performing camera movement processing on the panoramic video based on the camera movement trajectory matched with the initial playback viewpoint sequence, a panoramic camera movement video can be obtained that correlates the camera movement viewpoint with the initial playback viewpoint sequence. The resulting panoramic camera movement video includes at least a portion of the initial playback viewpoint sequence, thus highlighting the content within that sequence.

[0037] This application embodiment acquires a panoramic video to be processed, which corresponds to at least one initial playback viewpoint sequence. Each initial playback viewpoint sequence includes an initial playback viewpoint corresponding to each panoramic video frame. A camera movement trajectory matching the initial playback viewpoint sequence is determined, and the panoramic video is processed based on the camera movement trajectory to obtain a panoramic camera movement video associated with the initial playback viewpoint sequence. The obtained panoramic camera movement video includes at least a portion of the initial playback viewpoint sequence. In this scheme, different camera movement effects can be obtained for the same panoramic video based on different initial playback viewpoints, increasing the richness of camera movement. This not only lowers the barrier for users to achieve complex camera movement effects, but also allows for better expression of the user's editing intentions by matching camera movement trajectories based on the initial playback viewpoint. Furthermore, different camera movement trajectories also correspond to different camera movement effects. Through the application of camera movement trajectories, users can achieve richer camera movements.

[0038] In one embodiment, determining the camera movement trajectory matching the initial playback perspective sequence includes:

[0039] Determine a camera movement template that matches the initial playback viewpoint sequence, the camera movement template including: information indicating the relative positional relationship between the lens and the initial playback viewpoint;

[0040] Based on the camera movement template, a camera movement trajectory matching the initial playback perspective sequence is determined.

[0041] The camera movement template in this embodiment differs from those in related technologies. Related technology camera movement templates indicate the absolute position of the lens, specifying the direction the lens faces within a panoramic video frame. For example, in keyframe 1 at t=0, the lens faces a position with pitch = 45° and yaw = 180°; in keyframe 2 at t=1, the lens faces a position with pitch = -45° and yaw = 180°. Here, yaw refers to the horizontal angle of the lens, and pitch refers to the vertical angle. It is evident that the camera movement templates in related technologies are not associated with the initial playback viewpoint of the panoramic video.

[0042] The camera movement template in this embodiment includes information indicating the relative positional relationship between the lens and the initial playback viewpoint. For example, in keyframe 1 of the panoramic video, the initial playback viewpoint is oriented towards a position with pitch = 45° and yaw = 180°. The camera movement template indicates the positional change of the lens relative to the initial playback viewpoint in keyframe 1. For example, relative to the initial playback viewpoint, the lens's pitch shifts upward by 90 degrees, while the yaw remains unchanged.

[0043] Here, a camera movement recommendation model can be used to output a camera movement template that matches the initial playback viewpoint sequence, or users can manually select a camera movement template.

[0044] Based on the information in the camera movement template indicating the relative positional relationship between the lens and the initial playback viewpoint, the playback viewpoint of each panoramic video frame in the panoramic video can be determined, and a camera movement trajectory matching the initial playback viewpoint sequence can be obtained based on the playback viewpoint of each panoramic video frame.

[0045] In one embodiment, the camera movement template includes: multiple keyframes, each keyframe corresponding to position information and viewpoint information, the viewpoint information including: the relative positional relationship between the lens and the initial playback viewpoint.

[0046] The keyframe position information indicates the location of keyframes in the panoramic video. This position information can be represented by the frame number or a time point. For example, assuming the panoramic video comprises 100 frames, the position information of keyframe 1 in the camera movement template indicates that the 5th frame of the panoramic video is keyframe 1, and the position information of keyframe 2 in the camera movement template indicates that the 50th frame of the panoramic video is keyframe 2, and so on. If represented by time points, for example, the first panoramic video keyframe is at time point 00:01:10, the second panoramic video keyframe is at time point 00:20:00, and the third panoramic video keyframe is at time point 01:40:00.

[0047] The viewpoint information of a keyframe includes the relative positional relationship between the camera and the initial playback viewpoint. For example, the viewpoint information of a keyframe in a camera movement template is represented as follows:

[0048] Keyframe 1: pitch_delta = 90°, yaw_delta = 0°.

[0049] Keyframe 2: pitch_delta = 0°, yaw_delta = 0°.

[0050] The meaning of the perspective information of the keyframes above is as follows: At time t=0, in keyframe 1, the pitch is shifted upward by 90 degrees relative to the initial playback perspective, while the yaw is not shifted; at time t=1, in keyframe 2, the pitch is not shifted and the yaw is not shifted relative to the initial playback perspective.

[0051] When applying the camera movement template, keyframe 1 at time t=0 first obtains the pitch and yaw values ​​of the initial playback view at t=0, and then shifts the pitch by 90° and the yaw by 0° respectively; keyframe 2 at time t=1 first obtains the pitch and yaw values ​​of the initial playback view at t=1, and then shifts the pitch by 0° and the yaw by 0° respectively.

[0052] In addition, keyframes can also include attribute information, which can include the target type corresponding to the keyframe (target type can be a person, protagonist, building, sculpture, landscape, vehicle, etc.).

[0053] The attribute information can also indicate whether the current camera movement keyframe uses an absolute or relative position. For example, the viewpoint information for keyframe 1 in the camera movement template includes: pitch = 0°, yaw = 20°. If the attribute information indicates that keyframe 1 uses a relative position, it means that relative to the initial playback viewpoint, the pitch is offset by 0 degrees and the yaw by 20 degrees, thus obtaining the camera movement viewpoint of keyframe 1 in the panoramic video. If the attribute information indicates that keyframe 1 uses an absolute position, then the camera movement viewpoint of keyframe 1 in the panoramic video is pitch = 0°, yaw = 20°.

[0054] In one embodiment, determining the camera movement trajectory matching the initial playback viewpoint sequence based on the camera movement template includes:

[0055] Based on the location information corresponding to the keyframes, the panoramic video frames corresponding to each keyframe in the camera movement template are determined from the panoramic video as panoramic video keyframes.

[0056] Based on the perspective information and the initial playback perspective corresponding to the panoramic video keyframe, the target playback perspective corresponding to the panoramic video keyframe is determined.

[0057] Interpolation processing is performed on other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframe to obtain the target playback viewpoint corresponding to each panoramic video frame.

[0058] The camera movement trajectory is obtained based on the target playback viewpoint corresponding to each panoramic video frame.

[0059] For example, the location information corresponding to keyframe 1 indicates that the 5th frame of the panoramic video is a panoramic video keyframe. If the camera movement template has 5 keyframes, then 5 panoramic video keyframes can be determined in the panoramic video.

[0060] For example, if the viewpoint information of keyframe 1 is: pitch_delta = 90°, yaw_delta = 0°, then first obtain the pitch and yaw values ​​of the initial playback viewpoint of the panoramic video keyframe 1, and then offset the pitch by 90° and the yaw by 0° respectively, so as to obtain the target playback viewpoint corresponding to the panoramic video keyframe 1.

[0061] In some embodiments, the viewpoint information of each keyframe may further include attribute information indicating whether the current camera movement keyframe uses an absolute or relative position. For example, the viewpoint information of keyframe 1 is: pitch_delta = 90°, yaw_delta = 0°. If the attribute information of keyframe 1 indicates a relative position, the initial playback viewpoint pitch and yaw values ​​of the panoramic video keyframe 1 are first obtained, and then the pitch is offset by 90° and the yaw by 0° respectively to obtain the target playback viewpoint corresponding to the panoramic video keyframe 1. If the attribute information of keyframe 1 indicates an absolute position, then the target playback viewpoint corresponding to the panoramic video keyframe 1 is pitch_delta = 90°, yaw_delta = 0°.

[0062] Based on the target playback viewpoint corresponding to the keyframe of the panoramic video, interpolation processing can be performed on other panoramic video frames to obtain the target playback viewpoint corresponding to each panoramic video frame.

[0063] Interpolation processing includes two methods: linear interpolation and spherical interpolation. For example, the target playback angles of panoramic video keyframe 1 and panoramic video keyframe 2 are known. To achieve a smooth transition between panoramic video keyframe 1 and panoramic video keyframe 2, the target playback angle corresponding to each panoramic video frame between keyframe 1 and panoramic video keyframe 2 can be calculated using linear or spherical functions based on the target playback angles of panoramic video keyframe 1 and panoramic video keyframe 2.

[0064] In one embodiment, the step of interpolating other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframe to obtain the target playback viewpoint corresponding to each panoramic video frame includes:

[0065] Obtain the position information corresponding to the target playback viewpoint of each panoramic video keyframe.

[0066] Based on the position information corresponding to the target playback viewpoint of two adjacent panoramic video keyframes, interpolation processing is performed on the panoramic video frame located between the two panoramic video keyframes to obtain the target playback viewpoint corresponding to each intermediate panoramic video frame.

[0067] The positional information corresponding to the target playback viewpoint can include parameters such as horizontal angle (yaw), vertical angle (pitch), rotation angle (roll), field of view (Fov), and projection distance. The field of view refers to the angle formed by the two edges of the maximum range of the image after the panoramic video is projected; the projection distance refers to the distance from the center of the panoramic video projection to the projection surface.

[0068] For example, the target playback viewpoint of panoramic video keyframe 1 corresponds to pitch = 45° and yaw = 180°, while the target playback viewpoint of panoramic video keyframe 2 corresponds to pitch = -45° and yaw = 180°. Based on the position information of the target playback viewpoints of panoramic video keyframes 1 and 2, interpolation processing is performed on the panoramic video frames between them to obtain the target playback viewpoint for each intermediate panoramic video frame, thus achieving a smooth transition between panoramic video keyframes 1 and 2.

[0069] In one embodiment, determining the camera movement template matching the initial playback perspective sequence includes:

[0070] Determine the keyframe data of the initial playback view sequence, wherein the keyframe data includes keyframe positions and attributes;

[0071] Based on the keyframe data of the initial playback view sequence and the camera movement template recommendation model, a camera movement template matching the initial playback view sequence is determined.

[0072] Here, if the camera movement template is output using a camera movement template recommendation model, then a corresponding number of keyframes can be extracted from the initial playback view sequence based on the number of keyframes defined by the camera movement template recommendation model. For example, if the camera movement template recommendation model requires that the input initial playback view sequence have a maximum of 3 keyframes, then 3 frames can be evenly extracted from the initial playback view sequence as keyframes.

[0073] In exceptions, if the camera movement template is manually selected by the user, the keyframes of the initial playback view sequence can be determined based on the number of keyframes defined in the camera movement template and the time percentage corresponding to each keyframe (defined by the camera movement template). For example, if the length of the initial playback view sequence is 100 frames, and the camera movement template has 5 keyframes with time percentages of 1%-25%-50%-75%-100% respectively, then the keyframes in the initial playback view sequence are frames 1, 25, 50, 75, and 100.

[0074] Keyframe data includes keyframe position and attributes. The keyframe position can refer to the position of the keyframe in the initial playback view sequence (which can be represented by frame number or sequence number). The attributes can include the view attribute and target attribute of the keyframe. The view attribute includes playback view parameters such as asteroid, fixed FOV, fixed distance, fixed pitch, and yaw (indicating that the current keyframe uses an absolute position). The target attribute can include the target type of the target included in the current keyframe, such as people, protagonist, buildings, sculptures, landscapes, vehicles, etc.

[0075] The camera movement template recommendation model can be implemented and trained using methods such as Convolutional Neural Networks (CNNs) or Transformers. After training to obtain a usable camera movement template recommendation model, the keyframe data of the initial playback viewpoint sequence is input into the camera movement template recommendation model to obtain the camera movement template output by the model.

[0076] In one embodiment, a user-defined camera movement template can be obtained and stored in the user's camera movement library. The camera movement template includes: the positions of multiple keyframes, the viewpoint corresponding to each keyframe, and the speed between adjacent keyframes.

[0077] In one embodiment, the camera movement template further includes: camera movement speed;

[0078] The camera movement processing of the panoramic video based on the camera movement trajectory includes:

[0079] The camera movement speed between two adjacent keyframes in the camera movement template is obtained, and the camera movement is performed based on the camera movement speed and the camera movement trajectory.

[0080] By playing panoramic videos based on camera movement speed and trajectory, a corresponding panoramic video can be obtained.

[0081] In one embodiment, determining a camera movement template matching the initial playback viewpoint sequence based on the keyframe data and camera movement template recommendation model of the initial playback viewpoint sequence includes:

[0082] Extract several panoramic video frames from the panoramic video;

[0083] The keyframe data of the panoramic video frames and the initial playback viewpoint sequence are used as input to the camera movement template recommendation model to obtain the output camera movement template.

[0084] Here, several extracted panoramic video frames, along with the keyframe positions and attributes of the initial playback view sequence, can be used as input to the camera movement template recommendation model. The camera movement template recommendation model outputs a camera movement template that matches the initial playback view sequence based on the panoramic video frames and the keyframe positions and attributes of the initial playback view sequence.

[0085] The attribute information corresponding to the keyframes of the camera movement template can include the target type of the keyframe (the target type can be a person, protagonist, building, sculpture, landscape, vehicle, etc.). Therefore, when the camera movement template recommendation model outputs the camera movement template, it can refer to the target type of the target included in the keyframes of the initial playback view sequence to output the matching camera movement template.

[0086] The number of panoramic video frames extracted from the panoramic video can be a pre-set number, such as 10 frames, which means 10 panoramic video frames can be randomly extracted from the panoramic video.

[0087] Using the extracted panoramic video frames and keyframe data from the initial playback viewpoint sequence as input to the camera movement template recommendation model can make the output of the camera movement template recommendation model more consistent with the current panoramic video.

[0088] In one embodiment, acquiring the panoramic video to be processed includes:

[0089] Obtain the selected video time range or starting time point for camera movement processing from the original panoramic video;

[0090] Based on the selected video time range or starting time point, a panoramic video segment is determined for camera movement processing, and the panoramic video segment is used as the panoramic video to be processed.

[0091] Here, users can select a single panoramic video segment from the original panoramic video for camera movement processing. Users can choose a video time range and extract the panoramic video segment from the original video based on that range. Alternatively, users can select a start time point and determine the end time point based on the duration of the camera movement template, thus defining the panoramic video segment to be processed.

[0092] In one embodiment, the method further includes one of the following:

[0093] Obtain the target object selected by the user in the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence;

[0094] Identify the target object closest to the set center viewpoint in the preview interface of the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence.

[0095] For example, a panoramic video may contain multiple target objects (the target objects can be people, main characters, buildings, sculptures, landscapes, vehicles, etc.), and the video tracking sequence of each target object can be used as an initial playback view sequence.

[0096] For example, as shown in Figure 2, which is a schematic diagram of a panoramic video preview interface, the panoramic video preview interface shown in Figure 2 includes target 1, target 2 and target 3. The user can select target 1 as the target object and use the target object tracking sequence corresponding to target 1 as the initial playback view sequence.

[0097] Alternatively, in the panoramic video preview interface, the target object tracking sequence corresponding to the target object closest to the set center viewpoint can be used as the initial playback viewpoint sequence. In the panoramic video preview interface shown in Figure 2, target 1 is the target object closest to the set center viewpoint, so the target object tracking sequence corresponding to target 1 can be used as the initial playback viewpoint sequence.

[0098] After selecting the target object, running the target tracking algorithm on the panoramic video will produce a target object tracking sequence.

[0099] After obtaining the target object tracking sequence, the keyframes in the target object tracking sequence can be determined, and the keyframe data of the target object tracking sequence can be obtained.

[0100] If the camera movement template is output using a camera movement template recommendation model, then a corresponding number of keyframes can be extracted from the target object tracking sequence based on the number of keyframes defined by the camera movement template recommendation model. For example, if the camera movement template recommendation model requires that the input target object tracking sequence has a maximum of 3 keyframes, then 3 frames can be evenly extracted from the target object tracking sequence as keyframes.

[0101] Additionally, if the camera movement template is manually selected by the user, the keyframes of the target object tracking sequence can be determined based on the number of keyframes defined in the template and the time percentage corresponding to each keyframe (defined by the template). For example, if the target object tracking sequence is 100 frames long and the camera movement template has 5 keyframes with time percentages of 1%-25%-50%-75%-100%, then the keyframes in the target object tracking sequence would be frames 1, 25, 50, 75, and 100. 。

[0102] In this embodiment, the target object tracking sequence is used as the initial playback view sequence. A camera movement template that matches the target object tracking sequence is determined. By using the camera movement template that matches the target object tracking sequence, a panoramic camera movement video with target object tracking effect can be generated.

[0103] As shown in Figure 3, during the camera movement process of the related technology, from keyframe 1 to keyframe 2, the related technology cannot associate with the target object and cannot generate a panoramic camera movement video with target object tracking effect.

[0104] In this embodiment of the application, the camera movement is performed using a camera movement template that matches the target object tracking sequence. As shown in Figure 4, the camera movement from keyframe 1 to keyframe 2 can generate a panoramic camera movement video with target object tracking effect.

[0105] During camera movement processing, the viewpoint position and field of view of the target object in the tracking sequence can be automatically modified according to the time, positional relationship and attributes of each keyframe in the camera movement template, generating actual keyframes, and then the keyframes are strung together to form a camera movement animation, generating a panoramic camera movement video with tracking effect.

[0106] In one embodiment, after performing camera movement processing on the panoramic video based on the camera movement trajectory to obtain the corresponding panoramic camera movement video, the method further includes:

[0107] Display at least one camera movement editing point in the panoramic video, the camera movement editing point being used to allow the user to modify the camera movement angle of the corresponding panoramic video frame;

[0108] In response to the user's modification of the camera angle at the camera movement editing point, a modified panoramic camera movement video is generated, the modified panoramic camera movement video including: the modified camera angle.

[0109] For example, a user can select one of the camera movement editing points and modify the camera angle at that point. In response to the user's modification of the camera angle at the camera movement editing point, a modified panoramic camera movement video is generated based on the camera movement template. Specifically, the camera movement trajectory is regenerated based on the camera movement template, and the panoramic video clip is moved according to the regenerated camera movement trajectory to obtain the modified panoramic camera movement video. The modified panoramic camera movement video includes the modified camera angle.

[0110] Based on the position and number of camera movement editing points defined by the camera movement template, or the position and number of salient targets in the panoramic camera movement video, at least one camera movement editing point in the panoramic camera movement video is determined.

[0111] As shown in Figure 5, there are multiple camera movement editing points. Users can select one of these camera movement editing points and modify the target playback angle of the corresponding panoramic video frame at the camera movement editing point. Then, the camera movement video is regenerated based on the modified target playback angle.

[0112] In one embodiment, determining the camera movement template matching the initial playback perspective sequence includes:

[0113] The camera movement template selected by the user from multiple candidate camera movement templates is used as the matched camera movement template.

[0114] The candidate camera movement templates here can be multiple camera movement templates output by the camera movement template recommendation model, or multiple camera movement templates from the user's camera movement library. The user can select one camera movement template from the multiple candidate camera movement templates as the camera movement template that matches the initial playback viewpoint sequence.

[0115] In another embodiment, recommended camera movement templates can be presented to the user in order of recommendation confidence level, or the camera movement template with the highest confidence level can be directly used as the matching camera movement template. This saves users time in selecting camera movement templates and helps them quickly obtain the optimal panoramic video camera movement effect.

[0116] In one embodiment, target tracking is first performed on the panoramic video to generate one or more target object tracking sequences. The user selects one of the target object tracking sequences on the front-end interface, or the system automatically matches the nearest target object tracking sequence based on the user's input starting viewpoint. The user then selects a camera movement template. The back-end system automatically determines the target playback viewpoint of each video frame in the target object tracking sequence based on the position and viewpoint information corresponding to each keyframe of the camera movement template. Then, it obtains the camera movement trajectory based on the target playback viewpoint of each video frame in the target object tracking sequence, and performs camera movement on the panoramic video according to the camera movement trajectory to obtain a panoramic video with tracking effects.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0118] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0119] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0120] In addition, in the embodiments of the present invention, "first," "second," etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0121] To implement the method of the embodiments of this application, an electronic device is also provided. Figure 6 is a schematic diagram of the hardware composition structure of the electronic device according to an embodiment of this application. As shown in Figure 6, the electronic device includes:

[0122] A communication interface enables information exchange with other devices, such as network devices.

[0123] The processor is connected to the communication interface to enable information interaction with other devices;

[0124] The memory stores a computer program, which, when executed by the processor, configures the processor to acquire a panoramic video to be processed, the panoramic video corresponding to at least one initial playback view sequence, each initial playback view sequence including an initial playback view corresponding to each panoramic video frame; determine a camera movement trajectory matching the initial playback view sequence; and perform camera movement processing on the panoramic video based on the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein the panoramic camera movement video includes at least a portion of the initial playback view sequence.

[0125] In one embodiment, the processor is configured to determine a camera movement template matching the initial playback view sequence, the camera movement template including: information indicating the relative positional relationship between the lens and the initial playback view; and to determine a camera movement trajectory matching the initial playback view sequence based on the camera movement template.

[0126] In one embodiment, the camera movement template includes: multiple keyframes, each keyframe corresponding to position information and viewpoint information, the viewpoint information including: the relative positional relationship between the lens and the initial playback viewpoint.

[0127] In one embodiment, the processor is configured to determine panoramic video frames corresponding to each keyframe in the camera movement template from the panoramic video based on the position information corresponding to the keyframes, and to determine the target playback viewpoint corresponding to the panoramic video keyframes based on the viewpoint information and the initial playback viewpoint corresponding to the panoramic video keyframes; to perform interpolation processing on other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframes to obtain the target playback viewpoint corresponding to each panoramic video frame; and to obtain the camera movement trajectory based on the target playback viewpoint corresponding to each panoramic video frame.

[0128] In one embodiment, the processor is configured to acquire position information corresponding to the target playback viewpoint of each panoramic video keyframe; and based on the position information corresponding to the target playback viewpoint of two adjacent panoramic video keyframes, to perform interpolation processing on the panoramic video frame located between the two panoramic video keyframes to obtain the target playback viewpoint corresponding to each intermediate panoramic video frame.

[0129] In one embodiment, the initial playback view sequence includes a target object tracking sequence.

[0130] In one embodiment, the processor is configured to determine keyframe data of the initial playback view sequence, the keyframe data including keyframe positions and attributes; and based on the keyframe data of the initial playback view sequence and a camera movement template recommendation model, determine a camera movement template that matches the initial playback view sequence.

[0131] In one embodiment, the processor is configured to extract a plurality of panoramic video frames from the panoramic video; use the plurality of panoramic video frames and keyframe data of the initial playback viewpoint sequence as input to the camera movement template recommendation model, and obtain the output camera movement template.

[0132] In one embodiment, the processor is configured to acquire a selected video time range or starting time point for camera movement processing from the original panoramic video; determine a panoramic video segment for camera movement processing based on the selected video time range or starting time point, and use the panoramic video segment as the panoramic video to be processed.

[0133] In one embodiment, the processor is configured to perform one of the following:

[0134] Obtain the target object selected by the user in the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence;

[0135] Identify the target object closest to the set center viewpoint in the preview interface of the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence.

[0136] In one embodiment, the processor is configured to display at least one camera movement edit point in the panoramic video, the camera movement edit point being used to allow a user to modify the camera movement angle of the corresponding panoramic video frame; in response to the user's modification of the camera movement angle at the camera movement edit point, a modified panoramic video movement is generated, the modified panoramic video movement including the modified camera movement angle.

[0137] In one embodiment, the processor is configured to use the camera movement template selected by the user from a plurality of candidate camera movement templates as the matched camera movement template.

[0138] In one embodiment, the camera movement template further includes a camera movement speed, and the processor is configured to acquire the camera movement speed between two adjacent keyframes in the camera movement template, and perform camera movement based on the camera movement speed and the camera movement trajectory.

[0139] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system. It can be understood that the bus system is used to achieve communication and connection between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus systems in Figure 6.

[0140] The memory in this application embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0141] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM) and Synchronous Static Random Access Memory (SSRAM).

[0142] Memory), Dynamic Random Access Memory (DRAM)

[0143] The memories described in this application are intended to include, but are not limited to, these and any other suitable types of memories. These include: Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).

[0144] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0145] Optionally, when the processor executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0146] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor of an electronic device to perform the steps described in the method of this application embodiment.

[0147] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory storing a computer program, which can be executed by a processor of an electronic device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0149] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0151] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0152] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0153] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0154] In addition, in this application example, terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A panoramic video camera movement method, comprising: A panoramic video to be processed is obtained, wherein the panoramic video corresponds to at least one initial playback view sequence, and each initial playback view sequence includes an initial playback view corresponding to each panoramic video frame. Determine the camera movement trajectory that matches the initial playback perspective sequence; The panoramic video is processed based on the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein the panoramic camera movement video includes at least a portion of the initial playback view sequence.

2. The method according to claim 1, wherein, The determination of the camera movement trajectory matching the initial playback viewpoint sequence includes: Determine a camera movement template that matches the initial playback viewpoint sequence, the camera movement template including: information indicating the relative positional relationship between the lens and the initial playback viewpoint; Based on the camera movement template, a camera movement trajectory matching the initial playback perspective sequence is determined.

3. The method according to claim 2, wherein, The camera movement template includes multiple keyframes, each keyframe corresponding to position information and viewpoint information. The viewpoint information includes the relative positional relationship between the lens and the initial playback viewpoint.

4. The method according to claim 3, wherein, The step of determining the camera movement trajectory matching the initial playback viewpoint sequence based on the camera movement template includes: Based on the location information corresponding to the keyframes, the panoramic video frames corresponding to each keyframe in the camera movement template are determined from the panoramic video as panoramic video keyframes. Based on the perspective information and the initial playback perspective corresponding to the panoramic video keyframe, the target playback perspective corresponding to the panoramic video keyframe is determined. Interpolation processing is performed on other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframe to obtain the target playback viewpoint corresponding to each panoramic video frame. The camera movement trajectory is obtained based on the target playback viewpoint corresponding to each panoramic video frame.

5. The method according to claim 4, wherein, The step of interpolating other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframe to obtain the target playback viewpoint corresponding to each panoramic video frame includes: Obtain the position information corresponding to the target playback viewpoint of each panoramic video keyframe. Based on the position information corresponding to the target playback viewpoint of two adjacent panoramic video keyframes, interpolation processing is performed on the panoramic video frame located between the two panoramic video keyframes to obtain the target playback viewpoint corresponding to each intermediate panoramic video frame.

6. The method according to claim 1, wherein, The initial playback view sequence includes a target object tracking sequence.

7. The method according to claim 1, wherein, The step of determining the camera movement template that matches the initial playback perspective sequence includes: Determine the keyframe data of the initial playback view sequence, wherein the keyframe data includes keyframe positions and attributes; Based on the keyframe data of the initial playback view sequence and the camera movement template recommendation model, a camera movement template matching the initial playback view sequence is determined.

8. The method according to claim 7, wherein, The step of determining a camera movement template matching the initial playback viewpoint sequence based on the keyframe data and camera movement template recommendation model of the initial playback viewpoint sequence includes: Extract several panoramic video frames from the panoramic video; The keyframe data of the panoramic video frames and the initial playback viewpoint sequence are used as input to the camera movement template recommendation model to obtain the output camera movement template.

9. The method according to claim 1, wherein, The acquisition of the panoramic video to be processed includes: Obtain the selected video time range or starting time point for camera movement processing from the original panoramic video; Based on the selected video time range or starting time point, a panoramic video segment is determined for camera movement processing, and the panoramic video segment is used as the panoramic video to be processed.

10. The method according to claim 1, wherein, The method further includes one of the following: Obtain the target object selected by the user in the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence; Identify the target object closest to the set center viewpoint in the preview interface of the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence.

11. The method according to claim 1, wherein, After performing camera movement processing on the panoramic video based on the camera movement trajectory to obtain the corresponding panoramic camera movement video, the method further includes: Display at least one camera movement editing point in the panoramic video, the camera movement editing point being used to allow the user to modify the camera movement angle of the corresponding panoramic video frame; In response to the user's modification of the camera angle at the camera movement editing point, a modified panoramic camera movement video is generated, the modified panoramic camera movement video including: the modified camera angle.

12. The method according to claim 2, wherein, The step of determining the camera movement template that matches the initial playback perspective sequence includes: The camera movement template selected by the user from multiple candidate camera movement templates is used as the matched camera movement template.

13. The method according to claim 2, wherein, The camera movement template also includes: camera movement speed; The camera movement processing of the panoramic video based on the camera movement trajectory includes: The camera movement speed between two adjacent keyframes in the camera movement template is obtained, and the camera movement is performed based on the camera movement speed and the camera movement trajectory.

14. An electronic device, comprising a memory and a processor, the memory storing a computer program, wherein when the computer program is executed by the processor, the processor is configured to acquire a panoramic video to be processed, the panoramic video corresponding to at least one initial playback view sequence, each initial playback view sequence including an initial playback view corresponding to each panoramic video frame; determine a camera movement trajectory matching the initial playback view sequence; and perform camera movement processing on the panoramic video based on the camera movement trajectory to obtain a corresponding panoramic camera movement video, wherein... The panoramic video includes at least a portion of the initial playback view sequence.

15. The electronic device according to claim 14, wherein, The processor is configured to determine a camera movement template that matches the initial playback viewpoint sequence, the camera movement template including: information indicating the relative positional relationship between the lens and the initial playback viewpoint; and to determine a camera movement trajectory that matches the initial playback viewpoint sequence based on the camera movement template.

16. The electronic device according to claim 15, wherein the camera movement template comprises: Multiple keyframes, each keyframe corresponding to position information and viewpoint information, the viewpoint information including: the relative positional relationship between the camera and the initial playback viewpoint.

17. The electronic device according to claim 16, wherein, The processor is configured to determine panoramic video frames corresponding to each keyframe in the camera movement template from the panoramic video based on the position information corresponding to the keyframes, as panoramic video keyframes. Based on the perspective information and the initial playback perspective corresponding to the panoramic video keyframe, the target playback perspective corresponding to the panoramic video keyframe is determined. Interpolation processing is performed on other panoramic video frames based on the target playback viewpoint corresponding to the panoramic video keyframe to obtain the target playback viewpoint corresponding to each panoramic video frame. The camera movement trajectory is obtained based on the target playback viewpoint corresponding to each panoramic video frame.

18. The electronic device according to claim 17, wherein, The processor is configured to acquire position information corresponding to the target playback viewpoint of each panoramic video keyframe; based on the position information corresponding to the target playback viewpoint of two adjacent panoramic video keyframes, interpolation processing is performed on the panoramic video frame located between the two panoramic video keyframes to obtain the target playback viewpoint corresponding to each intermediate panoramic video frame.

19. The electronic device according to claim 14, wherein, The initial playback view sequence includes the target object tracking sequence.

20. The electronic device according to claim 14, wherein, The processor is configured to determine keyframe data of the initial playback view sequence, the keyframe data including keyframe positions and attributes; and based on the keyframe data of the initial playback view sequence and a camera movement template recommendation model, determine a camera movement template that matches the initial playback view sequence.

21. The electronic device according to claim 20, wherein, The processor is configured to extract several panoramic video frames from the panoramic video; use the several panoramic video frames and the keyframe data of the initial playback viewpoint sequence as input to the camera movement template recommendation model, and obtain the output camera movement template.

22. The electronic device according to claim 14, wherein, The processor is configured to acquire a selected video time range or start time point for camera movement processing from the original panoramic video. Based on the selected video time range or starting time point, a panoramic video segment is determined for camera movement processing, and the panoramic video segment is used as the panoramic video to be processed.

23. The electronic device according to claim 14, wherein, The processor is configured to perform one of the following: Obtain the target object selected by the user in the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence; Identify the target object closest to the set center viewpoint in the preview interface of the panoramic video, and use the target object tracking sequence corresponding to the target object as the initial playback view sequence.

24. The electronic device according to claim 14, wherein, The processor is configured to display at least one camera movement editing point in the panoramic video, the camera movement editing point being used to allow a user to modify the camera movement angle of the corresponding panoramic video frame; in response to the user's modification of the camera movement angle at the camera movement editing point, a modified panoramic video movement is generated, the modified panoramic video movement including the modified camera movement angle.

25. The electronic device according to claim 15, wherein, The processor is configured to use the camera movement template selected by the user from a plurality of candidate camera movement templates as the matched camera movement template.

26. The electronic device according to claim 15, wherein, The camera movement template further includes a camera movement speed, and the processor is configured to obtain the camera movement speed between two adjacent keyframes in the camera movement template, and perform camera movement based on the camera movement speed and the camera movement trajectory.

Images