Multi-frame image synthesis method and apparatus, and electronic device and storage medium

Abstract

Provided in the embodiments of the present disclosure are a multi-frame image synthesis method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a first image and a second image, wherein the first image presents an object within a first photographing range in a photographing environment on the basis of a first photographing angle, the second image presents an object within a second photographing range in the photographing environment on the basis of a second photographing angle, and there is an overlapping range between the first photographing range and the second photographing range; reconstructing the second image on the basis of the first photographing angle, so as to generate a reconstructed image, wherein the reconstructed image presents an object within the second photographing range on the basis of the first photographing angle; and translating and / or rotating the reconstructed image to align an overlapping region of the first image with that of the reconstructed image, so as to generate a composite image, wherein the overlapping regions are used for presenting objects within the overlapping range, the composite image is used for presenting an object within a merged range on the basis of the first photographing angle, and the merged range is composed of the first photographing range and the second photographing range.

Description

Multi-frame image synthesis method, apparatus, electronic device and storage medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510322207.9, filed on March 18, 2025, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a method, apparatus, electronic device, and storage medium for synthesizing multi-frame images. Background Technology

[0004] Currently, in wide-angle image and video synthesis applications, multiple shots can be taken of adjacent areas in the shooting environment to generate multiple frames of images. These multiple frames are then synthesized to create a composite image with a wider field of view.

[0005] For the above application scenarios, in the process of generating wide-angle composite images, due to the difference in shooting angles of multiple image captures, the captured images will present objects in the shooting environment from different shooting angles.

[0006] However, when multiple frames of images are used for image synthesis, the resulting synthesized image exhibits image distortion, affecting the realism and visual appeal of the synthesized image. Summary of the Invention

[0007] This disclosure provides a method, apparatus, electronic device, and storage medium for synthesizing multi-frame images to overcome the problem of image distortion in synthesized images.

[0008] This disclosure provides a method for synthesizing multi-frame images, including:

[0009] A first image and a second image are acquired, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, with an overlapping area between the first and second shooting ranges; the second image is reconstructed based on the first shooting angle to generate a reconstructed image, the reconstructed image displaying objects within the second shooting range based on the first shooting angle; the reconstructed image is translated and / or rotated to align the overlapping areas of the first image and the reconstructed image to generate a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within a merged range based on the first shooting angle, the merged range being composed of the first and second shooting ranges.

[0010] This disclosure also provides a multi-frame image synthesis apparatus, including:

[0011] An acquisition module is used to acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range;

[0012] The reconstruction module is used to reconstruct the second image based on the first shooting angle to generate a reconstructed image, wherein the reconstructed image displays objects within the second shooting range based on the first shooting angle;

[0013] A compositing module is used to translate and / or rotate the reconstructed image to align the overlapping areas of the first image and the reconstructed image, generating a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle, the merged range being composed of the first shooting range and the second shooting range.

[0014] This disclosure also provides an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the multi-frame image synthesis method as described above.

[0017] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the multi-frame image synthesis method described above.

[0018] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-frame image synthesis method described above. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is an application scenario diagram of the multi-frame image synthesis method provided in the embodiments of this disclosure;

[0021] Figure 2 is a schematic flowchart of the multi-frame image synthesis method provided in this embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram of a first image and a second image provided in an embodiment of this disclosure;

[0023] Figure 4 is a flowchart of the specific implementation of step S102 in the embodiment shown in Figure 2;

[0024] Figure 5 is a schematic diagram of a process for generating a synthetic image according to an embodiment of this disclosure;

[0025] Figure 6 is a schematic flowchart of the multi-frame image synthesis method provided in this embodiment of the present disclosure.

[0026] Figure 7 is a flowchart of the specific implementation of step S205 in the embodiment shown in Figure 6;

[0027] Figure 8 is a flowchart of the specific implementation of step S206 in the embodiment shown in Figure 6;

[0028] Figure 9 is a flowchart of the specific implementation of step S2062 in the embodiment shown in Figure 8;

[0029] Figure 10 is a structural block diagram of the multi-frame image synthesis apparatus provided in an embodiment of this disclosure;

[0030] Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure; and

[0031] Figure 12 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0034] The application scenarios of the embodiments of this disclosure are explained below:

[0035] The multi-frame image synthesis method provided in this disclosure can be applied to applications (APPs) with image processing functions, such as video editing applications, image processing applications, and intelligent agent applications. More specifically, it can be applied to application scenarios such as synthesizing wide-angle images from multiple frames and synthesizing wide-angle videos from multiple video segments. The execution subject of this embodiment can be a terminal device running the aforementioned application with image processing functions, a server deploying the server corresponding to the aforementioned application, or other electronic devices that perform similar functions. When the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the aforementioned application; when the execution subject is a server, the server of the aforementioned application with data and business management functions can run partially or entirely on the server, and the method provided in this embodiment is executed on the server side, while the terminal device runs the client of the application. Communication between the server and the terminal device is based on server-client communication, thereby enabling the terminal device to obtain the execution result of the method provided in this embodiment and display it as needed.

[0036] In some embodiments, the terminal device or server can implement the multi-frame image synthesis method provided in this disclosure by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be program-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be local applications, i.e., programs that need to be installed in the operating system to run, or small programs embedded in any APP, i.e., programs that run in a browser environment. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin; the specific implementation can be configured as needed. Furthermore, in implementing the multi-frame image synthesis method provided in this disclosure, the terminal device can execute the method by running computer-executable instructions or computer programs set locally, or by calling computer-executable instructions or computer programs set in an external server. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminal devices.

[0037] Figure 1 illustrates an application scenario of the multi-frame image synthesis method provided in this embodiment. Referring to Figure 1, taking a terminal device as an example, the terminal device runs a target application with image processing capabilities. The user inputs at least two different images to be processed through the interactive interface of the target application, such as image P1 and image P2 shown in the figure. For example, the images P1 and P2 can be images taken from different angles in the same shooting environment using the same shooting device. Then, in response to the user's trigger operation (clicking the "Generate" control), the terminal device merges image P1 and image P2 using the multi-frame image synthesis method provided in this embodiment to generate a synthesized image P3. As shown in the figure, image P3 is equivalent to a combination of images P1 and P2, containing the image content of each of images P1 and P2. Therefore, it has a larger imaging field of view, forming an effect similar to wide-angle lens shooting.

[0038] In the scenario described above for generating contract image P3, although images P1 and P2 are both taken from the same shooting area, the different shooting angles result in differences in how the same object is presented in images P1 and P2. Some solutions directly detect and merge the overlapping areas or boundaries of images P1 and P2, leading to image distortion in the generated composite image P3, which affects the realism and visual appeal of the composite image.

[0039] This disclosure provides a method for synthesizing multi-frame images to solve the above-mentioned problems.

[0040] Referring to Figure 2, which is a flowchart illustrating a multi-frame image synthesis method provided in this embodiment, the method can be applied to a terminal device or a server. In one possible implementation, the terminal device can execute locally deployed program code to implement the method. In another possible implementation, the server can be used to deploy functional services based on the multi-frame image synthesis method provided in this embodiment, and the terminal device can access the server and call the corresponding functional services to implement the multi-frame image synthesis method provided in this embodiment. For example, the multi-frame image synthesis method provided in this embodiment includes:

[0041] Step S101: Acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range.

[0042] Step S102: Based on the first shooting angle, reconstruct the second image to generate a reconstructed image. The reconstructed image displays objects within the second shooting range based on the first shooting angle.

[0043] Referring to the application scenario diagram shown in Figure 1, this embodiment describes the provided multi-frame image synthesis method using a terminal device as the execution subject. For example, the terminal device runs a target application and provides multi-frame image synthesis functionality to the user through the target application's interface. The user loads a first image and a second image, thereby enabling the terminal device to obtain the first and second images and perform subsequent processing. The first image displays objects within a first shooting range in the shooting environment from a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment from a second shooting angle. There is an overlap between the first and second shooting ranges. Specifically, the first and second images can be images captured twice from different angles using the same device (e.g., a smartphone). The first and second shooting ranges are two adjacent shooting ranges in the same shooting environment, with a partial overlap. Since the images are taken from adjacent first and second shooting ranges within the same shooting environment, the first and second images contain the same subject (all or part of it). For example, if the same table is photographed from different angles, forming the first and second images, then the first and second images will present the table's appearance and outline from different angles. However, due to the different shooting angles, the table's appearance and outline will differ in the first and second images. Alternatively, in another possible implementation, the first and second images can be captured synchronously or asynchronously by two shooting devices or units. For example, a binocular camera on a virtual reality device can simultaneously capture the first and second images.

[0044] Figure 3 is a schematic diagram of a first image and a second image provided in an embodiment of this disclosure. The above steps will be further described below with reference to Figure 3. As shown in Figure 3, images P1 and P2 are, for example, images captured by terminal device D1 in the same shooting environment. The content captured by image P1 corresponds to the shooting range Z1 (first shooting range) in the shooting environment, while the content captured by image P2 corresponds to the shooting range Z2 (second shooting range) in the shooting environment. The two have an overlap range Z0. As can be seen from the figure, since images P1 and P2 have different shooting angles, that is, their corresponding display planes (focal planes) are not parallel, the same object (e.g., the house shown in the figure) within the overlap range Z0 of shooting ranges Z1 and Z2 is not presented in the same way in image P1 and image P2. Therefore, if images P1 and P2 are directly stitched together, the objects in the generated merged image P3 will be distorted, that is, the image will be deformed, affecting the image realism.

[0045] Furthermore, after obtaining the first and second images taken from different shooting angles, in order to avoid the problem of different image content (object outlines and shapes) due to different shooting angles, this embodiment reconstructs the second image based on the first shooting angle of the first image, generating a reconstructed image of the object within the second shooting range based on the first shooting angle. This process is equivalent to rotating the second image with the vertical or horizontal direction in the shooting environment as the axis, so that the rotating second image (i.e., the reconstructed image) is consistent with the shooting angle of the first image, so that the two are located on the same display plane, reducing or eliminating the problem of inconsistent presentation effect of the same object in the second image and the first image due to different shooting angles.

[0046] Furthermore, in one possible implementation, as shown in Figure 4, step S102 is specifically implemented as follows:

[0047] Step S1021: Based on the first image, obtain the first direction vector representing the first shooting angle.

[0048] Step S1022: Obtain the first pixel matrix corresponding to the second image, and reconstruct the first pixel matrix based on the first direction vector to generate the second pixel matrix.

[0049] Step S1023: Obtain the reconstructed image based on the second pixel matrix.

[0050] For example, firstly, a camera space is constructed by detecting the first image, and the camera shooting point and the corresponding shooting angle vector, i.e., the first direction vector, are inferred based on the camera space. The specific implementation method is not elaborated here. Next, the second image is parsed to generate a corresponding first pixel matrix. This first pixel matrix includes multiple pixels, each with a corresponding pixel value. The pixel value is, for example, an array based on RGB channels, and each pixel is obtained based on the original pixels constituting the first image. In one possible implementation, the first pixel matrix is ​​composed of the original pixels of the second image. In another possible implementation, the first pixel matrix can be obtained by processing the original pixels of the second image, such as performing downsampling, oversampling, or sharpness enhancement. Next, the first pixel matrix is ​​reconstructed based on the first direction vector. For example, firstly, the second direction vector corresponding to the second image is obtained. The method for obtaining the second direction vector is similar to the method for obtaining the first direction vector based on the first image, and will not be repeated here. Then, the transformation mapping matrix / vector between the first and second direction vectors is calculated. Next, based on the first pixel matrix, the first pixel matrix and the transformation mapping vector / matrix are calculated to obtain the second pixel matrix. Finally, the second pixel matrix is ​​scaled to generate an image that matches the image size of the first image, i.e., the reconstructed image.

[0051] In this embodiment, the second image is reconstructed based on the first direction vector corresponding to the first image to generate a reconstructed image for displaying objects within the second shooting range based on the first shooting angle. This allows the second image to be reconstructed onto the same display plane as the first image for display, avoiding inconsistencies in image content within overlapping areas due to different shooting angles. Ultimately, this provides the image realism and image quality of the generated synthetic image.

[0052] Step S103: Translate and / or rotate the reconstructed image to align the overlapping areas of the first image and the reconstructed image to generate a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle, and the merged range is composed of the first shooting range and the second shooting range.

[0053] For example, after generating the reconstructed image, it is equivalent to mapping the second image onto the same display plane as the first image. However, there are still differences in displacement and / or rotation angles between the reconstructed second image (reconstructed image) and the first image. Therefore, the reconstructed image is further processed, i.e., translated by a certain distance and / or rotated by a certain angle, to align the overlapping areas of the first image and the reconstructed image. Then, the first image and the reconstructed image are stitched together to generate a composite image. Although the second image is reconstructed based on the first shooting angle, the semantic content of the image in the reconstructed image does not change relative to the second image; it still displays content within the second shooting range. Therefore, the composite image after stitching together the first image and the reconstructed image contains the content of the first image and the reconstructed image (second image) displayed based on the first shooting angle, i.e., objects within the merged range formed by the first and second shooting ranges, thereby achieving the goal of generating a wide-field composite image based on multiple frames. In this embodiment, the reconstructed image may be translated or rotated, or both may be translated and rotated simultaneously. The specific settings can be made according to the needs, and no specific restrictions are imposed here.

[0054] Figure 5 is a schematic diagram of a process for generating a composite image according to an embodiment of this disclosure. The process will be described in more detail below with reference to Figure 5. As shown in Figure 5, firstly, images P1 and P2, taken from different shooting angles, are obtained. Image P1 includes image region Z11 and image region Z12, and image P2 includes image region Z21 and image region Z22. Image region Z12 and image region Z21 (i.e., the overlapping area) correspond to the overlap between the first shooting range and the second shooting range. Next, image P2 is reconstructed using the first shooting angle corresponding to image P1 to generate image P3 (the reconstructed image). Image P3 includes image region Z31 and image region Z32. Then, images P1 and P3 are translated and rotated to align image region Z12 of image P1 and image region Z32 of image P3. Furthermore, the pixels within image region Z12 and image region Z32 are fused, for example, by calculating the average pixel value of every two corresponding pixels to obtain the fused image region Z0. Finally, image regions Z11, Z0, and Z31 are stitched together to generate the final composite image P4. The shooting range corresponding to the composite image P4 is the merging range under the shooting environment.

[0055] In this embodiment, a first image and a second image are acquired. The first image displays objects within a first shooting range in the shooting environment from a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment from a second shooting angle. There is an overlap between the first and second shooting ranges. Based on the first shooting angle, the second image is reconstructed to generate a reconstructed image, which displays objects within the second shooting range from the first shooting angle. The reconstructed image is then translated and / or rotated to align the overlapping areas of the first and reconstructed images, generating a composite image. The overlapping area displays objects within the overlapping range, and the composite image displays objects within the merged range from the first shooting angle. The merged range is composed of the first and second shooting ranges. By acquiring first and second images captured from different shooting angles, reconstructing the second image to generate a reconstructed image at the same shooting angle as the first image, and then aligning the first and reconstructed images to generate a composite image, the problem of different outlines for objects within the same shooting range in the first and second images due to different shooting angles is avoided. This prevents image distortion in the composite image and improves the realism and visual appeal of the composite image.

[0056] Referring to Figure 6, which is a schematic flowchart of the multi-frame image synthesis method provided in this embodiment, this embodiment further refines step S103 based on the embodiment shown in Figure 2. The multi-frame image synthesis method includes:

[0057] Step S201: Acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range.

[0058] Step S202: Based on the first shooting angle, reconstruct the second image to generate a reconstructed image. The reconstructed image displays objects within the second shooting range based on the first shooting angle.

[0059] Step S203: Detect the first overlapping region in the first image and the second overlapping region in the reconstructed image.

[0060] Step S204: Based on the distance deviation between the first overlapping region and the second overlapping region, obtain the corresponding translation distance, and / or, based on the angle deviation between the first overlapping region and the second overlapping region, obtain the corresponding rotation angle.

[0061] Step S205: Align the overlapping areas of the first image and the reconstructed image according to the translation distance and / or rotation angle.

[0062] Step S206: Generate a composite image based on the first image after the overlapping regions are aligned and the reconstructed image.

[0063] For example, after reconstructing the second image to generate a reconstructed image, a display space plane is created based on the image content in the first image and the reconstructed image. The display space plane can be a plane used to describe the planar coordinates of objects in the shooting environment. For example, this display space plane is, for instance, the plane containing the first image and the reconstructed image, or the first image and the reconstructed image are mapped to this display space plane respectively. Then, within this display space plane, a first overlapping region in the first image and a second overlapping region in the reconstructed image are detected; that is, regions in the first image and the reconstructed image that have the same content. This can be achieved by calculating region similarity, which will not be elaborated here. Next, based on the contours of the first and second overlapping regions in the display space plane, the distance deviation and angular deviation between the first and second overlapping regions are calculated. Based on the distance deviation and / or angular deviation between the first and second overlapping regions, the translation distance and / or rotation angle that needs to be adjusted for the second overlapping region are obtained. Next, based on the translation distance and / or rotation angle, the first and second overlapping regions are aligned, that is, the overlapping regions of the first image and the reconstructed image are aligned. Finally, based on the first image and the reconstructed image after the overlapping regions are aligned, the images are stitched and fused to generate a composite image.

[0064] In one possible implementation, after determining the first and second overlapping regions, the angular deviation between the contours of the first and second overlapping regions is first measured. Then, the image corresponding to the second overlapping region is rotated to correct the angular deviation between the first and second overlapping regions. Next, the distance deviation between the contours of the first and second overlapping regions is measured, and a translation distance is determined based on this distance deviation. Then, the reconstructed image corresponding to the second overlapping region is translated based on the translation distance, so that the contours of the first and second overlapping regions completely overlap, achieving the effect of aligning the overlapping regions of the first and reconstructed images. Finally, a composite image is generated based on the first and reconstructed images after the alignment of the first and second overlapping regions.

[0065] Furthermore, in one possible implementation, as shown in Figure 7, step S205 is specifically implemented as follows:

[0066] Step S2051: Based on the image content of the first image and the reconstructed image, obtain the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image.

[0067] Step S2052: Determine the first rotation angle of the first image and the second rotation angle of the reconstructed image based on the horizontal line of the first image and the horizontal line of the reconstructed image, respectively.

[0068] Step S2053: Based on the first rotation angle and the second rotation angle, rotate the first image and the reconstructed image respectively to obtain the first rotated image corresponding to the first image and the second rotated image corresponding to the reconstructed image.

[0069] Step S2054: Based on the translation distance, translate the second rotated image to align the overlapping area of ​​the first rotated image and the second rotated image.

[0070] For example, firstly, based on the image content of the first image and the reconstructed image, such as objects with horizontal outlines like tables, floors, and buildings, the horizontal line corresponding to the first image and the horizontal line of the reconstructed image are determined. For example, the line segment where the edge of the table surface in the image is located is determined as the horizontal line of that image. Then, based on the horizontal lines in the first image and the reconstructed image, a first rotation angle of the first image and a second rotation angle of the reconstructed image are determined respectively. Specifically, within the display space plane, the angle required to rotate the horizontal line to horizontal (0 degrees) is calculated. That is, the rotation angle when rotating the horizontal line corresponding to the first image to horizontal is the first rotation angle; the rotation angle when rotating the horizontal line corresponding to the second image to horizontal is the second rotation angle. Next, based on the first and second rotation angles, the first image and the reconstructed image are rotated respectively, correcting the angular deviations between the first image and the reconstructed image, resulting in a first rotated image corresponding to the first image and a second rotated image corresponding to the reconstructed image. Finally, the translation distance between the corresponding contours of the first and second rotated images in the display space plane is measured, and the second rotated image is translated based on the translation distance to align the overlapping area of ​​the first and second rotated images, that is, the first overlapping area and the second overlapping area.

[0071] Accordingly, step S206 is implemented as follows: image synthesis is performed based on the first rotated image and the translated second rotated image to generate a synthesized image.

[0072] Furthermore, regarding step S206, in one possible implementation, as shown in Figure 8, the specific implementation of step S206 includes:

[0073] Step S2061: After aligning the overlapping regions, obtain the first pixel in the overlapping region of the first image and the second pixel in the reconstructed image corresponding to the first pixel. The relative positions of the first pixel and the corresponding second pixel in the corresponding overlapping region are the same.

[0074] Step S2062: Obtain the first weight corresponding to the first pixel and the second weight corresponding to the second pixel respectively, and calculate the weighted sum of the pixel value corresponding to the first pixel and the pixel value corresponding to the second pixel based on the first weight and the second weight to obtain the pixel value of the superimposed pixel corresponding to the overlapping area.

[0075] Step S2063: Obtain the synthesized image based on the pixel values ​​of the superimposed pixels corresponding to the overlapping regions.

[0076] For example, the overlapping areas in the first and second images correspond to the overlapping range in the shooting environment. After aligning the overlapping areas of the first and reconstructed images (i.e., after generating the first and second rotated images in the above embodiments), for the overlapping areas, the first pixel in the overlapping area of ​​the first image and the second pixel in the reconstructed image corresponding to the first pixel are obtained respectively. In one possible implementation, each pixel in the overlapping area (first overlapping area) of the first image (first rotated image) can be traversed, and each traversed pixel is used as the first pixel to perform the above processing steps. Accordingly, when processing each first pixel, the corresponding pixel in the overlapping area (second overlapping area) of the reconstructed image (second rotated image), i.e., the second pixel, is obtained respectively. The relative positions of the first pixel and the corresponding second pixel in the corresponding overlapping area are the same. For example, if the first pixel is an extreme point on the edge of the overlapping area, then the second pixel is also an extreme point at the corresponding position on the edge of the overlapping area. The mapping relationship between the first pixel and the corresponding second pixel is determined during the detection of the overlapping area, which will not be elaborated here. Next, for each first pixel, the pixel value and weight value corresponding to that first pixel are obtained, i.e., the first weight; and the pixel value and weight value corresponding to the second pixel corresponding to that first pixel are obtained, i.e., the second weight. Then, based on the pixel value and weight value corresponding to the first pixel, and the pixel value and weight value corresponding to the second pixel, a weighted sum is calculated to obtain a weighted pixel value. This weighted pixel value is the pixel value of the superimposed pixel at the corresponding position in the overlapping region. Repeating the above steps yields the pixel value corresponding to each superimposed pixel within the overlapping region in the synthesized image. Then, the pixels in the non-overlapping regions of the first image (first rotated image) and the non-overlapping regions of the second image (second rotated image) are obtained respectively. The non-overlapping regions in the first image, the overlapping region of the synthesized image, and the non-overlapping regions in the second image are then stitched together to generate the synthesized image.

[0077] For example, in one possible implementation, the first weight and the second weight can be fixed values, such as both being 0.5. That is, for the overlapping region, the pixel value of each pixel in the overlapping region is determined by calculating the average value of the first image and the second image. The above calculation method is simple, but it may cause artifacts in the overlapping region of the generated synthetic image, affecting the viewpoint effect of the image.

[0078] In this embodiment, another method for calculating the first weight and the second weight is also provided. Further, as shown in Figure 9, the specific implementation of step S2062 includes:

[0079] Step S2062-1: Obtain the first distance and the second distance of the first pixel within the overlapping region, wherein the first distance is the distance of the first pixel from the boundary of the overlapping region closer to the first image; and the second distance is the distance of the second pixel corresponding to the first pixel from the boundary of the overlapping region closer to the reconstructed image.

[0080] Step S2062-2: Obtain the first weight and the second weight based on the first distance and the second distance.

[0081] For example, during the traversal of the first pixel, the first distance and the second distance of the first pixel within the overlapping region are first obtained. The first distance is calculated, for example, by calculating the distance of the first pixel from the boundary of the overlapping region closer to the first image. The second distance is calculated, for example, by calculating the distance of the second pixel corresponding to the first pixel from the boundary of the overlapping region closer to the reconstructed image. The artifacts appearing within the overlapping region are due to the inconsistent representation of the same pixel between the first image (first rotated image) and the reconstructed image (second rotated image). More specifically, in the overlapping region of the first image, the RGB value of the first pixel p1 is (10, 10, 10). However, in the overlapping region of the reconstructed image, due to inevitable errors and distortions during image generation, it is difficult to completely restore and display the object from the first shooting angle; therefore, the RGB value of the second pixel p2 corresponding to the first pixel p1 is (80, 80, 80), which differs from the corresponding pixel in the first image. In this case, generating the final pixel value through simple average weighting can lead to abrupt changes with other adjacent pixels, resulting in artifacts.

[0082] In this embodiment, the first distance and the second distance corresponding to the first pixel point within the overlapping area are calculated respectively, and the corresponding weight value is determined based on the first distance and the second distance. Specifically, if the distance between the first pixel point and the boundary of the overlapping area closer to the first image is smaller, the target pixel value is closer to the pixel value of the first pixel point (i.e., the first weight is larger); if the distance between the second pixel point and the boundary of the overlapping area closer to the second image is smaller, the target pixel value is closer to the pixel value of the second pixel point (i.e., the second weight is larger).

[0083] Furthermore, regarding step S2062-1, one possible implementation includes: determining the first overlapping region boundary point closest to the first pixel and the second overlapping region boundary point closest to the corresponding second pixel; determining a first distance based on the coordinate difference between the first pixel's coordinates and the second overlapping region boundary point's coordinates; and determining a second distance based on the coordinate difference between the third pixel's coordinates and the fourth overlapping region boundary point's coordinates. For example, if the first overlapping region boundary point closest to the first pixel p1 is p01, and the second overlapping region boundary point closest to the corresponding second pixel p2 is p02, then the distance L1 between the first pixel p1 and the first overlapping region boundary point p01 is calculated. This distance can be a horizontal coordinate distance, a vertical coordinate distance, or the vector length between the two pixels. Similarly, the distance L2 between the second pixel p2 and the second overlapping region boundary point p02 is calculated, and the corresponding weights are determined based on the length relationship between L1 and L2, such as a proportional relationship or a length difference relationship. Among them, distance and weight are inversely proportional.

[0084] Further, regarding step S2062-2, one possible implementation includes: obtaining the region distance corresponding to the overlapping region, where the region distance is the sum of a first distance and a second distance; calculating the ratios of the first distance and the second distance to the region distance, respectively, to obtain the first weight corresponding to the first distance and the second weight corresponding to the second distance. For example, if the first distance is L1 and the second distance is L2, then the region distance is (L1+L2). Accordingly, the weight corresponding to the first pixel is L1 / (L1+L2); the weight corresponding to the second pixel is L2 / (L1+L2).

[0085] In this embodiment, the implementation of steps S201 and S202 is the same as that of steps S101 and S102 in the embodiment shown in FIG2 of this disclosure, and will not be described in detail here.

[0086] Corresponding to the multi-frame image synthesis method in the above embodiments, Figure 10 is a structural block diagram of the multi-frame image synthesis apparatus provided in this disclosure embodiment. The method described in the above embodiments can be executed by this multi-frame image synthesis apparatus, which can be implemented by software and / or hardware, and can be integrated into an electronic device with certain data processing capabilities. The electronic device may include, but is not limited to, mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0087] For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. Referring to FIG10, the multi-frame image synthesis method apparatus 3 includes:

[0088] The acquisition module 31 is used to acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlapping range between the first shooting range and the second shooting range.

[0089] The reconstruction module 32 is used to reconstruct the second image based on the first shooting angle, generate a reconstructed image, and display the objects within the second shooting range based on the first shooting angle.

[0090] The compositing module 33 is used to translate and / or rotate the reconstructed image to align the overlapping areas of the first image and the reconstructed image to generate a composite image. The overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle. The merged range is composed of the first shooting range and the second shooting range.

[0091] According to one or more embodiments of this disclosure, the synthesis module 33 is specifically configured to: detect a first overlapping region in the first image and a second overlapping region in the reconstructed image; obtain a corresponding translation distance based on the distance deviation between the first overlapping region and the second overlapping region, and / or obtain a corresponding rotation angle based on the angular deviation between the first overlapping region and the second overlapping region; align the overlapping regions of the first image and the reconstructed image according to the translation distance and / or the rotation angle, and generate a synthesized image based on the first image and the reconstructed image after the overlapping regions are aligned.

[0092] According to one or more embodiments of this disclosure, when the compositing module 33 aligns the overlapping areas of the first image and the reconstructed image based on the translation distance and / or rotation angle, and generates a composite image based on the first image and the reconstructed image after the overlap area is aligned, it is specifically configured to: obtain the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image based on the image content of the first image and the reconstructed image; determine the first rotation angle of the first image and the second rotation angle of the reconstructed image based on the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image, respectively; rotate the first image and the reconstructed image based on the first rotation angle and the second rotation angle, respectively, to obtain the first rotated image corresponding to the first image and the second rotated image corresponding to the reconstructed image; translate the second rotated image based on the translation distance to align the overlapping areas of the first rotated image and the second rotated image; and perform image compositing based on the first rotated image and the translated second rotated image to generate a composite image.

[0093] According to one or more embodiments of this disclosure, when the synthesis module 33 generates a synthesized image based on the first image after overlapping region alignment and the reconstructed image, it is specifically configured to: after overlapping region alignment, obtain a first pixel point in the overlapping region of the first image and a second pixel point in the reconstructed image corresponding to the first pixel point, wherein the relative positions of the first pixel point and the corresponding second pixel point are the same in the corresponding overlapping region; obtain a first weight corresponding to the first pixel point and a second weight corresponding to the second pixel point respectively, and calculate a weighted sum of the pixel values ​​corresponding to the first pixel point and the second pixel point based on the first weight and the second weight to obtain the pixel value of the superimposed pixel point corresponding to the overlapping region; and obtain the synthesized image based on the pixel value of the superimposed pixel point corresponding to the overlapping region.

[0094] According to one or more embodiments of this disclosure, when the synthesis module 33 obtains the first weight corresponding to the first pixel and the second weight corresponding to the second pixel, it is specifically used to: obtain the first distance and the second distance of the first pixel in the overlapping area, wherein the first distance is the distance of the first pixel from the boundary of the overlapping area closer to the first image; the second distance is the distance of the second pixel corresponding to the first pixel from the boundary of the overlapping area closer to the reconstructed image; and obtain the first weight and the second weight based on the first distance and the second distance.

[0095] According to one or more embodiments of this disclosure, when the synthesis module 33 obtains the first distance and the second distance of the first pixel point within the overlapping region, it is specifically configured to: determine the first overlapping region boundary point closest to the first pixel point and the second overlapping region boundary point closest to the second pixel point corresponding to the first pixel point; determine the first distance based on the coordinate difference between the first pixel point coordinates of the first pixel point and the second pixel point coordinates of the first overlapping region boundary point; and determine the second distance based on the coordinate difference between the third pixel point coordinates of the second pixel point and the fourth pixel point coordinates of the second overlapping region boundary point.

[0096] According to one or more embodiments of this disclosure, when the synthesis module 33 obtains the first weight and the second weight based on the first distance and the second distance, it is specifically used to: obtain the region distance corresponding to the overlapping region, wherein the region distance is the sum of the first distance and the second distance; calculate the ratio of the first distance and the second distance to the region distance respectively, and obtain the first weight corresponding to the first distance and the second weight corresponding to the second distance.

[0097] According to one or more embodiments of this disclosure, the reconstruction module 32 is specifically configured to: obtain a first direction vector representing a first shooting angle based on a first image; obtain a first pixel matrix corresponding to a second image, and reconstruct the first pixel matrix based on the first direction vector to generate a second pixel matrix; and obtain a reconstructed image based on the second pixel matrix.

[0098] The acquisition module 31, reconstruction module 32, and synthesis module 33 are connected sequentially. The multi-frame image synthesis device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0099] Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. As shown in Figure 11, the electronic device 4 includes:

[0100] Processor 41, and memory 42 communicatively connected to processor 41;

[0101] Memory 42 stores instructions executed by the computer;

[0102] The processor 41 executes computer execution instructions stored in the memory 42 to implement the multi-frame image synthesis method in the embodiments shown in Figures 2-9.

[0103] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0104] The relevant explanations can be understood by referring to the descriptions and effects of the steps in the embodiments corresponding to Figures 2-9, which will not be elaborated on here.

[0105] This disclosure provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement the multi-frame image synthesis method provided in any of the embodiments corresponding to Figures 2-9 of this disclosure.

[0106] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the multi-frame image synthesis method provided in any of the embodiments corresponding to Figures 2-9 of this disclosure.

[0107] To implement the above embodiments, this disclosure also provides an electronic device.

[0108] Referring to Figure 12, a schematic diagram of the structure of an electronic device 900 suitable for implementing embodiments of the present disclosure is shown. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 12 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0109] As shown in Figure 12, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0110] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 shows electronic device 900 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0111] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0112] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0115] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the specific unit itself.

[0118] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] According to one or more embodiments of this disclosure, a multi-frame image synthesis method is provided, comprising:

[0121] A first image and a second image are acquired, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, with an overlapping area between the first and second shooting ranges; the second image is reconstructed based on the first shooting angle to generate a reconstructed image, the reconstructed image displaying objects within the second shooting range based on the first shooting angle; the reconstructed image is translated and / or rotated to align the overlapping areas of the first image and the reconstructed image to generate a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within a merged range based on the first shooting angle, the merged range being composed of the first and second shooting ranges.

[0122] According to one or more embodiments of this disclosure, translating and / or rotating the reconstructed image to align the overlapping regions of the first image and the reconstructed image to generate a composite image includes: detecting a first overlapping region in the first image and a second overlapping region in the reconstructed image; obtaining a corresponding translation distance based on the distance deviation between the first overlapping region and the second overlapping region, and / or obtaining a corresponding rotation angle based on the angular deviation between the first overlapping region and the second overlapping region; aligning the overlapping regions of the first image and the reconstructed image according to the translation distance and / or the rotation angle, and generating a composite image based on the first image and the reconstructed image after the overlapping regions are aligned.

[0123] According to one or more embodiments of this disclosure, aligning the overlapping areas of the first image and the reconstructed image based on the translation distance and / or the rotation angle, and generating a composite image based on the first image and the reconstructed image after the overlapping areas are aligned, includes: obtaining an image horizontal line corresponding to the first image and an image horizontal line of the reconstructed image based on the image content of the first image and the reconstructed image; determining a first rotation angle of the first image and a second rotation angle of the reconstructed image based on the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image; rotating the first image and the reconstructed image based on the first rotation angle and the second rotation angle to obtain a first rotated image corresponding to the first image and a second rotated image corresponding to the reconstructed image; translating the second rotated image based on the translation distance to align the overlapping areas of the first rotated image and the second rotated image; and performing image compositing based on the first rotated image and the translated second rotated image to generate a composite image.

[0124] According to one or more embodiments of this disclosure, generating a composite image based on a first image aligned with the overlapping regions and a reconstructed image includes: after aligning the overlapping regions, obtaining a first pixel point within the overlapping region of the first image and a second pixel point corresponding to the first pixel point in the reconstructed image, wherein the relative positions of the first pixel point and the corresponding second pixel point are the same within the corresponding overlapping region; obtaining a first weight corresponding to the first pixel point and a second weight corresponding to the second pixel point, respectively, and calculating a weighted sum of the pixel values ​​corresponding to the first pixel point and the second pixel point based on the first weight and the second weight to obtain the pixel value of the superimposed pixel point corresponding to the overlapping region; and obtaining the composite image based on the pixel value of the superimposed pixel point corresponding to the overlapping region.

[0125] According to one or more embodiments of this disclosure, obtaining the first weight corresponding to the first pixel and the second weight corresponding to the second pixel includes: obtaining a first distance and a second distance of the first pixel within the overlapping region, wherein the first distance is the distance of the first pixel from the region boundary of the overlapping region closer to the first image; the second distance is the distance of the second pixel corresponding to the first pixel from the region boundary of the overlapping region closer to the reconstructed image; and obtaining the first weight and the second weight based on the first distance and the second distance.

[0126] According to one or more embodiments of this disclosure, obtaining the first distance and the second distance of the first pixel point within the overlapping region includes: determining a first overlapping region boundary point closest to the first pixel point and a second overlapping region boundary point closest to the second pixel point corresponding to the first pixel point; determining the first distance based on the coordinate difference between the first pixel point coordinates and the second pixel point coordinates of the first overlapping region boundary point; and determining the second distance based on the coordinate difference between the third pixel point coordinates of the second pixel point and the fourth pixel point coordinates of the second overlapping region boundary point.

[0127] According to one or more embodiments of this disclosure, obtaining the first weight and the second weight based on the first distance and the second distance includes: obtaining the region distance corresponding to the overlapping region, wherein the region distance is the sum of the first distance and the second distance; calculating the ratios of the first distance and the second distance to the region distance, respectively, to obtain the first weight corresponding to the first distance and the second weight corresponding to the second distance.

[0128] According to one or more embodiments of this disclosure, the step of reconstructing the second image based on the first shooting angle to generate a reconstructed image includes: obtaining a first direction vector representing the first shooting angle based on the first image; obtaining a first pixel matrix corresponding to the second image, and reconstructing the first pixel matrix based on the first direction vector to generate a second pixel matrix; and obtaining the reconstructed image based on the second pixel matrix.

[0129] According to one or more embodiments of this disclosure, a multi-frame image synthesis apparatus is also provided, comprising:

[0130] An acquisition module is used to acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range;

[0131] The reconstruction module is used to reconstruct the second image based on the first shooting angle to generate a reconstructed image, wherein the reconstructed image displays objects within the second shooting range based on the first shooting angle;

[0132] A compositing module is used to translate and / or rotate the reconstructed image to align the overlapping areas of the first image and the reconstructed image, generating a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle, the merged range being composed of the first shooting range and the second shooting range.

[0133] According to one or more embodiments of this disclosure, the synthesis module is specifically configured to: detect a first overlapping region in the first image and a second overlapping region in the reconstructed image; obtain a corresponding translation distance based on the distance deviation between the first overlapping region and the second overlapping region, and / or obtain a corresponding rotation angle based on the angular deviation between the first overlapping region and the second overlapping region; align the overlapping regions of the first image and the reconstructed image according to the translation distance and / or the rotation angle, and generate a synthesized image based on the first image and the reconstructed image after the overlapping regions are aligned.

[0134] According to one or more embodiments of this disclosure, when the compositing module aligns the overlapping areas of the first image and the reconstructed image based on the translation distance and / or the rotation angle, and generates a composite image based on the first image and the reconstructed image after the overlapping areas are aligned, it is specifically configured to: obtain the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image based on the image content of the first image and the reconstructed image; determine the first rotation angle of the first image and the second rotation angle of the reconstructed image based on the image horizontal line corresponding to the first image and the image horizontal line of the reconstructed image; rotate the first image and the reconstructed image based on the first rotation angle and the second rotation angle to obtain the first rotated image corresponding to the first image and the second rotated image corresponding to the reconstructed image; translate the second rotated image based on the translation distance to align the overlapping areas of the first rotated image and the second rotated image; and perform image compositing based on the first rotated image and the translated second rotated image to generate a composite image.

[0135] According to one or more embodiments of this disclosure, when the compositing module generates a composite image based on the first image aligned with the overlapping region and the reconstructed image, it is specifically configured to: after the overlapping region is aligned, obtain a first pixel in the overlapping region of the first image and a second pixel in the reconstructed image corresponding to the first pixel, wherein the relative positions of the first pixel and the corresponding second pixel are the same in the corresponding overlapping region; obtain a first weight corresponding to the first pixel and a second weight corresponding to the second pixel, respectively, and calculate a weighted sum of the pixel values ​​corresponding to the first pixel and the second pixel based on the first weight and the second weight to obtain the pixel value of the superimposed pixel corresponding to the overlapping region; and obtain the composite image based on the pixel value of the superimposed pixel corresponding to the overlapping region.

[0136] According to one or more embodiments of this disclosure, when the synthesis module obtains the first weight corresponding to the first pixel and the second weight corresponding to the second pixel, it is specifically configured to: obtain a first distance and a second distance of the first pixel in the overlapping region, wherein the first distance is the distance of the first pixel from the region boundary of the overlapping region closer to the first image; the second distance is the distance of the second pixel corresponding to the first pixel from the region boundary of the overlapping region closer to the reconstructed image; and obtain the first weight and the second weight based on the first distance and the second distance.

[0137] According to one or more embodiments of this disclosure, when the synthesis module obtains the first distance and the second distance of the first pixel point within the overlapping region, it is specifically configured to: determine the first overlapping region boundary point closest to the first pixel point and the second overlapping region boundary point closest to the second pixel point corresponding to the first pixel point; determine the first distance based on the coordinate difference between the first pixel point coordinates and the second pixel point coordinates of the first overlapping region boundary point; and determine the second distance based on the coordinate difference between the third pixel point coordinates of the second pixel point and the fourth pixel point coordinates of the second overlapping region boundary point.

[0138] According to one or more embodiments of this disclosure, when the synthesis module obtains the first weight and the second weight based on the first distance and the second distance, it is specifically configured to: obtain the region distance corresponding to the overlapping region, wherein the region distance is the sum of the first distance and the second distance; calculate the ratios of the first distance and the second distance to the region distance, respectively, to obtain the first weight corresponding to the first distance and the second weight corresponding to the second distance.

[0139] According to one or more embodiments of this disclosure, the reconstruction module is specifically configured to: obtain a first direction vector representing the first shooting angle based on the first image; obtain a first pixel matrix corresponding to the second image, and reconstruct the first pixel matrix based on the first direction vector to generate a second pixel matrix; and obtain the reconstructed image based on the second pixel matrix.

[0140] According to one or more embodiments of the present disclosure, an electronic device is also provided, comprising: at least one processor and a memory;

[0141] The memory stores computer-executed instructions;

[0142] The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the multi-frame image synthesis method as described above.

[0143] According to one or more embodiments of the present disclosure, a computer-readable storage medium is also provided, wherein computer-executable instructions are stored in the computer-executable storage medium, and when a processor executes the computer-executable instructions, the multi-frame image synthesis method described above is implemented.

[0144] According to one or more embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the multi-frame image synthesis method as described above.

[0145] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0146] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0147] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for synthesizing multi-frame images, comprising: Acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range; Based on the first shooting angle, the second image is reconstructed to generate a reconstructed image, which displays objects within the second shooting range based on the first shooting angle; The reconstructed image is translated and / or rotated to align the overlapping areas of the first image and the reconstructed image to generate a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle, the merged range being composed of the first shooting range and the second shooting range.

2. The method according to claim 1, wherein, The step of translating and / or rotating the reconstructed image to align the overlapping areas of the first image and the reconstructed image to generate a composite image includes: Detect the first overlapping region in the first image and the second overlapping region in the reconstructed image; Based on the distance deviation between the first overlapping region and the second overlapping region, the corresponding translation distance is obtained, and / or, based on the angle deviation between the first overlapping region and the second overlapping region, the corresponding rotation angle is obtained; The overlapping areas of the first image and the reconstructed image are aligned according to the translation distance and / or the rotation angle, and the composite image is generated based on the first image and the reconstructed image after the overlapping areas are aligned.

3. The method according to claim 2, wherein, The step of aligning the overlapping regions of the first image and the reconstructed image according to the translation distance and / or the rotation angle, and generating the composite image based on the first image and the reconstructed image after aligning the overlapping regions, includes: Based on the image content of the first image and the reconstructed image, the horizontal line corresponding to the first image and the horizontal line of the reconstructed image are obtained; Based on the horizontal line of the first image and the horizontal line of the reconstructed image, the first rotation angle of the first image and the second rotation angle of the reconstructed image are determined respectively. Based on the first rotation angle and the second rotation angle, the first image and the reconstructed image are rotated respectively to obtain a first rotated image corresponding to the first image and a second rotated image corresponding to the reconstructed image; Based on the translation distance, the second rotated image is translated to align the overlapping area of ​​the first rotated image and the second rotated image; The composite image is generated by combining the first rotated image with the translated second rotated image.

4. The method according to claim 2, wherein, A composite image is generated based on the first image aligned with the overlapping region and the reconstructed image, including: After the overlapping region is aligned, a first pixel in the overlapping region of the first image and a second pixel in the reconstructed image corresponding to the first pixel are obtained, wherein the relative positions of the first pixel and the corresponding second pixel in the corresponding overlapping region are the same. The first weight corresponding to the first pixel and the second weight corresponding to the second pixel are obtained respectively. Based on the first weight and the second weight, the weighted sum of the pixel value corresponding to the first pixel and the pixel value corresponding to the second pixel is calculated to obtain the pixel value of the superimposed pixel corresponding to the overlapping region. The synthesized image is obtained based on the pixel values ​​of the superimposed pixels corresponding to the overlapping regions.

5. The method according to claim 4, wherein, The step of obtaining the first weight corresponding to the first pixel and the second weight corresponding to the second pixel includes: Obtain a first distance and a second distance of the first pixel within the overlapping region, wherein the first distance is the distance of the first pixel from the boundary of the overlapping region on the side closer to the first image; and the second distance is the distance of the second pixel corresponding to the first pixel from the boundary of the overlapping region on the side closer to the reconstructed image. The first weight and the second weight are obtained based on the first distance and the second distance.

6. The method according to claim 5, wherein, The step of obtaining the first distance and the second distance of the first pixel within the overlapping area includes: Determine the first overlapping region boundary point closest to the first pixel and the second overlapping region boundary point closest to the second pixel corresponding to the first pixel; The first distance is determined based on the coordinate difference between the first pixel coordinates of the first pixel and the second pixel coordinates of the boundary point of the first overlapping region. The second distance is determined based on the coordinate difference between the third pixel coordinate of the second pixel and the fourth pixel coordinate of the boundary point of the second overlapping region.

7. The method according to claim 5 or 6, wherein, The step of obtaining the first weight and the second weight based on the first distance and the second distance includes: Obtain the region distance corresponding to the overlapping region, wherein the region distance is the sum of the first distance and the second distance; Calculate the ratios of the first distance and the second distance to the region distance, respectively, to obtain the first weight corresponding to the first distance and the second weight corresponding to the second distance.

8. The method according to any one of claims 1-7, wherein, The step of reconstructing the second image based on the first shooting angle to generate a reconstructed image includes: Based on the first image, a first direction vector representing the first shooting angle is obtained; Obtain the first pixel matrix corresponding to the second image, and reconstruct the first pixel matrix based on the first direction vector to generate the second pixel matrix; The reconstructed image is obtained based on the second pixel matrix.

9. A multi-frame image synthesis apparatus, comprising: The acquisition module is configured to acquire a first image and a second image, wherein the first image displays objects within a first shooting range in the shooting environment based on a first shooting angle, and the second image displays objects within a second shooting range in the shooting environment based on a second shooting angle, and there is an overlap between the first shooting range and the second shooting range; The reconstruction module is configured to reconstruct the second image based on the first shooting angle to generate a reconstructed image, wherein the reconstructed image displays objects within the second shooting range based on the first shooting angle; The compositing module is configured to translate and / or rotate the reconstructed image to align the overlapping areas of the first image and the reconstructed image, generating a composite image, wherein the overlapping area is used to display objects within the overlapping range, and the composite image is used to display objects within the merged range based on the first shooting angle, the merged range being composed of the first shooting range and the second shooting range.

10. An electronic device comprising a processor and a memory, wherein, The memory stores computer-executed instructions; When the processor executes the computer-executable instructions stored in the memory, it implements the multi-frame image synthesis method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, wherein, When the processor executes the computer execution instructions, it implements the multi-frame image synthesis method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program, wherein, When the computer program is executed by the processor, it implements the multi-frame image synthesis method as described in any one of claims 1 to 8.

Images