Image stitching method and apparatus, and device, storage medium and program product

Abstract

The present application belongs to the technical field of image processing. Disclosed are an image stitching method and apparatus, and a device, a storage medium and a program product. The method comprises: determining stitching reference information corresponding to a plurality of images, wherein the stitching reference information comprises stitching complexity and / or a network state; on the basis of the stitching reference information, sending an image stitching request to a cloud platform, wherein the image stitching request is used for instructing the cloud platform to assist a terminal device in stitching the plurality of images; and receiving stitching assistance data sent by the cloud platform, and on the basis of the stitching assistance data, determining a stitched image corresponding to the plurality of images. Thus, it can be seen that the terminal device evaluates the stitching capability itself and the performance of a communication network, so as to intelligently decide whether to request the cloud platform to assist in stitching the plurality of images, and uses the stitching reference information to accurately guide the cloud platform to provide necessary processing support. Therefore, the powerful computing capability of the cloud platform is effectively utilized, the resource burden on the terminal device is reduced, and the efficiency and accuracy of image stitching are ensured.

Description

Image stitching methods, apparatus, equipment, storage media and program products

[0001] This application claims priority to Chinese Patent Application No. 202510469208.6, filed on April 14, 2025, entitled “Image Stitching Method, Apparatus, Device, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image stitching method, apparatus, device, storage medium, and program product. Background Technology

[0003] Panoramic stitching technology can stitch together multiple images with partially overlapping areas to generate a stitched image with an ultra-wide field of view. This not only greatly expands the field of view but also ensures the complete preservation of details in the original image.

[0004] In related technologies, users capture multiple images using terminal devices (such as mobile phones or cameras) and then send these images to a cloud platform via a network. Upon receiving the images, the cloud platform uses a high-precision image stitching algorithm to stitch them together, resulting in a stitched image. This stitched image is then sent to the terminal device via the network for preview and viewing.

[0005] Therefore, how to generate stitched images through end-to-cloud collaboration while ensuring image stitching quality and reducing communication latency has become an urgent technical problem to be solved. Summary of the Invention

[0006] This application provides an image stitching method, apparatus, device, storage medium, and program product, which can generate stitched images through edge-cloud collaboration, ensuring image stitching quality while reducing edge-cloud communication latency. The technical solution is as follows:

[0007] In a first aspect, an image stitching method is provided, applied to a terminal device. The method includes: obtaining stitching reference information corresponding to multiple images, the stitching reference information including stitching complexity and / or network status, the stitching complexity indicating the difficulty of stitching the multiple images by the terminal device, and the network status indicating the performance of the communication network between the terminal device and a cloud platform; sending an image stitching request to the cloud platform based on the stitching reference information, the image stitching request being used to instruct the cloud platform to assist the terminal device in stitching the multiple images; receiving stitching assistance data sent by the cloud platform, and determining the stitched image corresponding to the multiple images based on the stitching assistance data.

[0008] Among them, multiple images can be raw images acquired by the terminal device and not processed, or images that have been preprocessed by the terminal device. Preprocessing includes noise reduction, color adjustment, and restoration, etc., and this application does not limit them.

[0009] Before sending multiple images to be stitched to the cloud platform, the terminal device assesses its own ability to stitch multiple images and intelligently decides whether to request the cloud platform's assistance in stitching the images. Simultaneously, it assesses the performance of the communication network between the terminal device and the platform, intelligently deciding on the uplink and downlink data transmission during the collaborative generation of the stitched image. Thus, by having the cloud platform assist in stitching multiple images and providing stitching assistance data to the terminal device, the powerful computing capabilities of the cloud platform can be effectively utilized, reducing the resource burden on the terminal device while ensuring the quality of the stitched images. Furthermore, the technical solution of this application, where the cloud platform assists in stitching multiple images and provides stitching assistance data, reduces the communication latency between the terminal device and the cloud platform compared to a solution where the cloud platform independently stitches and generates the image and then feeds it back to the terminal device. Moreover, with the cloud platform providing stitching assistance data, the speed at which the terminal device generates the stitched image is also increased, significantly improving the user experience. The aforementioned edge-cloud collaborative image stitching solution can adapt to different scenarios and changing needs of terminal devices, possessing a certain degree of flexibility and scalability.

[0010] Since the stitching reference information includes stitching complexity and / or network status, the content of the image stitching request sent by the terminal device to the cloud platform will be different depending on the stitching reference information, and the content of the stitching auxiliary data provided by the cloud platform will also be different.

[0011] Next, we will introduce three scenarios in which terminal devices send image stitching requests to the cloud platform.

[0012] In the first scenario, the stitching reference information is the stitching complexity. Sending an image stitching request to the cloud platform based on the stitching reference information includes: determining the level corresponding to the stitching complexity from multiple complexity levels, wherein the multiple complexity levels include a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level; and sending an image stitching request to the cloud platform based on the level corresponding to the stitching complexity.

[0013] In one possible implementation, the level corresponding to the stitching complexity is the first complexity level, and the image stitching request carries the multiple images and first coordination instruction information, which is used to instruct the cloud platform to stitch the multiple images.

[0014] In one possible implementation, the level corresponding to the stitching complexity is the second complexity level, and the image stitching request carries the multiple images and second coordination indication information, the second coordination indication information being used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0015] In one possible implementation, the level corresponding to the stitching complexity is the third complexity level, and the image stitching request carries the downsampled image corresponding to each of the multiple images and the second coordination indication information, which is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0016] Therefore, when the stitching complexity level is first, the terminal device instructs the cloud platform to stitch multiple images. In this case, the cloud platform provides a complete stitched image, thus maximizing the quality of the stitched image. When the stitching complexity level is second or third, an edge-cloud collaborative approach is used to stitch multiple images. In this case, the terminal device instructs the cloud platform to distort the multiple images. The cloud platform provides stitching auxiliary data as the target distorted mesh, rather than the stitched image itself. This significantly reduces the amount of downlink data transmitted between the terminal device and the cloud platform, lowering downlink communication latency and network bandwidth consumption. In the case where the stitching complexity level is the third level, the terminal device independently performs most of the calculations in the image stitching process. However, to ensure the image stitching quality, the terminal device can also interact with the cloud platform to transmit a small amount of data and generate a higher quality stitched image based on the stitching auxiliary data (i.e., the target twisted mesh) provided by the cloud platform. At this time, the amount of uplink and downlink data transmitted between the terminal device and the cloud platform is minimized, which can further reduce the communication latency between the terminal device and the cloud platform, as well as the network traffic consumption when transmitting uplink and downlink data.

[0017] In the second scenario, the stitching reference information includes the stitching complexity and the network status. Sending the image stitching request to the cloud platform based on the stitching reference information includes: determining the level corresponding to the stitching complexity from multiple complexity levels, wherein the multiple complexity levels include a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level; and sending the image stitching request to the cloud platform based on the level corresponding to the stitching complexity and the network status.

[0018] In one possible implementation, the level corresponding to the stitching complexity is the first complexity level, and the state evaluation value of the network state is greater than the weak network threshold. The image stitching request carries the multiple images and the first coordination indication information, which is used to instruct the cloud platform to stitch the multiple images.

[0019] In one possible implementation, the level corresponding to the stitching complexity is the second complexity level, and the state evaluation value of the network state is greater than the weak network threshold. The image stitching request carries the multiple images and second coordination indication information, which is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0020] In one possible implementation, the stitching complexity level is the third complexity level, and the network state evaluation value is less than or equal to the weak network threshold. The image stitching request carries the downsampled images and second coordination indication information corresponding to the multiple images respectively. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0021] Therefore, this application can also combine the two factors of stitching complexity and network status to intelligently decide whether to request the cloud platform to assist in stitching multiple images, as well as the uplink and downlink data transmitted between the terminal device and the cloud platform when the edge and cloud collaboratively generate stitched images.

[0022] In the third scenario, the stitching reference information is the network state. If the network state's state evaluation value is greater than the weak network threshold, the image stitching request carries the multiple images and first coordination indication information. The first coordination indication information instructs the cloud platform to stitch the multiple images. If the network state's state evaluation value is less than or equal to the weak network threshold, the image stitching request carries the downsampled images corresponding to the multiple images and second coordination indication information. The second coordination indication information instructs the cloud platform to perform distortion transformations on the multiple images.

[0023] The higher the network state assessment value, the better the performance of the communication network; conversely, the lower the network state assessment value, the worse the performance of the communication network.

[0024] As an example, the network status assessment value can be determined based on at least one performance parameter of the communication network between the terminal device and the cloud platform. This at least one performance parameter includes, but is not limited to, network bandwidth, latency, packet loss rate, jitter, and connection stability. For example, the higher the network bandwidth, the higher the communication network status assessment value; the lower the latency, the higher the communication network status assessment value. Similarly, the weak network threshold can also be determined based on the critical values ​​of one or more of the above performance parameters.

[0025] Based on the first or second collaborative instruction information carried by the terminal device in the image stitching request, the following section introduces two scenarios in which the cloud platform feeds back stitching auxiliary data to the terminal device based on the above two types of collaborative instruction information.

[0026] In the first scenario, when the image stitching request carries first collaborative instruction information, receiving the stitching auxiliary data sent by the cloud platform includes: receiving the stitched image obtained by the cloud platform after stitching the multiple images, and using the stitched image as the stitching auxiliary data.

[0027] In other words, the stitching assistance data sent by the cloud platform is the stitched image corresponding to multiple images. The terminal device only needs to receive and download the complete stitched image, without having to perform the image stitching operation itself.

[0028] In the second scenario, when the image stitching request carries the second collaborative indication information, receiving the stitching auxiliary data sent by the cloud platform includes: receiving the target distorted mesh obtained by the cloud platform after performing distortion transformation on the multiple images, using the target distorted mesh as the stitching auxiliary data, and the target distorted mesh indicating the geometric transformation operation for stitching the multiple images.

[0029] In other words, the stitching auxiliary data sent by the cloud platform is the target twisted mesh. The terminal device needs to perform geometric transformation operations on multiple images based on the target twisted mesh in order to generate the corresponding stitched image.

[0030] Parallax and the motion of objects in the scene are two important factors affecting the difficulty of stitching. Therefore, this application can evaluate the difficulty (i.e., stitching complexity) of stitching multiple images based on the parallax parameters of the multiple images or based on the image offset of the multiple images. The following sections will introduce these two methods for determining stitching complexity.

[0031] In the first approach, the stitching reference information is the stitching complexity, and the multiple images are continuously acquired in the same scene. Obtaining the stitching reference information corresponding to the multiple images includes: acquiring the disparity parameters corresponding to the multiple images, the disparity parameters including scene depth and / or the device displacement of the terminal device during the acquisition of the multiple images; and obtaining the stitching complexity based on the disparity parameters.

[0032] When the disparity parameter is the scene depth or the device displacement, the disparity parameter can be directly determined as the stitching complexity. That is, the value of the stitching complexity is positively correlated with the value of the disparity parameter. The larger the value of the disparity parameter, the larger the corresponding stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images. Conversely, the smaller the value of the disparity parameter, the smaller the corresponding stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0033] In one possible implementation, the disparity parameter is the scene depth or the device displacement, and determining the level corresponding to the stitching complexity from multiple complexity levels includes: if the disparity parameter is greater than or equal to a first disparity threshold, then the level corresponding to the stitching complexity is the first complexity level; if the disparity parameter is less than the first disparity threshold but greater than a second disparity threshold, then the level corresponding to the stitching complexity is the second complexity level, where the first disparity threshold is greater than the second disparity threshold; if the disparity parameter is less than or equal to the second disparity threshold, then the level corresponding to the stitching complexity is the third complexity level.

[0034] Wherein, when the disparity parameter is the scene depth, the first disparity threshold is a first depth threshold and the second disparity threshold is a second depth threshold; when the disparity parameter is the device displacement, the first disparity threshold is a first displacement threshold and the second disparity threshold is a second displacement threshold.

[0035] When the parallax parameters include the scene depth and the device displacement, the stitching complexity can be determined based on the scene depth and the device displacement. For example, the stitching complexity can be the sum of the values ​​of the scene depth and the device displacement, or it can be a weighted average of the values ​​of the scene depth and the device displacement; this application does not impose any restrictions on this.

[0036] Similarly, when the parallax parameters include scene depth and device displacement, the value of the stitching complexity is positively correlated with the value of these two parallax parameters. The larger the value of the parallax parameter, the larger the corresponding stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images; conversely, the smaller the value of the parallax parameter, the smaller the corresponding stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0037] In one possible implementation, the disparity parameter includes the scene depth and the device displacement. Determining the level corresponding to the stitching complexity from multiple complexity levels includes: if the scene depth is greater than or equal to a first depth threshold and the device displacement is greater than or equal to a first displacement threshold, then the level corresponding to the stitching complexity is the first complexity level; if the scene depth is less than the first depth threshold and greater than a second depth threshold, or the device displacement is less than the first displacement threshold and greater than the second displacement threshold, then the level corresponding to the stitching complexity is the second complexity level, the first disparity threshold is greater than the second disparity threshold, and the first displacement threshold is greater than the second displacement threshold; if the scene depth is less than or equal to the second depth threshold, or the device displacement is less than or equal to the second displacement threshold, then the level corresponding to the stitching complexity is the third complexity level.

[0038] In the second approach, the stitching reference information is the stitching complexity, and the multiple images are continuously acquired from the same scene. Obtaining the stitching reference information corresponding to the multiple images includes: acquiring the image offset corresponding to the multiple images, whereby the image offset is used to describe the motion of the same object in the multiple images; and obtaining the stitching complexity based on the image offset.

[0039] As an example, the image offset can be directly determined as the stitching complexity. That is, the value of the stitching complexity is positively correlated with the value of the image offset. The larger the value of the image offset, the larger the corresponding value of the stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images; conversely, the smaller the value of the image offset, the smaller the corresponding value of the stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0040] In one possible implementation, determining the level corresponding to the stitching complexity from multiple complexity levels includes: if the image offset is greater than or equal to a first offset threshold, then the level corresponding to the stitching complexity is the first complexity level; if the image offset is less than the first offset threshold and greater than a second offset threshold, then the level corresponding to the stitching complexity is the second complexity level, where the first offset threshold is greater than the second offset threshold; if the image offset is less than or equal to the second offset threshold, then the level corresponding to the stitching complexity is the third complexity level.

[0041] In one possible implementation, the image stitching request also carries stitching parameters, which include the image depth corresponding to each of the multiple images, the movement trajectory of the terminal device during the acquisition of the multiple images, and the optical flow information corresponding to each of the multiple images. The optical flow information is used to indicate the movement of pixels in two adjacent images in the acquisition sequence.

[0042] In one possible implementation, the stitching parameters may further include one or more of the following: image alignment information, acquisition order, and acquisition position of the multiple images. The image alignment information is used to describe the alignment of two images acquired in adjacent order.

[0043] In one possible implementation, before sending the image stitching request to the cloud platform based on the stitching reference information, the method further includes:

[0044] The multiple images are initially stitched together to obtain an intermediate stitching result; in this case, the stitching parameters also include the intermediate stitching result.

[0045] The aforementioned auxiliary information helps the cloud platform to stitch or distort multiple images. Therefore, when the network status assessment value between the terminal device and the cloud platform is greater than the weak network threshold, one or more of the above auxiliary information can be sent to the cloud platform to accelerate the efficiency of the cloud platform in generating stitching auxiliary data and ensure the quality of the stitched image to the greatest extent.

[0046] In one possible implementation, the stitching auxiliary data is a target warped mesh, which indicates the geometric transformation operation for stitching the multiple images; determining the stitched image corresponding to the multiple images based on the stitching auxiliary data includes: performing transformation processing on the multiple images respectively using an image warping function based on the target warped mesh; and stitching the transformed multiple images according to the acquisition order to obtain the stitched image.

[0047] The image warping function is a function that performs geometric transformations on an image based on a target warped mesh. It takes the image and the target warped mesh as input and outputs the geometrically transformed image. In the process of stitching multiple images together, the image warping function is used to map each image to its correct position in the stitched image, based on the geometric transformation operation indicated by the calculated target warped mesh.

[0048] In this way, by using the cloud platform to assist in stitching multiple images and providing stitching assistance data to the terminal device, the powerful computing capabilities of the cloud platform can be effectively utilized, reducing the resource burden on the terminal device and ensuring the stitching quality of the images.

[0049] Secondly, an image stitching apparatus is provided, which has the function of implementing the image stitching method described in the first aspect. The image stitching apparatus includes at least one module for implementing the image stitching method provided in the first aspect.

[0050] Thirdly, a terminal device is provided, comprising a processor and a memory, the memory being used to store a computer program for executing the image stitching method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the image stitching method described in the first aspect.

[0051] In one possible implementation, the terminal device may further include a communication bus for establishing a connection between the processor and the memory.

[0052] Fourthly, a computer-readable storage medium is provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the image stitching method described in the first aspect.

[0053] Fifthly, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the image stitching method described in the first aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the image stitching method described in the first aspect.

[0054] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0055] Figure 1 is a schematic diagram of a panoramic shooting scene provided in an embodiment of this application;

[0056] Figure 2 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0057] Figure 3 is a flowchart illustrating an image stitching method provided in an embodiment of this application;

[0058] Figure 4 is a flowchart illustrating another image stitching method provided in an embodiment of this application;

[0059] Figure 5 is a flowchart illustrating another image stitching method provided in an embodiment of this application;

[0060] Figure 6 is a structural schematic diagram of an image stitching device provided in an embodiment of this application;

[0061] Figure 7 is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0063] To facilitate understanding, before explaining the image stitching method provided in this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0064] First, the terminology used in the embodiments of this application will be introduced.

[0065] 1. Inertial Measurement Unit (IMU)

[0066] An IMU is a device that measures and reports the acceleration and angular velocity of an object, and typically consists of the following sensors:

[0067] (1) Accelerometer: Used to detect the acceleration signal of an object along the three independent axes of the carrier coordinate system. An accelerometer can measure the acceleration of a carrier relative to inertial space, and the velocity and position information of the carrier can be obtained through integration. Common accelerometers include piezoresistive, capacitive, and microelectromechanical systems (MEMS) accelerometers.

[0068] (2) Gyroscope: Used to detect the angular velocity signal of the carrier relative to the navigation coordinate system. The gyroscope can measure the angular velocity of the carrier relative to inertial space, and the attitude information of the carrier can be obtained through integration. Common gyroscopes include mechanical gyroscopes, fiber optic gyroscopes, laser gyroscopes, and MEMS gyroscopes.

[0069] (3) Magnetometer (optional): Used to measure the direction of an object relative to the Earth's magnetic field. It is often used to correct the attitude information of the IMU and improve the accuracy of attitude measurement. The magnetometer is based on the Hall effect or magnetoresistive effect and calculates the direction of the magnetic field by measuring the change in current or voltage caused by the magnetic field.

[0070] In addition, the IMU may also include auxiliary components such as temperature sensors, calibration circuits, and data processing units to improve measurement accuracy and stability.

[0071] In practical applications, IMUs operate based on Newton's law of inertia, determining an object's motion state by measuring its linear acceleration and angular velocity in inertial space. Accelerometers and gyroscopes measure the object's linear acceleration and angular velocity, respectively. These data are processed and fused by a data processing unit to ultimately obtain the object's attitude, velocity, and position information. To improve measurement accuracy, IMUs typically employ high-precision sensors and complex algorithms to reduce errors and noise.

[0072] 2. Optical flow information

[0073] Optical flow information is a two-dimensional vector field describing the trajectory and velocity of pixels over time in a series of image frames. It is used to capture the motion of objects in an image and the displacement information between adjacent frames. In other words, optical flow information is determined by analyzing the motion of pixels in an image sequence, reflecting the speed and direction of pixel movement in the image.

[0074] In the specific implementation process, the methods for determining optical flow information include the following two:

[0075] The first method is based on image sequence analysis. By utilizing the changes in pixels over time and the correlation between adjacent frames in the image sequence, the correspondence between the previous frame and the current frame is found, and then the motion information of objects between adjacent frames is calculated through algorithms.

[0076] The second method involves the application of optical flow algorithms. Optical flow algorithms, such as sparse optical flow (Lucas-Kanade algorithm, Horn-Schunck algorithm, etc.) or dense optical flow algorithms, are used to calculate the displacement between two image frames. Sparse optical flow algorithms focus on key pixels in the image, while dense optical flow algorithms calculate the motion of all pixels in the image.

[0077] Secondly, the application scenarios of the embodiments of this application will be introduced.

[0078] Panoramic stitching technology is an image processing technique whose core objective is to seamlessly stitch together multiple images with partially overlapping areas to form a wide-viewing-angle or 360-degree panoramic image. This technology is widely used in photography, virtual reality, geographic information systems (GIS), surveillance video analysis, and other fields, providing users with a broader and more immersive visual experience.

[0079] When capturing images using portable devices (such as mobile phones and cameras), due to the instability of handheld devices or the movement of the photographer, it is difficult to ensure that the camera's center (i.e., optical center) is in the same position every time a shot is taken. This change in the shooting center results in slight differences in the perspective of each image, creating parallax. Parallax is a common problem in panoramic stitching, which can cause misalignment or distortion in overlapping areas of the stitched image.

[0080] Furthermore, in dynamic scenes, such as streets and event venues, there are moving objects (such as pedestrians and vehicles). These moving objects will appear in different positions or shapes in different images. Since traditional stitching algorithms assume that all objects are stationary during the stitching process, moving objects in dynamic scenes will disrupt the continuity of the stitching, leading to errors in the stitching result.

[0081] To cope with complex scenarios such as large parallax and moving objects, image stitching algorithms require more sensory input and greater computing power to support the stability of their effects. Given the limited computing power of terminal devices, running image stitching algorithms in the cloud (i.e., cloud platform) becomes a reasonable choice.

[0082] When a high-precision image stitching algorithm is running on a cloud platform, the user captures multiple images using a terminal device and sends them to the cloud platform via the network. Upon receiving the images, the cloud platform uses the high-precision image stitching algorithm to stitch them together, obtaining a stitched image, which is then sent to the terminal device via the network for preview and viewing.

[0083] However, with the continuous development of terminal devices, taking mobile phones as an example, the image resolution that mobile phones can capture is getting larger and larger, and the file size of the input and output images of image stitching algorithms is also getting larger and larger, with the output stitched images reaching hundreds of millions of pixels. Although running image stitching algorithms on cloud platforms can save computing power on the edge, the excessively large file size of input and output images increases communication pressure, consumes more bandwidth, and greatly reduces the availability and real-time performance of image stitching algorithms under weak network conditions.

[0084] Based on this, this application provides an image stitching method. Before sending multiple images to be stitched to the cloud platform, the terminal device assesses its own ability to stitch multiple images and intelligently decides whether to request the cloud platform's assistance in stitching the multiple images. Simultaneously, it assesses the performance of the communication network between the terminal device and the cloud platform and intelligently decides the uplink and downlink data transmission during end-cloud collaborative image generation. This effectively utilizes the powerful computing capabilities of the cloud platform, reducing the resource burden on the terminal device while ensuring image stitching quality. Furthermore, the technical solution of this application involves the cloud platform assisting in stitching multiple images and providing stitching assistance data. Compared to the solution where the cloud platform independently stitches and generates the stitched image and then feeds it back to the terminal device, this reduces the communication latency between the terminal device and the cloud platform. Moreover, with the cloud platform providing stitching assistance data, it also increases the speed at which the terminal device generates the stitched image, thereby significantly improving the user experience. The above-mentioned end-cloud collaborative image stitching scheme can adapt to different scenarios and changing terminal device requirements, possessing a certain degree of flexibility and scalability.

[0085] Finally, the implementation environment of the embodiments of this application will be described.

[0086] Referring to Figure 1, Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of this application. This implementation environment includes a terminal device and a cloud platform, with communication between the terminal device and the cloud platform via a network. When executing the image stitching method provided by this embodiment, the terminal device determines stitching reference information corresponding to multiple images and sends an image stitching request to the cloud platform based on the stitching reference information. This image stitching request instructs the cloud platform to assist the terminal device in stitching multiple images. In response to the image stitching request, the cloud platform determines stitching auxiliary data and sends the stitching auxiliary data to the terminal device. Finally, the terminal device determines the stitched image corresponding to the multiple images based on the stitching auxiliary data sent by the cloud platform.

[0087] In some embodiments, the terminal device includes an image acquisition module, a first image stitching module, and an image viewing module. The image acquisition module is used to acquire multiple images, the first image stitching module is used to stitch the multiple images together, and the image viewing module is used to view the stitched image corresponding to the multiple images.

[0088] In some embodiments, the cloud platform includes a second image stitching module, which is used to stitch multiple images together to obtain an intermediate stitching result or a final stitched image.

[0089] It should be understood that although both the first image stitching module and the second image stitching module can stitch multiple images together, the image stitching capability of the first image stitching module is usually less than that of the second image stitching module due to limitations in the resources and computing power of the end-side device.

[0090] As an example, due to limitations in edge computing power, the first image stitching module can only achieve image stitching using some image stitching algorithms with lower computational requirements, such as linear registration. However, the cloud platform has powerful computing capabilities; therefore, the second image stitching module can achieve image stitching using some high-precision image stitching algorithms. Examples include feature point matching algorithms, algorithms combining affine and perspective transformations, and multi-scale fusion algorithms.

[0091] In some embodiments, multiple images can be images captured consecutively in the same scene. It should be noted that consecutive capture here can be temporally continuous, meaning the images are captured consecutively in time, with no other images between adjacent images. Of course, consecutive capture can also be spatially continuous. For example, multiple images can be determined based on a sequence of images captured consecutively in the same scene. These images may not be consecutive within the image sequence, but their content has a certain coherence, allowing them to be stitched together.

[0092] Among them, multiple images can be collected by the terminal device itself, or they can be collected by other devices and sent to the terminal device. This application embodiment does not limit this.

[0093] For example, if the terminal device is a smartphone with a camera function, multiple images can be obtained through the smartphone's panoramic shooting function. This panoramic shooting function captures multiple images (also called photos) consecutively and stitches them together to form a stitched image (also called a panoramic photo) with a larger field of view and image size. The process of obtaining the stitched image through the smartphone's panoramic shooting function may include the following steps S1-S7:

[0094] S1: Open the camera app on your smartphone.

[0095] S2: Select panorama mode from the shooting options on the camera interface and click to enter panorama mode.

[0096] It should be noted that for smartphones of different models and manufacturers, the way to enter the panorama mode and the name of the option to achieve the panorama shooting function may be different, and are not limited to the above-mentioned "panorama mode". Other methods or names are also possible, and this application embodiment does not limit this.

[0097] S3: Set shooting parameters. Based on actual shooting needs, select the corresponding focal length and mode from the optional shooting focal lengths (wide-angle, standard (1x), telephoto, etc.) and optional shooting modes (such as panorama, ultra-wide-angle, long image, 1x3, etc.) provided by the camera application, and select the shooting position of the first image according to the interface prompts of panorama mode.

[0098] S4: Tap the shutter button in the camera app to start taking multiple images in panorama mode.

[0099] Referring to Figure 2, after taking an image, the panorama mode interface will prompt the next shooting direction. Users can move their smartphones to the shooting direction indicated on the interface, and the camera application's shooting system will automatically take a picture or record.

[0100] In some cases, the shooting system may or may not restrict the direction and form of the shot. When restrictions are in place, the system will only trigger automatic shooting or recording when certain conditions are met.

[0101] In some embodiments, during the shooting process, the smartphone can transmit data traffic to the cloud platform. The amount of data transmitted is related to the user's shooting content, shooting method, network environment, and other conditions.

[0102] S5: The user can manually click the button to stop shooting, or the shooting system can automatically stop shooting when it detects that the user has reached the preset shooting terminal.

[0103] S6: Generate a stitched image. After shooting, the smartphone communicates with the cloud platform to stitch multiple images together according to the technical solution provided in the embodiments of this application, generating a stitched image.

[0104] S7: Preview and save stitched images. After generating the stitched image, users can preview it in the camera app, transfer it to other image editing modules, or choose to re-stitch it. Users can also choose to save the stitched image to their smartphone's storage.

[0105] It should be noted that the terminal device can begin data transmission with the cloud platform after acquiring multiple images, or it can send the necessary data for image stitching to the cloud platform beforehand, during the acquisition of multiple images. This application does not impose any limitations on this.

[0106] It should be understood that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of technologies and equipment, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0107] The image stitching method provided in the embodiments of this application will be explained in detail below.

[0108] Figure 3 is a flowchart of an image stitching method provided in an embodiment of this application. This method can be applied to the terminal device shown in Figure 2. Referring to Figure 3, the method includes the following steps.

[0109] Step 301: Obtain stitching reference information corresponding to multiple images. The stitching reference information includes stitching complexity and / or network status. Stitching complexity indicates the difficulty of stitching multiple images by the terminal device, and network status indicates the performance of the communication network between the terminal device and the cloud platform.

[0110] The multiple images are continuously captured by the terminal device in the same scene. These multiple images can be the original, unprocessed images captured by the terminal device, or images that have been preprocessed by the terminal device. Preprocessing includes noise reduction, color adjustment, and restoration, etc., and this application does not impose any restrictions on this.

[0111] It should be noted that higher stitching complexity means greater difficulty for the terminal device to stitch multiple images; conversely, lower complexity means less difficulty for the terminal device to stitch multiple images. In other words, stitching complexity is positively correlated with the difficulty of stitching multiple images for the terminal device.

[0112] In some embodiments, the network state corresponds to a state evaluation value. The magnitude of the state evaluation value is positively correlated with the performance of the communication network. A higher state evaluation value indicates better performance of the communication network between the terminal device and the cloud platform; conversely, a lower state evaluation value indicates worse performance of the communication network between the terminal device and the cloud platform. It should be understood that with good communication network performance, the terminal device and the cloud platform can transmit large amounts of data with low latency and high transmission speed.

[0113] The network status assessment value can be determined based on one or more performance parameters between the terminal device and the cloud platform, such as network bandwidth, packet loss rate, latency, jitter, and connection stability. For example, the higher the network bandwidth, the higher the communication network status assessment value; the lower the latency, the higher the communication network status assessment value.

[0114] As an example, the network state can be the average of the scores of the above-mentioned multiple performance parameters, or it can be the weighted average of the scores of the above-mentioned multiple performance parameters. This application embodiment does not limit this.

[0115] In some embodiments, parallax and the motion of objects in the scene are two important factors affecting the difficulty of stitching. Therefore, in determining the difficulty (i.e., stitching complexity) of stitching multiple images, this application embodiment can evaluate it based on the parallax parameters of the multiple images or based on the image offset of the multiple images. The implementation schemes for determining stitching complexity will be described below.

[0116] The first approach, the implementation process of obtaining the concatenation complexity in step 301 above, may include the following steps (11)-(12):

[0117] (11) Obtain the disparity parameters corresponding to multiple images, including the scene depth of the scene and / or the device displacement of the terminal device during the acquisition of multiple images.

[0118] Scene depth reflects the distance between objects in the scene and the terminal device (or the center of the camera). When the scene depth varies significantly across multiple images captured by the terminal device, the overlapping portions between adjacent images may not align, resulting in seams or misalignments during image stitching.

[0119] As an example, scene depth can be calculated based on the main camera and wide-angle heterogeneous binocular lens on the terminal device, using the principle of binocular stereo vision. Binocular stereo vision is a technology that mimics human binocular vision, using two cameras to capture the same scene from different angles. By matching corresponding points in the two images, disparity is calculated, and then depth information is obtained based on the principle of triangle similarity.

[0120] As another example, scene depth can be determined based on the depth range of multiple images, which can be the difference between the maximum and minimum values ​​within that depth range. For instance, if multiple images include image 1, image 2, image 3, and image 4, and image 2 has the furthest center of view from the terminal device, resulting in the largest image depth (denoted as D2), while image 4 has the closest center of view to the terminal device, resulting in the smallest image depth (denoted as D4), then the scene depth can be the difference between D2 and D4.

[0121] Device displacement reflects the movement of the terminal device when acquiring multiple images. When the device displacement between two adjacent images is too large, it will cause significant changes in the objects contained in these two images (or offsets), which will result in the loss of detailed information or blurring when the images are stitched together.

[0122] As an example, device displacement can be determined using IMU information. Taking a smartphone as an example, the gyroscope IMU can measure the smartphone's attitude changes (such as rotation, tilt, etc.) and acceleration information in real time, and then determine the amount of displacement of the terminal device during the acquisition of multiple images based on this information, thus obtaining the device displacement.

[0123] In this embodiment, scene depth can be used to analyze the changes in the depth of objects distributed in multiple images, and device displacement can be used to analyze the changes in the horizontal distance of objects distributed in multiple images. The greater the scene depth and / or device displacement, the more difficult the image stitching becomes; conversely, the smaller the scene depth and device displacement, the less difficult the image stitching becomes.

[0124] It should be noted that the embodiments of this application are only illustrative of the above acquisition method. In practical applications, the scene depth and / or device displacement of a scene in which multiple images are acquired can also be obtained by other methods. The embodiments of this application do not limit the acquisition method.

[0125] (12) Based on the disparity parameter, the splicing complexity is obtained.

[0126] When the disparity parameter is the scene depth or the device displacement, the disparity parameter can be directly determined as the stitching complexity. That is, the value of the stitching complexity is positively correlated with the value of the disparity parameter. The larger the value of the disparity parameter, the larger the corresponding stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images. Conversely, the smaller the value of the disparity parameter, the smaller the corresponding stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0127] When the parallax parameters include the scene depth and the device displacement, the stitching complexity can be determined based on the scene depth and the device displacement. For example, the stitching complexity can be the sum of the values ​​of the scene depth and the device displacement, or it can be a weighted average of the values ​​of the scene depth and the device displacement; this application does not impose any restrictions on this.

[0128] Similarly, when the parallax parameters include scene depth and device displacement, the value of the stitching complexity is positively correlated with the value of these two parallax parameters. The larger the value of the parallax parameter, the larger the corresponding stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images; conversely, the smaller the value of the parallax parameter, the smaller the corresponding stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0129] The second approach, the implementation process of obtaining the splicing complexity in step 301 above, may include the following steps (21)-(22):

[0130] (21) Obtain the image offsets corresponding to multiple images. The image offsets are used to describe the motion of the same object in multiple images.

[0131] In one possible implementation, the image offset can be determined based on the optical flow information of multiple images. Of course, considering the accuracy of the results, IMU information can also be used to compensate and correct the aforementioned optical flow information.

[0132] As an example, the image offset can be the average or maximum magnitude of the optical flow map vectors of multiple images.

[0133] (22) Based on the image offset, the stitching complexity is obtained.

[0134] As an example, the image offset can be directly determined as the stitching complexity. That is, the value of the stitching complexity is positively correlated with the value of the image offset. The larger the value of the image offset, the larger the corresponding value of the stitching complexity, which means that it is more difficult for the terminal device to stitch multiple images; conversely, the smaller the value of the image offset, the smaller the corresponding value of the stitching complexity, which means that it is easier for the terminal device to stitch multiple images.

[0135] In some embodiments, when the stitching reference information is stitching complexity, before performing step 302 below, it is necessary to determine the level corresponding to the stitching complexity from multiple complexity levels, and then perform the operation of sending an image stitching request to the cloud platform in step 302 below based on the level corresponding to the stitching complexity.

[0136] The complexity levels include a first complexity level, a second complexity level, and a third complexity level. The first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level.

[0137] Regarding step (21) above, when the disparity parameter is scene depth or device displacement, the stitching complexity level is determined from multiple complexity levels, including: if the disparity parameter is greater than or equal to a first disparity threshold, the stitching complexity level is the first complexity level; if the disparity parameter is less than the first disparity threshold but greater than a second disparity threshold, the stitching complexity level is the second complexity level; if the disparity parameter is less than or equal to the second disparity threshold, the stitching complexity level is the third complexity level. Wherein, the first disparity threshold is greater than the second disparity threshold.

[0138] It should be understood that when the disparity parameter is equal to the first disparity threshold or the disparity parameter is equal to the second disparity threshold, the level corresponding to the stitching complexity can also be determined as the second complexity level. This application does not limit this.

[0139] In practical applications, the disparity thresholds corresponding to scene depth and device displacement can be different. For example, a first depth threshold and a second depth threshold can be used as the disparity threshold corresponding to scene depth; a first displacement threshold and a second displacement threshold can be used as the disparity threshold corresponding to device displacement. The first depth threshold is greater than the second depth threshold, and the first displacement threshold is greater than the second displacement threshold.

[0140] Regarding step (21) above, when the disparity parameter is the scene depth, the level corresponding to the stitching complexity is determined from multiple complexity levels, including: if the scene depth is greater than or equal to the first depth threshold, the level corresponding to the stitching complexity is the first complexity level; if the scene depth is less than the first depth threshold but greater than the second depth threshold, the level corresponding to the stitching complexity is the second complexity level; if the scene depth is less than or equal to the second depth threshold, the level corresponding to the stitching complexity is the third complexity level.

[0141] Regarding step (21) above, when the parallax parameter is the device displacement, the level corresponding to the splicing complexity is determined from multiple complexity levels, including: if the device displacement is greater than or equal to the first displacement threshold, the level corresponding to the splicing complexity is the first complexity level; if the device displacement is less than the first displacement threshold but greater than the second displacement threshold, the level corresponding to the splicing complexity is the second complexity level; if the device displacement is less than or equal to the second displacement threshold, the level corresponding to the splicing complexity is the third complexity level.

[0142] Regarding step (21) above, when the parallax parameters include scene depth and device displacement, the level corresponding to the stitching complexity is determined from multiple complexity levels, including: if the scene depth is greater than or equal to a first depth threshold and the device displacement is greater than or equal to a first displacement threshold, then the level corresponding to the stitching complexity is the first complexity level; if the scene depth is less than the first depth threshold and greater than a second depth threshold, or the device displacement is less than the first displacement threshold and greater than the second displacement threshold, then the level corresponding to the stitching complexity is the second complexity level; if the scene depth is less than or equal to the second depth threshold, or the device displacement is less than or equal to the second displacement threshold, then the level corresponding to the stitching complexity is the third complexity level.

[0143] For step (22) above, the level corresponding to the stitching complexity is determined from multiple complexity levels, including: if the image offset is greater than or equal to the first offset threshold, the level corresponding to the stitching complexity is the first complexity level; if the image offset is less than the first offset threshold and greater than the second offset threshold, the level corresponding to the stitching complexity is the second complexity level, and the first offset threshold is greater than the second offset threshold; if the image offset is less than or equal to the second offset threshold, the level corresponding to the stitching complexity is the third complexity level.

[0144] It should be noted that the embodiments of this application are only illustrated with the example of three complexity levels. In practical applications, the number of complexity levels can be less or more, and the embodiments of this application do not limit the specific number. As the number of complexity levels increases, the end-to-cloud collaboration method can be further refined based on the implementation logic of this application. Without departing from the inventive concept of this application, the embodiments of this application do not limit the refinement method.

[0145] Step 302: Send an image stitching request to the cloud platform based on the stitching reference information. This image stitching request is used to instruct the cloud platform to assist the terminal device in stitching multiple images.

[0146] Since the stitching reference information includes stitching complexity and / or network status, the content of the image stitching request sent by the terminal device to the cloud platform will differ depending on the stitching reference information. Next, we will introduce three schemes for the terminal device to send image stitching requests to the cloud platform under three types of stitching reference information.

[0147] The first approach uses stitching complexity as the reference information. As described above, the stitching complexity levels in this embodiment can include a first complexity level, a second complexity level, and a third complexity level, where the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level. In this case, an image stitching request is sent to the cloud platform based on the stitching complexity level, including the following three scenarios:

[0148] In the first case, the stitching complexity level is the first complexity level. The image stitching request sent to the cloud platform carries multiple images and the first coordination instruction information, which is used to instruct the cloud platform to stitch multiple images.

[0149] In other words, when the stitching complexity level is the highest, it is quite difficult for a terminal device to stitch multiple images, and it may even be impossible to complete the stitching of multiple images. Therefore, to ensure the quality of the stitched image, the terminal device instructs the cloud platform to stitch multiple images together to generate a stitched image.

[0150] In some embodiments, the first collaborative instruction information may also be stitching reference information, that is, the cloud platform determines whether the terminal device needs the assistance of the cloud platform when stitching multiple images, and if the cloud platform needs to assist, the type of stitching auxiliary data that the cloud platform needs to feed back to the terminal device.

[0151] Therefore, when the stitching complexity is at level one, the cloud platform stitches multiple images and sends the stitched image back to the terminal device, which does not need to perform the stitching operation itself. This maximizes the quality of the stitched image.

[0152] In the second scenario, the stitching complexity level is the second complexity level. The image stitching request sent to the cloud platform carries multiple images and second coordination instruction information. The second coordination instruction information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0153] In other words, when the stitching complexity level is the second level, the entire image stitching process does not need to be completed by the cloud platform; the terminal device itself can perform some stitching operations. However, to ensure the image stitching quality, the terminal device can request the cloud platform to perform distortion transformations on multiple images and provide corresponding stitching auxiliary parameters (such as the target distortion mesh).

[0154] Therefore, when the stitching complexity level is second-highest, an edge-cloud collaborative approach is effective for stitching multiple images. In this case, the cloud platform provides stitching assistance data, rather than stitching the images themselves. This significantly reduces the amount of downlink data transmitted between the terminal device and the cloud platform, thereby lowering downlink latency and network bandwidth consumption.

[0155] In the third case, the stitching complexity level is the third complexity level. The image stitching request sent to the cloud platform carries the downsampled images corresponding to multiple images and the second coordination instruction information. The second coordination instruction information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0156] In this embodiment, multiple downsampled images have the same resolution, and all are smaller than the original image. The downsampling rate can be determined based on the lowest resolution of the multiple images stitched together by the cloud platform; this embodiment does not impose any restrictions on this.

[0157] In other words, when the stitching complexity level is the third level, the terminal device can basically complete the stitching of multiple images independently. However, in order to ensure the quality of image stitching, the terminal device can also provide the cloud platform with downsampled images corresponding to multiple images and request the cloud platform to perform distortion transformation on multiple images and provide corresponding stitching auxiliary parameters (such as target distortion mesh).

[0158] Therefore, when the stitching complexity level is the third level, the terminal device can independently perform most of the calculations in the image stitching process. While ensuring the image stitching quality, it only needs to interact with the cloud platform to transmit a small amount of data. At this time, the amount of uplink and downlink data transmitted between the terminal device and the cloud platform is minimized, which can further reduce the communication latency between the terminal device and the cloud platform, as well as the network traffic consumption when transmitting uplink and downlink data.

[0159] The second approach uses network status as the reference information for image stitching. In this case, an image stitching request is sent to the cloud platform based on the network status, including the following two scenarios:

[0160] In the first scenario, if the network status assessment value is greater than the weak network threshold, the image stitching request sent to the cloud platform carries multiple images and first coordination instruction information, which is used to instruct the cloud platform to stitch multiple images.

[0161] In some embodiments, if the terminal device can independently complete part or even all of the image stitching operation, then when the network status assessment value is greater than the weak network threshold, the image stitching request sent by the terminal device to the cloud platform can also carry downsampled images and second coordination indication information corresponding to multiple images respectively.

[0162] In this embodiment of the application, when the network status assessment value is greater than the weak network threshold, the data transmitted between the terminal device and the cloud platform is not restricted, and can be flexibly selected according to the actual situation. For example, when the network status assessment value is greater than the weak network threshold, the terminal device can send multiple images and first coordination indication information to the cloud platform; it can also send multiple images and second coordination indication information to the cloud platform; or it can send downsampled images and second coordination indication information corresponding to multiple images to the cloud platform.

[0163] In the second scenario, when the network condition is less than or equal to the weak network threshold, the image stitching request sent to the cloud platform carries downsampled images corresponding to multiple images and second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0164] In other words, when the network status assessment value is less than or equal to the weak network threshold, the performance of the communication network between the terminal device and the cloud platform is poor. To avoid data loss, increased communication latency, or network traffic consumption, only a small amount of necessary data needs to be transmitted between the terminal device and the cloud platform. Large original images and / or complete stitched images are not transmitted.

[0165] It should be understood that the aforementioned weak network threshold can be determined based on the critical value of one or more performance parameters of the communication network between the terminal device and the cloud platform, or it can be flexibly set in combination with the evaluation criteria of the terminal device in different application scenarios. This application embodiment does not limit this.

[0166] The third approach involves using stitching reference information including stitching complexity and network status. In this case, based on the stitching complexity level and network status, an image stitching request is sent to the cloud platform, including the following three scenarios:

[0167] In the first case, the stitching complexity level is the first complexity level, and the network state evaluation value is greater than the weak network threshold. The image stitching request sent to the cloud platform carries multiple images and the first coordination instruction information, which is used to instruct the cloud platform to stitch multiple images.

[0168] In the second case, the stitching complexity level is the second complexity level, and the network state evaluation value is greater than the weak network threshold. The image stitching request sent to the cloud platform carries multiple images and second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on multiple images.

[0169] In the third case, the stitching complexity level is the third complexity level, and the network state evaluation value is less than or equal to the weak network threshold. The image stitching request sent to the cloud platform carries the downsampled images corresponding to multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on multiple images.

[0170] In some embodiments, when the network status assessment value is greater than the weak network threshold, the image stitching request sent by the terminal device to the cloud platform can also carry stitching parameters, enabling the cloud platform to generate stitching auxiliary data based on the received multiple images (or corresponding multiple downsampled images) and stitching parameters.

[0171] The stitching parameters include the image depth corresponding to each of the multiple images, the movement trajectory of the terminal device during the acquisition of the multiple images, and the optical flow information corresponding to each of the multiple images. The optical flow information is used to indicate the movement of pixels in two adjacent images in the acquisition sequence.

[0172] In some embodiments, when the terminal device runs an image stitching algorithm (which has lower accuracy compared to the cloud platform), before sending an image stitching request to the cloud platform, the terminal device can perform initial stitching on multiple images based on its own capable image stitching algorithm to obtain intermediate stitching results. If the network state assessment value between the terminal device and the cloud platform is greater than the weak network threshold, the terminal device can also send the intermediate stitching results obtained from the image stitching algorithm running on the terminal device along with the image stitching request to the cloud platform. This provides the cloud platform with more reference data, reduces the computational load during the cloud platform's generation of stitched images or target distorted meshes, and accelerates the efficiency of the cloud platform in generating stitching auxiliary data.

[0173] As an example, intermediate results may include the initial twisted mesh determined by the terminal device, the initial stitched image obtained by stitching multiple images, or other data generated during the stitching process. This application embodiment does not limit these.

[0174] In some embodiments, based on the stitching assistance data returned by the cloud platform, the terminal device can directly process multiple images based on the stitching assistance data, or it can first correct the intermediate stitching results it generates based on the stitching assistance data, and then generate a stitched image based on the corrected intermediate stitching results. This application embodiment does not limit this.

[0175] In some embodiments, when a terminal device captures multiple images in panoramic mode, the preview user interface (UI) typically provides a series of auxiliary information to ensure a smooth shooting process and an ideal final stitching effect. This auxiliary information includes, but is not limited to, the following three items:

[0176] (1) Image alignment information, which describes the alignment of two adjacent images in the acquisition sequence, i.e. the alignment of the current frame image with the previous frame image.

[0177] During panoramic shooting, the preview UI displays the alignment of the current shot with the previous shot in real time. This is usually achieved through on-screen arrows or alignment lines, helping users maintain the stability of their phone's movement and ensuring proper overlap between adjacent images.

[0178] For example, when a user takes a picture of a coastline in panorama mode, the preview UI displays an arrow instructing the user to move the phone along the coastline. The user needs to keep the arrow moving along the coastline while ensuring that there is approximately 20% to 30% overlap between each photo. This way, during post-processing, the image stitching algorithm can accurately identify and stitch these photos together to form a complete and smooth stitched image.

[0179] (2) Acquisition order, that is, the order in which the terminal device acquires multiple images.

[0180] The preview UI prompts users to take photos in a specific order, usually from left to right or from top to bottom. This helps ensure the correct stitching order of the panoramic images, avoiding misalignment or breaks.

[0181] For example, when shooting cityscapes, users might need to take multiple photos sequentially from left to right to capture the entire city. The preview UI will prompt the user to shoot in this order and display the position of the current photo within the overall stitched image in real time during the shooting process.

[0182] (3) Acquisition location, which is the location of each image among multiple images acquired by the terminal device.

[0183] The preview UI also provides users with suggestions on shooting locations to ensure that the captured images are rich in content and aesthetically pleasing. These suggestions include choosing appropriate shooting angles, heights, and positions, as well as avoiding capturing unnecessary elements (such as pedestrians, pillars, etc.).

[0184] For example, when shooting a large conference room, users may need to raise their tripods to capture a wider view. The preview UI might prompt users to choose a high shooting point that overlooks the entire conference room, ensuring the image captures both the overall view and the layout and decor. The preview UI will also advise users to avoid shooting pillars or walls within the conference room to ensure a clean and aesthetically pleasing image.

[0185] The aforementioned auxiliary information helps the cloud platform to stitch or distort multiple images. Therefore, when the network status assessment value between the terminal device and the cloud platform is greater than the weak network threshold, the auxiliary information provided by the preview UI during the acquisition of multiple images can also be sent to the cloud platform, thereby maximizing the quality of the stitched images and accelerating the efficiency of the cloud platform in generating stitching auxiliary data.

[0186] Step 303: Receive stitching auxiliary data sent by the cloud platform, and determine the stitched image corresponding to multiple images based on the stitching auxiliary data.

[0187] Since the image stitching request sent by the terminal device to the cloud platform carries either the first collaboration instruction information or the second collaboration instruction information, the collaboration method indicated by the terminal device is different under these two types of information. The cloud platform responds to different collaboration instruction information and returns different stitching auxiliary data.

[0188] When the image stitching request carries the first collaborative instruction information, the implementation process of step 303 above can be as follows: receive the stitched image obtained after the cloud platform stitches multiple images, and use the stitched image as stitching auxiliary data.

[0189] In other words, the stitching assistance data sent by the cloud platform is the stitched image corresponding to multiple images. The terminal device only needs to receive and download the complete stitched image, without having to perform the image stitching operation itself.

[0190] When the image stitching request carries the second coordination instruction information, the implementation process of step 303 above can be as follows: receive the target distorted mesh obtained by the cloud platform after distorting multiple images, and use the target distorted mesh as stitching auxiliary data.

[0191] The target warped mesh indicates the geometric transformation operation for stitching multiple images. In some scenarios, the warped mesh is also called a deformable mesh or a mapped mesh. It is typically a two-dimensional mesh where each point corresponds to a pixel or a small region in the image. By adjusting the shape and position of the target warped mesh, various complex geometric transformations can be performed on the corresponding image, such as stretching, shrinking, and warping.

[0192] In one possible implementation, the process of determining the stitched image based on the target warped mesh can be as follows: based on the target warped mesh, multiple images are transformed separately using an image warping function; the transformed multiple images are stitched together according to the acquisition order to obtain the stitched image.

[0193] The image warping function is a function that performs geometric transformations on an image based on a target warped mesh. It takes the image and the target warped mesh as input and outputs the geometrically transformed image. In the process of stitching multiple images together, the image warping function is used to map each image to its correct position in the stitched image, based on the geometric transformation operation indicated by the calculated target warped mesh.

[0194] In some embodiments, during the process of stitching multiple transformed images into a stitched image, some boundary issues in the stitching result can also be addressed, such as color correction and brightness adjustment, to ensure the smoothness and consistency of the stitching result.

[0195] In summary, before sending multiple images to be stitched to the cloud platform, the terminal device assesses its own ability to stitch multiple images and intelligently decides whether to request the cloud platform's assistance in stitching the images. Simultaneously, it assesses the performance of the communication network between the terminal device and the platform, intelligently deciding on the uplink and downlink data transmission during the collaborative generation of the stitched image. Thus, by having the cloud platform assist in stitching multiple images and providing stitching assistance data to the terminal device, the powerful computing capabilities of the cloud platform can be effectively utilized, reducing the resource burden on the terminal device while ensuring the quality of the stitched images. Furthermore, the technical solution of this application, which involves the cloud platform assisting in stitching multiple images and providing stitching assistance data, reduces the communication latency between the terminal device and the cloud platform compared to a solution where the cloud platform independently stitches and generates the image and then feeds it back to the terminal device. Moreover, with the cloud platform providing stitching assistance data, the speed at which the terminal device generates the stitched image is also increased, significantly improving the user experience. The aforementioned edge-cloud collaborative image stitching solution can adapt to different scenarios and changing needs of terminal devices, possessing a certain degree of flexibility and scalability.

[0196] Based on the image stitching method shown in Figure 3 above, the technical solution provided by the embodiments of this application will be illustrated by way of example, taking the stitching reference information including parallax parameters (scene depth and device displacement) and image offset. Figures 4 and 5 below are merely exemplary combination schemes. Without departing from the inventive concept of this application, the relevant parameters in the embodiments of Figure 3 above can also be combined to form other schemes, which will not be listed one by one here, but should all be included within the protection scope of this application.

[0197] Referring to Figure 4, when the terminal device captures multiple images in panoramic mode, the scene depth can be determined based on binocular depth, the device displacement can be determined based on IMU information, and the image offset can be determined based on the preview optical flow of the captured images. This application embodiment does not limit the method of determining these parameters; please refer to the relevant examples and descriptions in the embodiment of Figure 3 above.

[0198] After acquiring N images, the terminal device determines the end-to-cloud collaboration strategy based on scene depth, device displacement, and image offset. This strategy is used to acquire cloud data and stitching auxiliary data that needs to be provided by the cloud platform.

[0199] The cloud-attached data refers to the data carried in the image stitching request. The cloud-attached data determined based on the edge-cloud collaboration strategy includes: multiple images and first collaboration indication information, or multiple images and second collaboration indication information, or downsampled images and second collaboration indication information corresponding to multiple images respectively.

[0200] Among them, the stitching auxiliary data includes the stitched image corresponding to multiple images, or the target distorted mesh indicating the geometric transformation operation for stitching multiple images.

[0201] Furthermore, after receiving the stitching assistance data returned by the cloud platform, the terminal device determines the stitched image based on the stitching assistance data and previews and displays the stitched image.

[0202] Referring to Figure 5, when the parallax parameter is scene depth or device displacement, and the stitching reference information also includes the network status assessment value, the end-to-cloud collaboration scheme determined by the terminal device includes:

[0203] (1) If the network state assessment value is less than or equal to the weak network threshold, or the disparity parameter is less than or equal to the second disparity threshold and the image offset is less than or equal to the second offset threshold, then multiple downsampled images corresponding to each image are sent to the cloud platform. The cloud platform runs an image stitching algorithm to stitch the multiple downsampled images together and returns the target warped mesh to the terminal device. The terminal device determines the stitched image based on the target warped mesh using an image warping function.

[0204] (2) If the network state assessment value is greater than the weak network threshold, and the second disparity threshold is less than the disparity parameter and less than the first disparity threshold, or the second offset threshold is less than the image offset and less than the first offset threshold, then multiple images are sent to the cloud platform. The cloud platform runs an image stitching algorithm to stitch the multiple images together and returns the target warped mesh to the terminal device. The terminal device determines the stitched image based on the target warped mesh using an image warping function.

[0205] (3) If the network status assessment value is greater than the weak network threshold, and the disparity parameter is greater than or equal to the first disparity threshold or the image offset is greater than or equal to the first offset threshold, then multiple images are sent to the cloud platform, and the cloud platform runs the image stitching algorithm to stitch the multiple images together and returns the stitched image to the terminal device.

[0206] It should be understood that the above example and the image stitching method shown in Figure 3 above belong to the same inventive concept. The relevant implementation methods and beneficial effects can also be referred to the description of the embodiment in Figure 3 above, and will not be repeated here.

[0207] The image stitching device provided in the embodiments of this application will be described next.

[0208] Figure 6 is a schematic diagram of an image stitching device provided in an embodiment of this application. This image stitching device can be implemented by software, hardware, or a combination of both as part or all of the terminal device shown in Figure 1. Referring to Figure 6, the image stitching device includes: an information acquisition module 601, a sending module 602, and a receiving module 603.

[0209] The information acquisition module 601 is used to obtain stitching reference information corresponding to multiple images. The stitching reference information includes stitching complexity and / or network status. The stitching complexity indicates the difficulty for the terminal device to stitch multiple images, and the network status indicates the performance of the communication network between the terminal device and the cloud platform. For detailed implementation process, please refer to the relevant description of step 301 in the embodiment shown in Figure 3, which will not be repeated here.

[0210] The sending module 602 is used to send an image stitching request to the cloud platform based on the stitching reference information. The image stitching request is used to instruct the cloud platform to assist the terminal device in stitching multiple images. For detailed implementation process, please refer to the relevant description of step 302 in the embodiment shown in Figure 3, which will not be repeated here.

[0211] The receiving module 603 is used to receive stitching auxiliary data sent by the cloud platform and determine the stitched image corresponding to multiple images based on the stitching auxiliary data. For detailed implementation process, please refer to the relevant description of step 303 in the embodiment shown in Figure 3, which will not be repeated here.

[0212] In one possible implementation, the splicing reference information is the splicing complexity, and the sending module includes:

[0213] The determining unit is used to determine the level corresponding to the splicing complexity from multiple complexity levels, including a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level.

[0214] The sending unit is used to send image stitching requests to the cloud platform based on the level corresponding to the stitching complexity.

[0215] In one possible implementation, the level corresponding to the stitching complexity is the first complexity level. The image stitching request carries multiple images and first collaborative instruction information, which is used to instruct the cloud platform to stitch multiple images.

[0216] In one possible implementation, the stitching complexity level is the second complexity level. The image stitching request carries multiple images and second collaborative instruction information, which is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0217] In one possible implementation, the stitching complexity level is the third complexity level. The image stitching request carries downsampled images corresponding to multiple images and second collaborative indication information. The second collaborative indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0218] In one possible implementation, the splicing reference information includes splicing complexity and network status, and the sending module 602 is specifically used for:

[0219] The determining unit is used to determine the level corresponding to the splicing complexity from multiple complexity levels, including a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level.

[0220] The sending unit is used to send image stitching requests to the cloud platform based on the level corresponding to the stitching complexity and the network status.

[0221] In one possible implementation, the stitching complexity level is the first complexity level, and the network state evaluation value is greater than the weak network threshold. The image stitching request carries multiple images and first collaborative instruction information, which is used to instruct the cloud platform to stitch multiple images.

[0222] In one possible implementation, the stitching complexity level is the second complexity level, and the network state evaluation value is greater than the weak network threshold. The image stitching request carries multiple images and second collaborative indication information, which is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0223] In one possible implementation, the stitching complexity level is the third complexity level, and the network state evaluation value is less than or equal to the weak network threshold. The image stitching request carries downsampled images corresponding to multiple images and second collaborative indication information. The second collaborative indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0224] In one possible implementation, the stitching reference information is the network state, the state evaluation value of the network state is greater than the weak network threshold, and the image stitching request carries multiple images and first coordination indication information, which is used to instruct the cloud platform to stitch multiple images.

[0225] In one possible implementation, the stitching reference information is the network state, the state evaluation value of the network state is less than or equal to the weak network threshold, and the image stitching request carries downsampled images corresponding to multiple images and second collaborative indication information. The second collaborative indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

[0226] In one possible implementation, the receiving module 603 is further configured to:

[0227] The system receives a stitched image obtained by stitching multiple images from a cloud platform and uses this stitched image as auxiliary data for stitching.

[0228] In one possible implementation, the receiving module 603 is further configured to:

[0229] The target warped mesh is obtained by receiving multiple images from the cloud platform after warping transformation. The target warped mesh is used as stitching auxiliary data, and the target warped mesh indicates the geometric transformation operation for stitching multiple images.

[0230] In one possible implementation, the stitching reference information includes stitching complexity, and multiple images are acquired consecutively from the same scene; the information acquisition module 601 includes:

[0231] The first acquisition unit is used to acquire disparity parameters corresponding to multiple images. The disparity parameters include the scene depth of the scene and / or the device displacement of the terminal device during the acquisition of multiple images.

[0232] The second acquisition unit is used to obtain the splicing complexity based on the disparity parameter.

[0233] In one possible implementation, the disparity parameter is either scene depth or device displacement, and the aforementioned determining unit is specifically used for:

[0234] If the disparity parameter is greater than or equal to the first disparity threshold, then the level corresponding to the stitching complexity is the first complexity level.

[0235] If the disparity parameter is less than the first disparity threshold and greater than the second disparity threshold, then the level corresponding to the stitching complexity is the second complexity level, and the first disparity threshold is greater than the second disparity threshold.

[0236] If the disparity parameter is less than or equal to the second disparity threshold, then the level of splicing complexity is the third level of complexity.

[0237] Specifically, when the parallax parameter is the scene depth, the first parallax threshold is the first depth threshold, and the second parallax threshold is the second depth threshold; when the parallax parameter is the device displacement, the first parallax threshold is the first displacement threshold, and the second parallax threshold is the second displacement threshold.

[0238] In one possible implementation, the parallax parameters include scene depth and device displacement, and the aforementioned determining unit is specifically used for:

[0239] If the scene depth is greater than or equal to the first depth threshold and the device displacement is greater than or equal to the first displacement threshold, then the level corresponding to the stitching complexity is the first complexity level.

[0240] If the scene depth is less than the first depth threshold and greater than the second depth threshold, or if the device displacement is less than the first displacement threshold and greater than the second displacement threshold, then the stitching complexity level is the second complexity level, the first disparity threshold is greater than the second disparity threshold, and the first displacement threshold is greater than the second displacement threshold.

[0241] If the scene depth is less than or equal to the second depth threshold, or the device displacement is less than or equal to the second displacement threshold, then the level corresponding to the stitching complexity is the third complexity level.

[0242] In one possible implementation, the stitching reference information includes stitching complexity, and multiple images are acquired consecutively from the same scene; the information acquisition module 601 includes:

[0243] The third acquisition unit is used to acquire the image offsets corresponding to multiple images. The image offsets are used to describe the motion of the same object in multiple images.

[0244] The fourth acquisition unit is used to obtain the stitching complexity based on the image offset.

[0245] In one possible implementation, the aforementioned determining unit is specifically used for:

[0246] If the image offset is greater than or equal to the first offset threshold, the stitching complexity level is the first complexity level.

[0247] If the image offset is less than the first offset threshold and greater than the second offset threshold, then the level of stitching complexity is the second complexity level, and the first offset threshold is greater than the second offset threshold.

[0248] If the image offset is less than or equal to the second offset threshold, the stitching complexity level is the third complexity level.

[0249] In one possible implementation, the image stitching request also carries stitching parameters, which include the image depths corresponding to the multiple images, the movement trajectory of the terminal device during the acquisition of the multiple images, and the optical flow information corresponding to the multiple images. The optical flow information is used to indicate the movement of pixels in two adjacent images in the acquisition sequence.

[0250] In one possible implementation, the stitching parameters also include one or more of the following: image alignment information of multiple images, acquisition order, and acquisition position. The image alignment information is used to describe the alignment of two images that are adjacent in the acquisition order.

[0251] In one possible implementation, the image stitching device further includes:

[0252] The image stitching module is used to initially stitch together multiple images to obtain intermediate stitching results;

[0253] The splicing parameters also include intermediate splicing results.

[0254] In one possible implementation, the stitching auxiliary data is a target warped mesh, which indicates the geometric transformation operation for stitching multiple images; the receiving module 603 is also used for:

[0255] Based on the target warped mesh, multiple images are transformed separately using an image warping function;

[0256] The transformed images are stitched together in the order they were captured to obtain a stitched image.

[0257] In this embodiment, before sending multiple images to be stitched to the cloud platform, the image stitching device intelligently decides whether to request the cloud platform's assistance in stitching multiple images by evaluating the terminal device's ability to stitch multiple images. Simultaneously, it evaluates the performance of the communication network between the terminal device and the platform, intelligently deciding on the uplink and downlink data transmission during end-cloud collaborative image generation. Thus, by using the cloud platform to assist in stitching multiple images and providing stitching assistance data to the terminal device, the powerful computing capabilities of the cloud platform can be effectively utilized, reducing the resource burden on the terminal device while ensuring image stitching quality. Furthermore, the technical solution of this application, where the cloud platform assists in stitching multiple images and provides stitching assistance data, reduces communication latency between the terminal device and the cloud platform compared to a solution where the cloud platform independently stitches and generates images and then feeds them back to the terminal device. Moreover, with the cloud platform providing stitching assistance data, the speed at which the terminal device generates stitched images is also increased. The aforementioned end-cloud collaborative image stitching scheme can adapt to different scenarios and varying terminal device requirements, exhibiting flexibility and scalability.

[0258] It should be noted that the image stitching device provided in the above embodiments is only illustrated by the division of the above functional modules when acquiring the stitched image corresponding to multiple images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image stitching device and the image stitching method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0259] The terminal device provided in the embodiments of this application will be described next.

[0260] Figure 7 is a structural block diagram of a terminal device provided in an embodiment of this application. The terminal device 700 can be the terminal device shown in Figure 2, used for data interaction with a cloud platform to implement the image stitching method provided in this embodiment. Exemplarily, the terminal device 700 can be a portable mobile terminal, such as a smartphone, camera, or laptop. In some scenarios, the terminal device 700 may also be referred to as a user device, portable terminal device, laptop terminal device, desktop terminal device, or other names. Referring to Figure 7, the terminal device 700 typically includes a processor 701 and a memory 702.

[0261] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0262] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 is used to store at least one instruction, which is executed by the processor 701 to implement the image stitching method shown in the above-described method embodiments of this application.

[0263] In some embodiments, the terminal device 700 further includes a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 704, a touch display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708, and a power supply 709.

[0264] Peripheral device interface 703 can be used to connect at least one input / output (I / O) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0265] The radio frequency (RF) circuit 704 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminal devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi networks. In some embodiments, the RF circuit 704 may also include circuitry related to near-field communication (NFC), which is not limited in this application embodiment.

[0266] Display screen 705 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 705, which serves as the front panel of the terminal device 700; in other embodiments, there may be at least two display screens 705, respectively disposed on different surfaces of the terminal device 700 or in a folded design; in some embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of the terminal device 700. Furthermore, display screen 705 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as liquid crystal display (LCD) or organic light-emitting diode (OLED).

[0267] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal device, and the rear-facing camera is located on the back of the terminal device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, virtual reality (VR) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0268] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0269] The positioning component 708 is used to locate the current geographic location of the terminal device 700 in order to enable navigation or location-based services (LBS).

[0270] Power supply 709 is used to supply power to the various components in terminal device 700. Power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 709 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0271] In some embodiments, the terminal device 700 further includes one or more sensors 710. The one or more sensors 710 include, but are not limited to, accelerometers, gyroscopes, pressure sensors, fingerprint sensors, optical sensors, and proximity sensors.

[0272] Those skilled in the art will understand that the structure shown in FIG7 does not constitute a limitation on the terminal device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0273] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the image stitching method described above.

[0274] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the above-described image stitching method.

[0275] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0276] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0277] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the device shooting parameters, stitching reference information, image data, etc. involved in the embodiments of this application were all obtained with full authorization.

[0278] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image stitching method, characterized in that, Applied to a terminal device, the method includes: Obtain stitching reference information corresponding to multiple images. The stitching reference information includes stitching complexity and / or network status. The stitching complexity indicates the difficulty of stitching the multiple images by the terminal device, and the network status indicates the performance of the communication network between the terminal device and the cloud platform. Based on the stitching reference information, an image stitching request is sent to the cloud platform, and the image stitching request is used to instruct the cloud platform to assist the terminal device in stitching the multiple images; The system receives stitching assistance data sent by the cloud platform and determines the stitched image corresponding to the multiple images based on the stitching assistance data.

2. The method as described in claim 1, characterized in that, The stitching reference information is the stitching complexity, and the step of sending an image stitching request to the cloud platform based on the stitching reference information includes: The concatenation complexity level is determined from multiple complexity levels, including a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level. The image stitching request is sent to the cloud platform based on the level corresponding to the stitching complexity.

3. The method as described in claim 2, characterized in that, The level corresponding to the stitching complexity is the first complexity level. The image stitching request carries the multiple images and the first coordination instruction information, which is used to instruct the cloud platform to stitch the multiple images.

4. The method as described in claim 2, characterized in that, The level corresponding to the stitching complexity is the second complexity level. The image stitching request carries the multiple images and the second coordination instruction information. The second coordination instruction information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

5. The method as described in claim 2, characterized in that, The level corresponding to the stitching complexity is the third complexity level. The image stitching request carries the downsampled image corresponding to each of the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

6. The method as described in claim 1, characterized in that, The stitching reference information includes the stitching complexity and the network status. Sending the image stitching request to the cloud platform based on the stitching reference information includes: The concatenation complexity level is determined from multiple complexity levels, including a first complexity level, a second complexity level, and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level. The image stitching request is sent to the cloud platform based on the level corresponding to the stitching complexity and the network status.

7. The method as described in claim 6, characterized in that, The level corresponding to the stitching complexity is the first complexity level, and the state evaluation value of the network status is greater than the weak network threshold. The image stitching request carries the multiple images and the first coordination indication information, which is used to instruct the cloud platform to stitch the multiple images.

8. The method as described in claim 6, characterized in that, The level corresponding to the stitching complexity is the second complexity level, and the state evaluation value of the network state is greater than the weak network threshold. The image stitching request carries the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

9. The method as described in claim 6, characterized in that, The level corresponding to the stitching complexity is the third complexity level, and the state evaluation value of the network state is less than or equal to the weak network threshold. The image stitching request carries the downsampled image and the second coordination indication information corresponding to the multiple images respectively. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

10. The method as described in claim 1, characterized in that, The stitching reference information is the network status, the state evaluation value of the network status is greater than the weak network threshold, and the image stitching request carries the multiple images and the first coordination indication information, which is used to instruct the cloud platform to stitch the multiple images.

11. The method as described in claim 1, characterized in that, The stitching reference information is the network state, the state evaluation value of the network state is less than or equal to the weak network threshold, and the image stitching request carries the downsampled image corresponding to each of the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

12. The method as described in claim 3, 7, or 10, characterized in that, The receipt of splicing assistance data sent by the cloud platform includes: The cloud platform receives the stitched image obtained by stitching together the multiple images, and uses the stitched image as the stitching auxiliary data.

13. The method as described in claim 4, 5, 8, 9 or 11, characterized in that, The receipt of splicing assistance data sent by the cloud platform includes: The system receives a target distorted mesh obtained by the cloud platform after distorting the multiple images, and uses the target distorted mesh as the stitching auxiliary data. The target distorted mesh indicates the geometric transformation operation for stitching the multiple images.

14. The method according to any one of claims 2-9, characterized in that, The multiple images were acquired consecutively in the same scene; the stitching reference information corresponding to the multiple images includes: Obtain the disparity parameters corresponding to the multiple images, wherein the disparity parameters include the scene depth of the scene and / or the device displacement of the terminal device during the acquisition of the multiple images; The stitching complexity is obtained based on the disparity parameter.

15. The method as described in claim 14, characterized in that, The parallax parameter is either the scene depth or the device displacement. Determining the stitching complexity level from multiple complexity levels includes: If the disparity parameter is greater than or equal to the first disparity threshold, then the level corresponding to the stitching complexity is the first complexity level; If the disparity parameter is less than the first disparity threshold and greater than the second disparity threshold, then the level corresponding to the stitching complexity is the second complexity level, and the first disparity threshold is greater than the second disparity threshold. If the disparity parameter is less than or equal to the second disparity threshold, then the level corresponding to the stitching complexity is the third complexity level; Wherein, when the disparity parameter is the scene depth, the first disparity threshold is a first depth threshold and the second disparity threshold is a second depth threshold; when the disparity parameter is the device displacement, the first disparity threshold is a first displacement threshold and the second disparity threshold is a second displacement threshold.

16. The method as described in claim 14, characterized in that, The parallax parameter includes the scene depth and the device displacement, and determining the stitching complexity level from multiple complexity levels includes: If the scene depth is greater than or equal to a first depth threshold, and the device displacement is greater than or equal to a first displacement threshold, then the level corresponding to the splicing complexity is the first complexity level. If the scene depth is less than the first depth threshold and greater than the second depth threshold, or if the device displacement is less than the first displacement threshold and greater than the second displacement threshold, then the level corresponding to the stitching complexity is the second complexity level, the first disparity threshold is greater than the second disparity threshold, and the first displacement threshold is greater than the second displacement threshold. If the scene depth is less than or equal to the second depth threshold, or if the device displacement is less than or equal to the second displacement threshold, then the level corresponding to the splicing complexity is the third complexity level.

17. The method according to any one of claims 2-9, characterized in that, The multiple images were acquired consecutively from the same scene, and the stitching reference information corresponding to the multiple images includes: Obtain the image offset corresponding to the multiple images, and the image offset is used to describe the motion of the same object in the multiple images; The stitching complexity is obtained based on the image offset.

18. The method as described in claim 17, characterized in that, Determining the level corresponding to the concatenation complexity from multiple complexity levels includes: If the image offset is greater than or equal to the first offset threshold, then the level corresponding to the stitching complexity is the first complexity level; If the image offset is less than the first offset threshold and greater than the second offset threshold, then the level corresponding to the stitching complexity is the second complexity level, and the first offset threshold is greater than the second offset threshold. If the image offset is less than or equal to the second offset threshold, then the level corresponding to the stitching complexity is the third complexity level.

19. The method according to any one of claims 1-18, characterized in that, The image stitching request also carries stitching parameters, which include the image depth corresponding to each of the multiple images, the movement trajectory of the terminal device during the acquisition of the multiple images, and the optical flow information corresponding to each of the multiple images. The optical flow information is used to indicate the movement of pixels in two adjacent images in the acquisition sequence.

20. The method as described in claim 19, characterized in that, The stitching parameters also include one or more of the following: image alignment information, acquisition order, and acquisition position of the multiple images. The image alignment information is used to describe the alignment of two images that are adjacent in the acquisition order.

21. The method as described in claim 19, characterized in that, Before sending the image stitching request to the cloud platform based on the stitching reference information, the method further includes: The multiple images are initially stitched together to obtain an intermediate stitching result; The splicing parameters also include the intermediate splicing results.

22. The method according to any one of claims 1-21, characterized in that, The stitching auxiliary data is a target distorted mesh, which indicates the geometric transformation operation for stitching the multiple images; determining the stitched image corresponding to the multiple images based on the stitching auxiliary data includes: Based on the target distorted mesh, the multiple images are transformed using an image distortion function; The transformed images are stitched together in the order they were captured to obtain the stitched image.

23. An image stitching device, characterized in that, The device is included in the terminal device, and the device includes: The information acquisition module is used to obtain stitching reference information corresponding to multiple images. The stitching reference information includes stitching complexity and / or network status. The stitching complexity indicates the difficulty of the terminal device stitching the multiple images, and the network status indicates the performance of the communication network between the terminal device and the cloud platform. The sending module is used to send an image stitching request to the cloud platform based on the stitching reference information. The image stitching request is used to instruct the cloud platform to assist the terminal device in stitching the multiple images. The receiving module is used to receive stitching auxiliary data sent by the cloud platform and determine the stitched image corresponding to the multiple images based on the stitching auxiliary data.

24. The apparatus as claimed in claim 23, characterized in that, The splicing reference information is the splicing complexity, and the sending module includes: A determining unit is configured to determine the level corresponding to the splicing complexity from multiple complexity levels, the multiple complexity levels including a first complexity level, a second complexity level and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level; The sending unit is used to send an image stitching request to the cloud platform based on the level corresponding to the stitching complexity.

25. The apparatus as claimed in claim 24, characterized in that, The level corresponding to the stitching complexity is the first complexity level. The image stitching request carries the multiple images and the first coordination instruction information, which is used to instruct the cloud platform to stitch the multiple images.

26. The apparatus as claimed in claim 24, characterized in that, The level corresponding to the stitching complexity is the second complexity level. The image stitching request carries the multiple images and the second coordination instruction information. The second coordination instruction information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

27. The apparatus as claimed in claim 24, characterized in that, The level corresponding to the stitching complexity is the third complexity level. The image stitching request carries the downsampled image corresponding to each of the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

28. The apparatus as claimed in claim 23, characterized in that, The splicing reference information includes the splicing complexity and the network status; the sending module is specifically used for: A determining unit is configured to determine the level corresponding to the splicing complexity from multiple complexity levels, the multiple complexity levels including a first complexity level, a second complexity level and a third complexity level, wherein the first complexity level is higher than the second complexity level, and the second complexity level is higher than the third complexity level; The sending unit is used to send an image stitching request to the cloud platform based on the level corresponding to the stitching complexity and the network status.

29. The apparatus as claimed in claim 28, characterized in that, The level corresponding to the stitching complexity is the first complexity level, and the state evaluation value of the network status is greater than the weak network threshold. The image stitching request carries the multiple images and the first coordination indication information, which is used to instruct the cloud platform to stitch the multiple images.

30. The apparatus as claimed in claim 28, characterized in that, The level corresponding to the stitching complexity is the second complexity level, and the state evaluation value of the network state is greater than the weak network threshold. The image stitching request carries the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

31. The apparatus as claimed in claim 28, characterized in that, The level corresponding to the stitching complexity is the third complexity level, and the state evaluation value of the network state is less than or equal to the weak network threshold. The image stitching request carries the downsampled image and the second coordination indication information corresponding to the multiple images respectively. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

32. The apparatus as claimed in claim 23, characterized in that, The stitching reference information is the network status, the state evaluation value of the network status is greater than the weak network threshold, and the image stitching request carries the multiple images and the first coordination indication information, which is used to instruct the cloud platform to stitch the multiple images.

33. The apparatus as claimed in claim 23, characterized in that, The stitching reference information is the network state, the state evaluation value of the network state is less than or equal to the weak network threshold, and the image stitching request carries the downsampled image corresponding to each of the multiple images and the second coordination indication information. The second coordination indication information is used to instruct the cloud platform to perform distortion transformation on the multiple images.

34. The apparatus as claimed in claim 25, 29 or 32, characterized in that, The receiving module is further configured to: The cloud platform receives the stitched image obtained by stitching together the multiple images, and uses the stitched image as the stitching auxiliary data.

35. The apparatus as claimed in claim 26, 27, 30, 31 or 33, characterized in that, The receiving module is further configured to: The system receives a target distorted mesh obtained by the cloud platform after distorting the multiple images, and uses the target distorted mesh as the stitching auxiliary data. The target distorted mesh indicates the geometric transformation operation for stitching the multiple images.

36. The apparatus according to any one of claims 24-31, characterized in that, The multiple images were acquired consecutively from the same scene; the information acquisition module includes: The first acquisition unit is used to acquire the disparity parameters corresponding to the multiple images, wherein the disparity parameters include the scene depth of the same scene and / or the device displacement of the terminal device during the acquisition of the multiple images; The second acquisition unit is used to obtain the stitching complexity based on the disparity parameter.

37. The apparatus as claimed in claim 36, characterized in that, The parallax parameter is either the scene depth or the device displacement; the determining unit is specifically used for: If the disparity parameter is greater than or equal to the first disparity threshold, then the level corresponding to the stitching complexity is the first complexity level; If the disparity parameter is less than the first disparity threshold and greater than the second disparity threshold, then the level corresponding to the stitching complexity is the second complexity level, and the first disparity threshold is greater than the second disparity threshold. If the disparity parameter is less than or equal to the second disparity threshold, then the level corresponding to the stitching complexity is the third complexity level; Wherein, when the disparity parameter is the scene depth, the first disparity threshold is a first depth threshold and the second disparity threshold is a second depth threshold; when the disparity parameter is the device displacement, the first disparity threshold is a first displacement threshold and the second disparity threshold is a second displacement threshold.

38. The apparatus as claimed in claim 36, characterized in that, The parallax parameters include the scene depth and the device displacement; the determining unit is specifically used for: If the scene depth is greater than or equal to a first depth threshold, and the device displacement is greater than or equal to a first displacement threshold, then the level corresponding to the splicing complexity is the first complexity level. If the scene depth is less than the first depth threshold and greater than the second depth threshold, or if the device displacement is less than the first displacement threshold and greater than the second displacement threshold, then the level corresponding to the stitching complexity is the second complexity level, the first disparity threshold is greater than the second disparity threshold, and the first displacement threshold is greater than the second displacement threshold. If the scene depth is less than or equal to the second depth threshold, or if the device displacement is less than or equal to the second displacement threshold, then the level corresponding to the splicing complexity is the third complexity level.

39. The apparatus according to any one of claims 24-31, characterized in that, The multiple images were acquired consecutively from the same scene; the information acquisition module includes: The third acquisition unit is used to acquire the image offset corresponding to the multiple images, and the image offset is used to describe the motion of the same object in the multiple images; The fourth acquisition unit is used to obtain the stitching complexity based on the image offset.

40. The apparatus as claimed in claim 39, characterized in that, The determining unit is specifically used for: If the image offset is greater than or equal to the first offset threshold, then the level corresponding to the stitching complexity is the first complexity level; If the image offset is less than the first offset threshold and greater than the second offset threshold, then the level corresponding to the stitching complexity is the second complexity level, and the first offset threshold is greater than the second offset threshold. If the image offset is less than or equal to the second offset threshold, then the level corresponding to the stitching complexity is the third complexity level.

41. The apparatus according to any one of claims 23-40, characterized in that, The image stitching request also carries stitching parameters, which include the image depth corresponding to each of the multiple images, the movement trajectory of the terminal device during the acquisition of the multiple images, and the optical flow information corresponding to each of the multiple images. The optical flow information is used to indicate the movement of pixels in two adjacent images in the acquisition sequence.

42. The apparatus as claimed in claim 41, characterized in that, The stitching parameters also include one or more of the following: image alignment information, acquisition order, and acquisition position of the multiple images. The image alignment information is used to describe the alignment of two images that are adjacent in the acquisition order.

43. The apparatus as claimed in claim 41, characterized in that, The device further includes: The image stitching module is used to perform initial stitching on the multiple images to obtain intermediate stitching results; The splicing parameters also include the intermediate splicing results.

44. The apparatus according to any one of claims 23-43, characterized in that, The stitching auxiliary data is a target warped mesh, which indicates the geometric transformation operation for stitching the multiple images; the receiving module is further configured to: Based on the target distorted mesh, the multiple images are transformed using an image distortion function; The transformed images are stitched together in the order they were captured to obtain the stitched image.

45. A terminal device, characterized in that, The terminal device includes a memory and a processor, the memory being used to store a computer program, and the processor being configured to execute the computer program stored in the memory to implement the method according to any one of claims 1-22.

46. ​​A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1-22.

47. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method according to any one of claims 1-22.

Images