Image processing method and electronic device

Abstract

The present application discloses an image processing method and an electronic device. The image processing method comprises: acquiring a plurality of images to be processed of a target object, said images comprising color images and initial depth images; respectively generating a plurality of corresponding initial three-dimensional meshes on the basis of the plurality of initial depth images; performing fusion and stitching on the plurality of initial depth images to generate a merged depth image; on the basis of position information of the plurality of color images, performing coordinate translation and curved surface reconstruction on the plurality of initial three-dimensional meshes to obtain a merged and stitched mesh; and correcting the merged and stitched mesh by means of the merged depth image to obtain a target three-dimensional mesh.

Description

Image processing methods and electronic devices

[0001] This application claims priority to Chinese Patent Application No. 202411960869.0, filed on December 27, 2024, entitled "Image Processing Method and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of machine vision, and more specifically, to image processing methods and electronic devices. Background Technology

[0003] In the field of machine vision, surface reconstruction of an object is typically achieved by acquiring its RGBD data (RGBD data includes RGB images and depth information). Related techniques involve constructing a triangular mesh based on points in the image's point cloud matrix and then stitching together all the triangular meshes. However, the surface obtained using this method sometimes does not match the real object, making it unsuitable for applications requiring high precision. Summary of the Invention

[0004] This application aims to address one of the technical problems in related technologies to a certain extent. To this end, this application provides an image processing method and an electronic device.

[0005] As a first aspect of this application, an image processing method is provided, wherein the image processing method includes:

[0006] Acquire multiple images of the target object to be processed, including a color image and an initial depth image;

[0007] Multiple corresponding initial 3D meshes are generated based on the multiple initial depth images;

[0008] Multiple initial depth images are fused and stitched together to generate a merged depth image;

[0009] Based on the positional information of multiple color images, coordinate translation and surface reconstruction are performed on multiple initial 3D meshes to obtain a merged and stitched mesh;

[0010] The merged and stitched mesh is corrected using the merged depth image to obtain the target 3D mesh.

[0011] Optionally, generating multiple corresponding initial stereo meshes based on the multiple initial depth images includes:

[0012] Multiple corresponding original 3D meshes are generated based on the multiple initial depth images;

[0013] Multiple original 3D meshes are simplified to obtain multiple initial 3D meshes, wherein each initial 3D mesh is a triangular mesh and the number of faces of the triangular mesh is a given value;

[0014] In the step of surface reconstruction based on the initial 3D mesh and the merged depth image, surface reconstruction is performed based on multiple triangular meshes and the stitched depth image to obtain the target 3D mesh.

[0015] Optionally, at least two of the plurality of images to be processed have overlapping areas, the plurality of images to be processed are images obtained by capturing images of the target object at the same height, and the plurality of images to be processed are captured from different angles, with the difference in the capture angle between two adjacent images to be processed not exceeding a set angle.

[0016] The step of performing coordinate translation and surface reconstruction on multiple initial 3D meshes based on the positional information of multiple color images to obtain a merged and stitched mesh includes:

[0017] The grid vertex coordinates of the initial 3D grids are translated based on the optimized image positions of the multiple color images;

[0018] Remove the overlapping mesh areas to obtain the intermediate mesh;

[0019] The intermediate mesh is updated by performing mesh vertex coordinate updates based on the merged depth image to obtain the updated mesh.

[0020] The updated mesh is implicitly reconstructed to obtain the merged and stitched mesh.

[0021] Optionally, the step of correcting the merged and stitched mesh using the merged depth image to obtain the target 3D mesh includes:

[0022] Based on the merged depth image, the mesh vertex depth values ​​of the merged stitched mesh are corrected to obtain a corrected 3D mesh;

[0023] The modified 3D mesh is simplified to obtain the target 3D mesh.

[0024] Optionally, before fusing and stitching the multiple initial depth images to generate a merged depth image, the image processing method further includes:

[0025] Image registration is performed on multiple color images according to feature matching to determine the position information of each color image.

[0026] Optionally, the step of performing image registration on the multiple color images according to feature matching to determine the position information of the multiple color images includes:

[0027] Feature points in each of the color images are extracted using a predefined operator.

[0028] Feature matching is performed between feature points in the color image and feature points in adjacent color images to determine the relative positions between two adjacent color images.

[0029] Optionally, the step of performing feature matching between feature points in the color image and adjacent color images to determine the relative positions between two adjacent color images includes:

[0030] The relative positions between the two color images are determined according to formula (1):

[0031] Among them, T ij Let represent the relative positions of the i-th color image and the j-th color image;

[0032] T i Let i be the position of the i-th color image;

[0033] Tj is the position of the j-th color image;

[0034] Let k be the coordinates of the k-th feature match pair on the i-th color image;

[0035] Let the coordinates of the k-th feature matching pair on the j-th color image be denoted as .

[0036] w k The confidence score of the k-th feature matching pair;

[0037] N ij Let be the number of matching pairs in the i-th and j-th color images.

[0038] Optionally, before fusing and stitching together the multiple initial depth images to generate a merged depth image, the image processing method further includes:

[0039] The positions of multiple color images are optimized and adjusted according to the global binding adjustment algorithm to obtain multiple optimized image positions.

[0040] Optionally, the step of optimizing and adjusting the positions of multiple color images according to a global bundling adjustment algorithm to obtain multiple optimized image positions includes:

[0041] Using the origin of the first image as the origin of the coordinate system, a global coordinate system is constructed, and the position of the feature matching point in the color image is optimized using formula (2);

[0042] The position of the color image is optimized and iterated using formula (3) until the position change of the color image between two adjacent iterations is less than a certain threshold:

[0043] Among them, T i Let i be the position of the i-th color image in the global coordinate system;

[0044] p j Let j be the position of the j-th point on the target object in the global coordinate system.

[0045] Let J be the coordinates of the j-th point on the target object in the i-th color image;

[0046] w j Let be the weight of the j-th point on the target object;

[0047] N i Let be the number of object points observed in the i-th color image;

[0048] N j denoted as the number of color images observed for the j-th object point.

[0049] Optionally, in the step of fusing and stitching multiple initial depth images to generate a merged depth image, based on the optimized image positions, multiple color images are fused with the corresponding multiple initial depth images respectively to obtain the merged depth image and the merged color image.

[0050] Optionally, in the step of fusing multiple color images with corresponding multiple initial depth images based on the optimized image position to obtain the merged depth image, the color image with the optimized image position is fused with the corresponding multiple initial depth images in any of the following ways:

[0051] Weighted fusion, Gaussian pyramid fusion, Laplace pyramid fusion, gradient pyramid fusion, average method fusion, and hat function method fusion.

[0052] As a second aspect of this application, an electronic device is provided, the electronic device comprising:

[0053] One or more processors;

[0054] The memory stores an executable program that, when invoked by the one or more processors, enables the implementation of the image processing method described in the first aspect of this application.

[0055] Optionally, the electronic device is a microscope, and the electronic device further includes an image acquisition module, which is used to acquire images of the target object under the control of the processor to obtain multiple images to be processed.

[0056] Optionally, the image acquisition module includes a stage, a stage moving component, and an image acquisition module;

[0057] The platform is used to support the target object;

[0058] The stage moving component is used to drive the stage to move under the control of the processor;

[0059] The image acquisition module is used to acquire images of the target object under the control of the processor and generate the image to be processed.

[0060] Optionally, the stage moving assembly is used to drive the stage to move along the X-axis and / or Y-axis under the control of the processor.

[0061] Optionally, the electronic device is a scanner;

[0062] The scanner further includes a light source, an image acquisition module, and a moving component. The light source and the image acquisition module are both mounted on the moving component. The light source is used to emit light onto the target object, and the image acquisition module is used to acquire images of the target object under the control of the processor to obtain multiple images to be processed.

[0063] As a third aspect of this application, a computer-readable medium is provided having one or more computer programs stored thereon, which, when invoked, are capable of implementing the image processing method described in the first aspect of this application.

[0064] As a fourth aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described in the first aspect of this application.

[0065] In the image processing method provided in this application, firstly, multiple initial 3D meshes are reconstructed into a complete and continuous surface model through surface reconstruction technology, and the surface continuity is optimized globally to eliminate splicing traces between the initial 3D meshes and avoid mesh edge breaks.

[0066] Secondly, the merged and stitched mesh is obtained through surface reconstruction. During the surface reconstruction process, operations such as smoothing and interpolation may lead to blurring or loss of local details. In contrast, the merged depth image is created by fusing multiple initial depth maps, preserving the unsmoothed depth information from the original sensor data to the greatest extent possible. Therefore, by correcting the merged and stitched mesh based on the merged depth image, more original detail data can be added to the "smooth surface."

[0067] By using the above image processing methods, the final reconstructed surface (i.e., the target 3D mesh) can be made more consistent with the real state of the target object. Attached Figure Description

[0068] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0069] Figure 1 is a flowchart of one embodiment of the image processing method provided in this application;

[0070] Figure 2 is a flowchart of one embodiment of step S130;

[0071] Figure 3 is a flowchart of another embodiment of the image processing method provided in this application;

[0072] Figure 4 is a flowchart of an optional implementation of step S120;

[0073] Figure 5 is a flowchart of another embodiment of the image processing method provided in this application;

[0074] Figure 6 is a flowchart of an optional implementation of step S160;

[0075] Figure 7 is a flowchart of an optional implementation of step S170;

[0076] Figure 8 is a schematic diagram of the modules of the electronic device provided in this application;

[0077] Figure 9 is a schematic diagram of the computer-readable medium provided in this application;

[0078] Figure 10 is a schematic diagram of the image acquisition module provided in this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0080] In related technologies, when reconstructing surfaces, sub-meshes are pieced together using patch indexing. While this achieves visual mesh stitching, the discrete nature of the mesh means that mesh edge breaks become visible when magnified sufficiently, making it unsuitable for scenes requiring high detail. Furthermore, when performing point cloud fusion using a weighted average of matching points, abrupt depth changes can occur in overlapping areas of the point clouds to be fused, leading to a mismatch between the reconstructed surface and the actual target object.

[0081] In view of this, as a first aspect of the embodiments of this application, an image processing method is provided, wherein, as shown in FIG1, the image processing method includes:

[0082] In step S110, multiple images of the target object to be processed are acquired, including a color image and an initial depth image;

[0083] In step S130, multiple corresponding initial 3D meshes are generated based on the multiple initial depth images;

[0084] That is, for each initial depth image, an initial 3D mesh corresponding to that initial depth image is generated;

[0085] In step S150, the multiple initial depth images are fused and stitched together to generate a merged depth image;

[0086] Here, "fusion and stitching" refers to stitching together multiple initial depth images according to their positional information, and for every two initial depth images that have overlapping areas during stitching, fusing the overlapping areas between the two initial depth images.

[0087] In step S160, coordinate translation and surface reconstruction are performed on multiple initial 3D meshes based on the position information of multiple color images to obtain a merged and stitched mesh;

[0088] That is, based on the positional information of multiple color images, multiple initial 3D meshes are translated by coordinates, and the surface reconstruction is performed on the initial 3D meshes after coordinate translation. The surface reconstruction result is the merged and stitched mesh.

[0089] In step S170, the merged mesh is corrected using the merged depth image to obtain the target 3D mesh.

[0090] It should be noted that, in the embodiments of this application, the image to be processed including a color image and an initial depth image means that the image to be processed is an RGBD image. In other words, multiple images to be processed of the target object actually refer to multiple sets of images to be processed. Each set of images to be processed includes one color image and one initial depth image, and the color image and initial depth image included in each set of images to be processed are acquired synchronously. The image's location information is used to characterize the image's shooting location, i.e., from which position and angle the image was taken.

[0091] First, multiple initial 3D meshes are reconstructed into a complete and continuous surface model using surface reconstruction technology. The surface continuity is optimized globally, thereby eliminating splicing marks between the initial 3D meshes and avoiding mesh edge breaks.

[0092] Secondly, the merged and stitched mesh is obtained through surface reconstruction. During the surface reconstruction process, operations such as smoothing and interpolation may lead to blurring or loss of local details. In contrast, the merged depth image is created by fusing multiple initial depth maps, preserving the unsmoothed depth information from the original sensor data to the greatest extent possible. Therefore, by correcting the merged and stitched mesh based on the merged depth image, more original detail data can be added to the "smooth surface."

[0093] By using the above image processing methods, the final reconstructed surface (i.e., the target 3D mesh) can be made more consistent with the real state of the target object.

[0094] To improve the efficiency of the image processing method, as an optional implementation, as shown in Figure 2, the step of generating multiple corresponding initial 3D meshes based on multiple initial depth images (i.e., step S130) includes:

[0095] In step S131, multiple corresponding original 3D meshes are generated based on the multiple initial depth images;

[0096] That is, for each initial depth image, an initial 3D mesh corresponding to the initial depth image is generated, wherein the original 3D mesh is a triangular mesh;

[0097] In step S132, the original three-dimensional meshes are simplified to obtain a plurality of initial three-dimensional meshes. The initial three-dimensional meshes are triangular meshes, and the number of faces of the triangular meshes is a given value. The meshes can be simplified by means of vertex clustering, edge folding, face shrinking, etc. The specific method of mesh simplification is not limited in the embodiments of this application.

[0098] Accordingly, in the step of performing coordinate translation and surface reconstruction on multiple initial 3D meshes based on the position information of multiple color images to obtain a merged and stitched mesh, the initial 3D mesh is also a triangular mesh.

[0099] Each pixel in a depth map corresponds to a point in 3D space. Generating a triangular mesh based on a depth map essentially involves connecting adjacent points in 3D space using triangles. Since depth maps contain a large number of pixels, this results in a large number of facets after connection. Therefore, in related technologies, directly generating triangular meshes from depth maps (i.e., generating the original 3D mesh from the initial depth image) typically produces a large number of mesh facets, especially for high-resolution depth maps, where the number of facets in the triangular mesh generated directly from the depth map is even greater. The larger the number of facets in the triangular mesh, the longer the subsequent surface reconstruction process requires.

[0100] In the image processing method provided in this application embodiment, before image stitching and surface reconstruction based on the image to be processed, the acquired image to be processed is first processed to obtain a simplified triangular mesh (i.e., an initial 3D mesh), reducing the number of mesh patches in the triangular mesh. Thus, in the subsequent surface reconstruction process in step S160, the number of mesh patches that need to be processed is reduced, thereby reducing the time consumed in surface reconstruction.

[0101] In this embodiment of the application, as shown in FIG3, the image processing method further includes:

[0102] In step S120, the position information of multiple color images is determined respectively.

[0103] In the embodiments of this application, steps S110 to S132 are all operations performed on multiple images.

[0104] For example, steps S110 to S132 can be performed sequentially for all images. That is, after acquiring all the images to be processed, the position information of all color images is determined, and all the original 3D meshes are generated based on all the initial depth images; the original 3D meshes are simplified to obtain multiple corresponding initial 3D meshes (i.e., triangular meshes).

[0105] For example, steps S110 to S132 can also be: for each image, steps S110, S120, S131 and S132 are executed sequentially: for each image to be processed, the following steps are performed sequentially: determining the position information of the color image, generating the original 3D mesh based on the initial depth image, and simplifying the original 3D mesh.

[0106] In other words, for each image acquired at each location, steps S110 to S132 can be executed sequentially. For example, for location A, the image processing method may include:

[0107] In step S110, a color image and an initial depth image are acquired at location A;

[0108] In step S120, the location information of the color image acquired at location A is determined;

[0109] In step S131, a corresponding original 3D mesh is generated based on the initial depth image acquired at position A;

[0110] In step S132, the original 3D mesh diagram corresponding to position A is simplified to obtain the corresponding initial 3D mesh (i.e., triangular mesh).

[0111] In this embodiment of the application, steps S110 to S132 are equivalent to preprocessing the acquired image to be processed.

[0112] This application does not impose any special limitations on how to obtain the positional information of each color image, as long as the positional information of each color image can be determined. For example, the positional information of each color image can be determined by external input. Alternatively, image registration can be performed on multiple color images separately using feature matching to determine the positional information of each color image individually.

[0113] As an optional implementation, as shown in Figure 4, step S120 may specifically include:

[0114] In step S121, a set operator is used to extract feature points from each of the color images;

[0115] In step S122, feature points in the color image are matched with feature points in adjacent color images to determine the relative positions between two adjacent color images.

[0116] In this embodiment, the "defined operator" refers to a pre-defined "rule" or "algorithm" for selecting pixels that meet specific conditions. Accordingly, based on the defined operator, pixels meeting specific conditions can be selected from a color image as feature points. Specifically, any one of the FAST, ORB, SIFT, and SURF operators can be used to extract feature points from a color image. In this embodiment, a robust operator with sufficient time consumption can be selected to extract feature points from the color image.

[0117] In this embodiment, for a given color image, adjacent color images refer to the color images whose corresponding acquisition positions are adjacent to the acquisition position of that color image. For example, when controlling the image acquisition module to move sequentially between acquisition positions 1-6 (acquisition positions 1-6 are arranged in a 2x3 matrix, with the first row being positions 1, 2, and 3, and the second row being positions 4, 5, and 6) to acquire different parts of the target object, for a color image acquired at acquisition position 2, its adjacent color images refer to the color images acquired at acquisition positions 1, 3, and 5. As another example, when image acquisition modules AD are arranged sequentially in a clockwise direction around the target object, for a color image acquired by image acquisition model B, its adjacent color images refer to the color images acquired by image acquisition modules A and C.

[0118] As an optional implementation, the step of performing feature matching between feature points in the color image and feature points in adjacent color images to determine the relative position between two adjacent color images (i.e., step S122) specifically includes:

[0119] The relative positions between the two color images are determined according to formula (1):

[0120] Among them, T ij Let represent the relative positions of the i-th color image and the j-th color image;

[0121] T i Let i be the position of the i-th color image;

[0122] T j Let j be the position of the j-th color image;

[0123] Let k be the coordinates of the k-th feature match pair on the i-th color image;

[0124] Let the coordinates of the k-th feature matching pair on the j-th color image be denoted as .

[0125] w kThe confidence score of the k-th feature matching pair;

[0126] N ij Let be the number of feature matching pairs in the i-th and j-th color images.

[0127] Here, a "feature matching pair" refers to two feature points (a feature point i in the i-th color image) that successfully establish a matching relationship between two color images (the i-th image and the j-th image) through feature matching. n With feature point j in the j-th color image n The feature point pairs formed by )

[0128] As an optional implementation, multiple images to be processed can be numbered according to the shooting time. The first image to be processed is the image taken the first time, the second image to be processed is the image taken the second time, and so on.

[0129] To make the final target 3D mesh more closely resemble the real object, as an optional implementation method, as shown in Figure 5, before step S150, the method may further include:

[0130] In step S140, the positions of the multiple color images are optimized and adjusted according to the global bundle adjustment algorithm (BA) to obtain multiple optimized image positions.

[0131] Accordingly, step S150 is specifically performed as follows: based on the optimized image positions, multiple color images are fused with the corresponding multiple initial depth images to obtain the merged depth image. That is, on the one hand, based on the optimized image positions, multiple color images are fused, and the resulting fusion result (i.e., the merged color image) is used to provide color information for the final target 3D mesh; on the other hand, based on the optimized image positions, multiple initial depth images are fused, and the resulting fusion result is the merged depth image.

[0132] As an optional implementation, the step of optimizing the positions of multiple color images according to a global bundling adjustment algorithm to obtain multiple optimized image positions (i.e., step S140) includes:

[0133] Using the origin of the first image as the origin of the coordinate system, a global coordinate system is constructed, and the position of the feature matching point in the color image is optimized using formula (2);

[0134] The position of the color image is optimized and iterated using formula (3) until the position change of the color image between two adjacent iterations is less than a certain threshold:

[0135] Among them, T i Let i be the position of the i-th color image in the global coordinate system;

[0136] p j Let j be the position of the j-th object point in the global coordinate system;

[0137] Let J be the coordinates of the j-th point on the target object in the i-th color image;

[0138] w j Let be the weight of the j-th point on the target object;

[0139] N i Let be the number of object points observed in the i-th color image;

[0140] N j denoted as the number of color images observed for the j-th object point.

[0141] In this embodiment of the application, the weight of the object point can be set according to the confidence level of the feature point.

[0142] It should be noted that a "point of interest" refers to a physical point on the target object, which becomes a pixel on a color image. Each "feature matching point" has a corresponding "point of interest". By using the weights of different points of interest in different color images, the positions of pixels in each color image can be optimized to approximate the positions of points of interest on the target object.

[0143] In this embodiment, no special limitation is made on how to perform image fusion between multiple optimized color images and corresponding multiple initial depth images. For example, any of the following methods can be used to perform image fusion between the optimized color images and the corresponding multiple initial depth images respectively:

[0144] Weighted fusion, Gaussian pyramid fusion, Laplace pyramid fusion, gradient pyramid fusion, average method fusion, and hat function method fusion.

[0145] To achieve a balanced transition in overlapping areas and eliminate stitching seams, a weighted fusion method can optionally be used to fuse multiple optimized color images with corresponding initial depth images. By using weights for smooth transitions, abrupt changes in color and / or depth in the overlapping areas of the two overlapping images can be avoided.

[0146] Specifically, assuming img1 and img2 are the overlapping parts of the left and right images in the two images to be stitched, respectively, and L is the column number of the overlapping part, where l represents the l-th column of the left image, then the weighted fusion result is:

[0147] Among them, res l This represents the weighted fusion result corresponding to the l-th column in the left figure. This represents the fusion information (color or depth) in the l-th column of the overlapping portion of the left image in the two images to be stitched together. This represents the fusion information (color or depth) in the l-th column of the overlapping portion of the right image in the two images to be stitched together.

[0148] Traditional methods, when fusing depth information from two images using point cloud averaging, exhibit abrupt depth changes from outside the stitching region to within it, leading to abrupt depth shifts in the fused result. The solution in this application employs a weighted fusion method, where the weights of the depth information in each image transition smoothly during the fusion process, achieving a gradual transition from outside the stitching region to within it, thus avoiding abrupt depth changes.

[0149] In this embodiment, the positional relationship between multiple images to be processed is not specifically limited, as long as sufficient object points on the target object can be acquired. To obtain a high-resolution stereo image (i.e., a target stereo mesh) and to achieve wide-angle observation based on the stereo image, overlapping areas may exist between multiple images to be processed. In this embodiment, the positional information of multiple depth images and multiple color images is fused, resulting in a smooth transition of depth information in the final corrected target stereo mesh.

[0150] As an optional implementation, the multiple images to be processed are images obtained by capturing the target object at the same height, and the multiple images to be processed are captured from different angles, with the difference in capture angle between two adjacent images not exceeding a set angle. That is, in this embodiment, the corresponding pixels in the multiple images to be processed have the same height (i.e., for the same object point in each image, its height in the camera coordinate system corresponding to each image is the same), with only offsets in the xy direction.

[0151] Of course, the embodiments of this application are not limited to this. The heights of the multiple images to be processed can also be different (i.e., the multiple images to be processed may not be acquired at the same height). For multiple images to be processed with different heights, image transformation can be used to convert the multiple images to be processed to the same height.

[0152] In this embodiment of the application, as shown in FIG6, step S160 may include:

[0153] In step S161, the grid vertex coordinates of the multiple initial 3D grids are translated based on the optimized image positions of the multiple color images;

[0154] Since each initial depth image has a precise registration relationship with its corresponding color image, the two remain synchronized in terms of positional information. Therefore, after optimizing the image position of the color image, the image position of its corresponding initial depth image is also corrected. Accordingly, the spatial coordinates of the initial 3D mesh generated based on the initial depth image also need to be updated synchronously. Specifically, for each initial depth image, based on the optimized image position of the corresponding color image, the mesh vertex coordinates of the initial 3D mesh generated based on that initial depth image are translated accordingly.

[0155] In step S162, the grid in the overlapping area is removed to obtain the intermediate grid;

[0156] After translating the vertex coordinates of each initial 3D mesh, the overlapping areas between the initial 3D meshes can be removed, leaving only one mesh within the overlapping area to obtain the intermediate mesh. For example, if initial 3D mesh 1 and initial 3D mesh 2 have an overlapping area, then the mesh corresponding to initial 3D mesh 1 within the overlapping area can be removed, leaving only the mesh corresponding to initial 3D mesh 2 within the overlapping area; or, the mesh corresponding to initial 3D mesh 2 within the overlapping area can be removed, leaving only the mesh corresponding to initial 3D mesh 1 within the overlapping area.

[0157] In step S163, the grid vertex coordinates of the intermediate grid are updated according to the merged depth image to obtain the updated grid;

[0158] In step S164, implicit surface reconstruction is performed on the updated mesh to obtain the merged and stitched mesh.

[0159] Removing overlapping meshes can prevent distortion in the merged mesh. Updating the vertex coordinates of the intermediate mesh using the merged depth image can prevent abrupt depth changes in the merged mesh.

[0160] Implicit surface reconstruction includes: expressing the surface using an implicit function; fitting the implicit function based on the point cloud position information; constructing a contour surface; and discretizing the zero level set of the implicit function into a polygonal mesh.

[0161] The surface obtained by implicit surface reconstruction has continuous smoothness. Therefore, the 3D mesh (i.e., merged and spliced ​​mesh) obtained by reconstruction in step S164 is also overly smooth in regions with steep gradient changes such as vertical boundaries.

[0162] Optionally, as shown in FIG7, step S170 may include:

[0163] In step S171, the mesh vertex depth values ​​of the merged stitched mesh are corrected based on the merged depth image to obtain a corrected 3D mesh;

[0164] Both the merged depth image and the merged stitched mesh are generated based on the initial depth image. Therefore, for a point in the merged stitched mesh, the corresponding point can be found in the merged depth image. Thus, for a mesh vertex of the merged stitched mesh, the depth value corresponding to the image coordinates and the coordinates of the mesh vertex can be found in the merged depth image. Then, the found depth value is assigned to the mesh vertex to correct the depth value of the mesh vertex.

[0165] In step S172, the modified 3D mesh is simplified to obtain the target 3D mesh.

[0166] The merged and stitched mesh is obtained through surface reconstruction. During the surface reconstruction process, smoothing and interpolation operations may cause local details to be blurred or lost. The merged depth image, however, is formed by fusing multiple initial depth maps, preserving the depth information from the original sensor data to the greatest extent possible without excessive smoothing. Therefore, by correcting the mesh vertex depth values ​​in step S171, the overly smoothed mesh vertices in the merged and stitched mesh due to surface reconstruction can be assigned true depth values, thus correcting overly smoothed areas in the merged and stitched mesh. Finally, the corrected 3D mesh is simplified to obtain a target 3D mesh with fewer, more complete target patches and no seams.

[0167] As a second aspect of this application, an electronic device is provided, as shown in FIG8, the electronic device comprising:

[0168] One or more processors 101;

[0169] The memory 102 stores an executable program, which, when invoked by one or more processors 101, enables the implementation of the image processing method provided in the embodiments of this application.

[0170] The electronic device may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.

[0171] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the processor and the memory, including but not limited to a data bus (Bus).

[0172] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the control device.

[0173] As a third aspect of this application, as shown in FIG9, a computer-readable medium is provided, on which one or more computer programs are stored, which, when invoked, can implement the image processing method provided in the embodiments of this application.

[0174] As an optional implementation, the electronic device may be a microscope, and accordingly, the electronic device further includes an image acquisition module for acquiring images of the target object to obtain the image to be processed.

[0175] Figure 10 shows a schematic diagram of one possible method for the image acquisition module.

[0176] As shown in Figure 10, the image acquisition module includes a stage 100, a stage moving component 200, and an image acquisition module 300.

[0177] The stage 100 is used to carry the target object "O", and the stage moving component 200 is used to move the stage 100 under the control of the processor 101.

[0178] The image acquisition module 300 is used to acquire an image of the target object "O" under the control of the processor 101 and generate the image to be processed.

[0179] As an optional implementation, the processor 101 controls the stage moving component 200 to move to multiple different positions. At each position, the processor 101 controls the image acquisition module 300 to acquire an image of the target object "O".

[0180] As described above, since multiple images to be processed have the same height, the stage moving assembly 200 is used to move the stage along the X-axis and / or Y-axis under the control of the processor 101.

[0181] In this embodiment, the specific structure of the stage moving assembly 200 is not specifically limited. For example, the stage moving assembly 200 may include an X-axis drive motor, an X-axis translation mechanism, a Y-axis drive motor, and a Y-axis translation mechanism. The output end of the X-axis translation mechanism is connected to the housing of the Y-axis translation mechanism, and the output end of the Y-axis translation mechanism is connected to the stage. Both the X-axis drive motor and the Y-axis drive motor operate under the control of the processor.

[0182] The X-axis drive motor is used to drive the output end of the X-axis translation mechanism to move along the X direction, so as to drive the Y-axis translation mechanism and the stage to move along the X direction; the Y-axis drive motor is used to drive the output end of the Y-axis translation mechanism to move along the Y direction, so as to drive the stage to move along the Y direction.

[0183] Of course, the embodiments of this application are not limited to this, and the electronic device can also be a scanner. In this embodiment, the scanner further includes a light source, an image acquisition module, and a moving component, wherein the light source and the image acquisition module are both disposed on the moving component; the light source is used to emit light onto the target object, and the image acquisition module is used to acquire images of the target object under the control of the processor to obtain multiple images to be processed.

[0184] The scanner is a 3D scanner that can be used for 3D reconstruction of a target object.

[0185] Of course, the embodiments of this application are not limited to this. The electronic device can also be used in CT three-dimensional reconstruction, digital protection of cultural relics, virtual reality and other fields to achieve high-definition and wide-angle observation in 3D.

[0186] As a fourth aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method provided in the embodiments of this application.

[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0188] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, The image processing method includes: Acquire multiple images of the target object to be processed, including a color image and an initial depth image; Multiple corresponding initial 3D meshes are generated based on the multiple initial depth images; Multiple initial depth images are fused and stitched together to generate a merged depth image; Based on the positional information of multiple color images, coordinate translation and surface reconstruction are performed on multiple initial 3D meshes to obtain a merged and stitched mesh; The merged and stitched mesh is corrected using the merged depth image to obtain the target 3D mesh.

2. The image processing method of claim 1, wherein, Before fusing and stitching the multiple initial depth images to generate a merged depth image, the image processing method further includes: Image registration is performed on multiple color images according to feature matching to determine the position information of each color image.

3. The image processing method of claim 2, wherein, The step of performing image registration on the multiple color images according to feature matching to determine the position information of the multiple color images includes: Feature points in each of the color images are extracted using a predefined operator. Feature matching is performed between feature points in the color image and feature points in adjacent color images to determine the relative positions between two adjacent color images.

4. The image processing method of claim 3, wherein, The step of performing feature matching between feature points in the color image and feature points in adjacent color images to determine the relative positions between two adjacent color images includes: determining the relative position between two of said color images according to equation (1): wherein T ij is the relative position of the i-th color image and the j-th color image; T i is the position of the i-th color image; T j is the position of the jth color image; Let k be the coordinates of the k-th feature match pair on the i-th color image; Let the coordinates of the k-th feature matching pair on the j-th color image be denoted as . w k confidence of the kth feature match pair; N ij N is the number of matching pairs in the i-th color image and the j-th color image.

5. The image processing method of claim 2, wherein, Before fusing and stitching the multiple initial depth images to generate a merged depth image, the image processing method further includes: The positions of multiple color images are optimized and adjusted according to the global binding adjustment algorithm to obtain multiple optimized image positions.

6. The image processing method of claim 5, wherein, The step of optimizing and adjusting the positions of multiple color images according to a global bundling adjustment algorithm to obtain multiple optimized image positions includes: Using the origin of the first image as the origin of the coordinate system, a global coordinate system is constructed, and the position of the feature matching point in the color image is optimized using formula (2); The position of the color image is optimized and iterated using formula (3) until the variation of the position of the color image between two adjacent iterations is less than a certain threshold: wherein T i is the position of the i-th color image in the global coordinate system; p j Pj is the position of the jth object point on the target object in the global coordinate system; Let J be the coordinates of the j-th point on the target object in the i-th color image; w j weight for the jth object point on the target object; N i N is the number of observed points in the i-th color image; N j N is the number of color images in which the jth object point is observed.

7. The image processing method of claim 5, wherein, In the step of fusing and stitching multiple initial depth images to generate a merged depth image, based on the optimized image positions, multiple color images are fused with the corresponding multiple initial depth images respectively to obtain the merged depth image and the merged color image.

8. The image processing method of claim 7, wherein, In the step of fusing multiple color images with corresponding initial depth images based on the optimized image position to obtain the merged depth image and merged color image, any of the following methods are used to fuse the color image with the corresponding initial depth images: Weighted fusion, Gaussian pyramid fusion, Laplace pyramid fusion, gradient pyramid fusion, average method fusion, and hat function method fusion.

9. The image processing method of claim 5, wherein, The plurality of images to be processed have at least two overlapping regions, the plurality of images to be processed have the same shooting height, and the plurality of images to be processed have different shooting angles, with the difference in shooting angle between two adjacent images to be processed not exceeding a set angle. The step of performing coordinate translation and surface reconstruction on the plurality of initial 3D meshes based on the positional information of the plurality of color images to obtain a merged and stitched mesh includes: The grid vertex coordinates of the initial 3D grids are translated based on the optimized image positions of the multiple color images; Remove the overlapping mesh areas to obtain the intermediate mesh; The intermediate mesh is updated by performing mesh vertex coordinate updates based on the merged depth image to obtain the updated mesh. The updated mesh is implicitly reconstructed to obtain the merged and stitched mesh.

10. The image processing method of claim 9, wherein, The step of correcting the merged and stitched mesh using the merged depth image to obtain the target 3D mesh includes: Based on the merged depth image, the mesh vertex depth values ​​of the merged stitched mesh are corrected to obtain a corrected 3D mesh; The modified 3D mesh is simplified to obtain the target 3D mesh.

11. The image processing method of any one of claims 1 to 10, characterized in that, The step of generating multiple corresponding initial 3D meshes based on multiple initial depth images includes: Multiple corresponding original 3D meshes are generated based on the multiple initial depth images; Multiple original 3D meshes are simplified to obtain multiple initial 3D meshes, wherein each initial 3D mesh is a triangular mesh and the number of faces of the triangular mesh is a given value.

12. An electronic device, comprising: The electronic device includes: One or more processors; A memory having an executable program stored thereon, which, when invoked by the one or more processors, enables the implementation of the image processing method according to any one of claims 1 to 11.

13. The electronic device of claim 12, wherein, The electronic device is a microscope, and the electronic device also includes an image acquisition module, which is used to acquire images of the target object under the control of the processor to obtain multiple images to be processed.

14. The electronic device of claim 13, wherein, The image acquisition module includes a stage, a stage moving component, and an image acquisition module; The platform is used to support the target object; The stage moving component is used to drive the stage to move under the control of the processor; The image acquisition module is used to acquire images of the target object under the control of the processor and generate the image to be processed.

15. The electronic device of claim 14, wherein, The stage moving assembly is used to move the stage along the X-axis and / or Y-axis under the control of the processor.

16. The electronic device of claim 12, wherein, The electronic device is a scanner; The scanner further includes a light source, an image acquisition module, and a moving component. The light source and the image acquisition module are both mounted on the moving component. The light source is used to emit light onto the target object, and the image acquisition module is used to acquire images of the target object under the control of the processor to obtain multiple images to be processed.

17. A computer readable storage medium characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

18. A computer program product, characterised in that, A computer program comprising computer program elements which, when executed by a processor, perform the method of any one of claims 1-11. A computer program comprising computer program elements which, when executed by a processor, perform the method of any one of claims 1-11.

Images